Datasets

Datasets

The machine-learning track (v0.2.0, in progress) trains flow-based detectors on two public intrusion-detection datasets. Neither is shipped with Prohori: you download them into the git-ignored data/ directory. The full reference is docs/DATASETS.md.

	UNSW-NB15	CIC-IDS-2017 (improved, CNS 2022)
Files	Official training and testing partition CSVs	CICIDS2017_improved.zip with one CSV per capture day
How to get it	By hand from the UNSW SharePoint folder	fetch cicids2017
Licence	Free for academic research; commercial use by agreement	None stated; research use, cite the papers

Getting the data

Install the ml extra (from a clone, uv sync already includes it), then:

uv run python scripts/download_datasets.py list                # what to get, licences, citations
uv run python scripts/download_datasets.py fetch cicids2017    # about 330 MB download
uv run python scripts/download_datasets.py verify              # sizes and SHA-256
uv run python scripts/download_datasets.py prepare cicids2017 --sample 0.1

For UNSW-NB15:

1. Open https://research.unsw.edu.au/projects/unsw-nb15-dataset and follow the "HERE" link.
2. Open "CSV Files", then "Training and Testing Sets".
3. Download UNSW_NB15_training-set.csv and UNSW_NB15_testing-set.csv.
4. Put both in data/raw/unsw-nb15/ without renaming them.
5. Run uv run python scripts/download_datasets.py verify unsw-nb15.

The CIC-IDS-2017 download is checked against a pinned size and SHA-256 before it is opened, and only the five expected CSVs are extracted.

Leakage controls

Published intrusion-detection scores are often inflated by information leaking from the test data. Prohori's preparation, in order:

1. One label scheme for both datasets: attack_cat and a 0/1 label. "Attempted" CIC-IDS-2017 flows, which carried no attack payload, become benign.
2. Identifier, address, port, timestamp and label-derived columns are dropped.
3. Infinite values become missing; missing values are filled with training medians.
4. Duplicates are removed, and so are rows with identical features but different classes.
5. UNSW-NB15 keeps its official split (with a validation set from the training file); CIC-IDS-2017 gets a seeded split stratified by class.
6. Test rows that also occur in training are removed and counted.
7. Transforms are fitted on the training split only.

A leakage check runs on every prepared dataset and refuses data that fails it. The reasons for each choice are in ADR-0019 of the decision log.