Model Card

Model Card

Prohori's first flow detectors (v0.2.0, in progress): gradient boosting, trained on labelled flows, and an Isolation Forest trained on benign flows only. Both are trained and tested on UNSW-NB15 and the improved CIC-IDS-2017 after the leakage controls described on Datasets. The full card, with every table, is docs/MODEL_CARD.md.

How the models are judged

• A threshold is chosen on the validation split for a 1 % or 0.1 % false-positive rate, then frozen. The test split is scored once per seed.
• Reported: PR-AUC, the false-positive rate actually reached on test, recall, false alerts per 10,000 flows and recall per attack class with its number of test flows. Mean and standard deviation over seeds 0, 1 and 2. Never accuracy.

What the results say

	UNSW-NB15	CIC-IDS-2017
Gradient boosting, PR-AUC	0.95	0.999
Gradient boosting, recall at the 0.1 % threshold	about 46 %	99.98 %
Isolation Forest, recall at the 1 % threshold	under 5 %	under 1 %

• On UNSW-NB15 the threshold chosen for 1 % false positives on validation gave 4.5 % on the test file: thresholds must be re-checked on the traffic a model is used on.
• Gradient boosting misses most Fuzzers and Analysis flows on UNSW-NB15.
• CIC-IDS-2017's near-perfect scores reflect a lab week with a few scripted attack tools, not readiness for a real network.
• Without the leakage controls the same pipeline reports a PR-AUC of 0.989 instead of 0.952 on UNSW-NB15.

Do they work on another network?

Not usefully. The cross-dataset test trains on one dataset's twelve shared features (a lossy mapping: durations, packet and byte counts, sizes, inter-arrival times, rates and protocol) and scores the other:

Trained on → tested on	ROC-AUC, same dataset	ROC-AUC, other dataset	PR-AUC (chance), other dataset	Benign flows flagged at the 1 % threshold
UNSW-NB15 → CIC-IDS-2017	0.95	0.64	0.32 (0.16)	42 %
CIC-IDS-2017 → UNSW-NB15	1.00	0.61	0.40 (0.33)	60 %

A flow model trained on one lab's traffic learns that lab: train it and set its threshold on the network it will watch.

Score your own flows

prohori score flows.csv --model cicids2017/hgb-seed0

prohori score (needs the ml extra) reads a CSV in the layout the model was trained on (CICFlowMeter output for a CIC-IDS-2017 model), scores it in chunks and lists the flagged flows with their addresses and a summary. It exits with 3 when flows are flagged. See docs/CLI.md.

Reproduce

uv run python -m prohori.ml.train --dataset unsw-nb15 --seeds 0 1 2
uv run python -m prohori.ml.train --dataset cicids2017 --seeds 0 1 2
uv run python -m prohori.ml.transfer --seeds 0 1 2
uv run python scripts/generate_results_table.py

Saved models go to the git-ignored models/ directory. Model files are pickles: loading one runs code from it. Load only models you trained yourself.