Random Forest Model Training on CIC‐IDS2017 Dataset(Tuesday Working Hours)
Purpose of this document: This wiki documents the complete Random Forest training pipeline for BlackTrace — every architectural decision, every parameter choice, every result, and every conclusion. It explains the algorithm from first principles, documents the class imbalance problem and its solution, interprets the final results, and provides a complete technical reference. Written to be understood by a newcomer and rigorous enough for a senior engineer.
A single decision tree makes decisions by asking a sequence of yes/no questions about features:
Is Destination Port == 21?
Yes → Is Init_Win_bytes_backward < 65535?
Yes → Predict FTP-Patator
No → Predict BENIGN
No → Is Destination Port == 22?
Yes → Predict SSH-Patator
No → Predict BENIGN
A single tree is fast and interpretable but fragile. It memorises the training data, small changes in training data produce very different trees. This is called high variance.
Random Forest solves this by building many trees and combining their votes.
Each tree makes different errors because it was trained on different data with different features. When you aggregate their votes, the errors cancel out while the correct signals reinforce each other. This principle that combining many weak learners produces a strong learner is called ensemble learning.
The Isolation Forest's zero recall on brute force attacks revealed a fundamental gap: per-flow anomaly detection cannot identify attacks whose individual flows look statistically normal. The attack signature only becomes visible when labels are available to teach the model what distinguishes an FTP brute force flow from a normal FTP connection.
Random Forest is supervised, it learns from labelled examples. When trained on thousands of FTP-Patator flows with their labels, it discovers that Destination Port = 20 combined with specific TCP window sizes and packet length distributions reliably identifies brute force activity. It encodes this knowledge into its trees. At inference time, a new flow with those same characteristics is classified as FTP-Patator with high confidence.
The two models serve complementary roles:
flowchart LR
subgraph "What Isolation Forest does"
A["Learns shape of\nnormal traffic"]
B["Flags statistical\noutliers"]
C["Can detect\nunknown attacks"]
end
subgraph "What Random Forest does"
D["Learns distinguishing\nfeatures of each attack type"]
E["Names the specific\nattack type"]
F["High recall on\nknown attack types"]
end
G["Together:\ncomplete detection coverage"]
A --> G
D --> G
style G fill:#276749,color:#c6f6d5
Understanding the differences between the two models is essential for explaining the BlackTrace architecture.
| Aspect | Isolation Forest | Random Forest |
|---|---|---|
| Learning type | Unsupervised | Supervised |
| Uses labels | No | Yes — required |
| Training data | BENIGN rows only | All rows |
| What it learns | Shape of normal traffic | Distinguishing features of each class |
| Output | Anomaly score (-1 or +1) | Class prediction + probability |
| Detects unknown attacks | Yes — if they look unusual | No — only trained attack types |
| Performance on brute force | 0% recall | ~100% recall |
| Requires scaling | Yes | No |
| Label used | y_binary (for evaluation) | y_multiclass (for training) |
The processed dataset has a severe class imbalance:
| Class | Training rows | Percentage |
|---|---|---|
| BENIGN | 345,659 | 96.9% |
| FTP-Patator | 6,350 | 1.8% |
| SSH-Patator | 4,718 | 1.3% |
Consider what happens if you train a Random Forest on this data without any imbalance handling. The model sees 345,659 BENIGN examples for every 6,350 FTP-Patator examples. The mathematically optimal strategy and the one that minimises overall loss is to predict BENIGN for everything.
If the model always predicts BENIGN:
Correct on 345,659 + 86,415 = 432,074 rows
Wrong on 6,350 + 4,718 + 1,588 + 1,179 = 13,835 rows
Accuracy = 432,074 / 445,909 = 96.9%
A model that detects zero attacks achieves 96.9% accuracy. This is the class imbalance trap. Accuracy is a completely useless metric here. The metric that matters is recall for attack classes — what fraction of actual attacks did the model correctly identify.
BlackTrace uses two complementary techniques together. Neither alone is sufficient.
flowchart TD
A["Imbalanced training data\n96.9% BENIGN / 3.1% attacks"] --> B
A --> C
B["Layer 1 — SMOTE\nCreate synthetic attack samples\nBalance the data distribution\nBEFORE training"] --> D
C["Layer 2 — class_weight='balanced'\nAdjust the loss function\nPenalise attack misclassifications\nmore heavily DURING training"] --> D
D["Combined effect:\nModel sees equal class examples\nAND treats attack errors as more costly"] --> E
E["Result:\nHigh recall on all three classes\nModel cannot default to predicting BENIGN"]
style E fill:#276749,color:#c6f6d5
Why both layers are needed:
SMOTE alone balances the data but does not change how the model weights errors during training. class_weight='balanced' alone adjusts training but leaves the underlying data imbalanced — some tree paths still see mostly BENIGN data. Together they produce significantly better recall than either technique alone.
flowchart TD
A([Processed CSV\n445,909 × 82]) --> B
B["Load dataset and label mapping\nX = 79 features\ny_multiclass = 0/1/2\nclass_names from label_mapping.json"] --> C
C["Stratified train/test split\nstratify=y_multiclass\nrandom_state=42\n80/20 split"] --> D
D["Apply SMOTE to training set only\n345,659 → 345,659 BENIGN\n6,350 → 345,659 FTP-Patator\n4,718 → 345,659 SSH-Patator"] --> E
E["Train RandomForestClassifier\nn_estimators=100\nclass_weight='balanced'\nrandom_state=42\nn_jobs=-1"] --> F
F["Evaluate on untouched test set\nclassification report\nconfusion matrix\nfeature importances"] --> G
G["Save model\nrandom_forest.joblib"] --> H
H([Model saved\ndetection_engine/models/random_forest.joblib])
style A fill:#2d3748,color:#e2e8f0
style H fill:#276749,color:#c6f6d5
df = pd.read_csv(PROCESSED_PATH)
X = df.drop(columns=NON_FEATURE_COLS)
y_multi = df["y_multiclass"]
with open(LABEL_MAPPING_PATH) as f:
label_mapping = json.load(f)
int_to_label = {v: k for k, v in label_mapping.items()}
class_names = [int_to_label[i] for i in sorted(int_to_label.keys())]Why y_multiclass and not y_binary:
Random Forest is a multiclass classifier. Its job is to distinguish between BENIGN, FTP-Patator, and SSH-Patator — three separate classes. Using y_binary would collapse FTP-Patator and SSH-Patator into a single ATTACK class. The model would learn that something is an attack but not which type. That information is what drives the incident reports and recommended actions in BlackTrace's downstream pipeline.
Why the label mapping is loaded:
y_multiclass contains integers (0, 1, 2). The classification report needs human-readable names ("BENIGN", "FTP-Patator", "SSH-Patator") to be interpretable. The label mapping saved during preprocessing is loaded here and reversed to create int_to_label.
The label leakage assertion:
leaked = set(NON_FEATURE_COLS) & set(X.columns)
assert len(leaked) == 0This hard check runs every time the script executes. If any label column were accidentally included in features, the assertion crashes the script immediately with a clear message rather than silently training a leaky model.
Feature matrix entering training: (445,909, 79)
X_train, X_test, y_train, y_test = train_test_split(
X, y_multi,
test_size=0.2,
random_state=42,
stratify=y_multi
)Split results:
| Set | Total rows | BENIGN | FTP-Patator | SSH-Patator |
|---|---|---|---|---|
| Train | 356,727 | 345,659 | 6,350 | 4,718 |
| Test | 89,182 | 86,415 | 1,588 | 1,179 |
Why stratify=y_multiclass and not stratify=y_binary:
Stratifying on the multiclass label guarantees all three classes appear in both sets proportionally. If you stratify on y_binary, the split ensures the correct 97%/3% binary ratio but does not guarantee the FTP-Patator / SSH-Patator sub-ratio is preserved. With only 5,897 SSH-Patator rows in the full dataset, a non-stratified or binary-stratified split could accidentally assign almost no SSH-Patator rows to the test set, making precision/recall for that class completely meaningless.
Why random_state=42:
Every script that recreates this exact split — training, evaluation, future scripts — must use the same seed. Different seeds produce different splits. If the evaluate script uses seed 99, it tests on rows that were in training, inflating all metrics. The seed is a contract between all scripts.
Critical rule — SMOTE comes after the split:
The split must happen before SMOTE. This is the most common mistake in imbalanced classification. If SMOTE is applied before splitting, synthetic minority samples generated from training rows will appear in the test set. Evaluating on synthetic data you created yourself produces fictional metrics that collapse completely on real-world data.
flowchart LR
A["Full dataset"] --> B["Split first"]
B --> C["Training set\n→ Apply SMOTE here"]
B --> D["Test set\n→ Never touch\nReal distribution only"]
style D fill:#276749,color:#c6f6d5
style C fill:#2c5282,color:#bee3f8
smote = SMOTE(random_state=42)
X_resampled, y_resampled = smote.fit_resample(X_train, y_train)What SMOTE does mechanically:
SMOTE does not simply duplicate existing minority rows. It generates entirely new synthetic rows by interpolating between existing minority class samples in feature space.
For each minority sample, SMOTE:
- Finds its k nearest neighbours within the same class (default k=5)
- Randomly selects one of those neighbours
- Creates a new synthetic sample at a random point along the line connecting the original sample to its neighbour
flowchart LR
A["Existing FTP-Patator\nflow A\n79 features"] --> C
B["Existing FTP-Patator\nflow B\n79 features"] --> C
C["SMOTE interpolates\nnew_sample = A + random(0,1) × (B - A)"] --> D
D["Synthetic FTP-Patator flow\nlooks plausible\nnot identical to A or B"]
style D fill:#276749,color:#c6f6d5
Before and after SMOTE:
| Class | Before SMOTE | After SMOTE |
|---|---|---|
| BENIGN | 345,659 | 345,659 (unchanged) |
| FTP-Patator | 6,350 | 345,659 (+339,309 synthetic) |
| SSH-Patator | 4,718 | 345,659 (+340,941 synthetic) |
| Total | 356,727 | 1,036,977 |
The training set grew from 356,727 to 1,036,977 rows. This is expected and correct.
Why SMOTE on training data only:
The test set must reflect the real-world distribution — 96.9% BENIGN, 3.1% attacks. If you SMOTE the test set, you are evaluating the model on synthetic data that does not exist in production. Your metrics become fictional. In real deployment, the model will encounter real-world distribution and perform completely differently from what your synthetic-test metrics predicted.
model = RandomForestClassifier(
n_estimators=100,
class_weight="balanced",
random_state=42,
n_jobs=-1
)
model.fit(X_resampled, y_resampled)Parameter explanations:
| Parameter | Value | Reason |
|---|---|---|
n_estimators |
100 | Number of trees. More trees = more stable predictions. 100 is the standard starting point — diminishing returns beyond 200. |
class_weight |
'balanced' |
Automatically weights each class inversely proportional to its frequency. Works alongside SMOTE as a second imbalance layer. |
random_state |
42 | Reproducibility. Same seed = same trees = same results every run. |
n_jobs |
-1 | Use all available CPU cores. Building 100 trees is fully parallelisable. |
Why no StandardScaler:
Random Forest splits data using information gain — it asks "does splitting on feature X at value V reduce impurity?" This calculation is entirely based on the relative ordering of values, not their absolute magnitude. Whether Flow Bytes/s is 145,000,000 or 1.0 after scaling, the tree makes the same split decisions. Scaling has zero effect on tree-based models and is intentionally omitted. This also means the saved scaler.joblib from Isolation Forest training is not loaded or used here.
Training on resampled data:
The model trains on X_resampled — 1,036,977 rows with equal class representation. It has never seen the test set. The test set has been untouched since the split in Step 2.
y_pred = model.predict(X_test)
print(classification_report(y_test, y_pred, target_names=class_names))The test set used for evaluation contains real rows in their real distribution — 86,415 BENIGN, 1,588 FTP-Patator, 1,179 SSH-Patator. No synthetic rows. No SMOTE. This is what the model will encounter in production.
No sklearn convention conversion needed:
Unlike Isolation Forest which returns -1 and +1, Random Forest's predict() directly returns class integers (0, 1, 2). No conversion required.
joblib.dump(model, MODEL_DIR / "random_forest.joblib")The Random Forest model is saved alone. Unlike Isolation Forest which requires its scaler to be saved alongside it, Random Forest requires no scaler — data enters the model in its original unscaled form.
The saved file path: detection_engine/models/random_forest.joblib
precision recall f1-score support
BENIGN 1.00 1.00 1.00 86415
FTP-Patator 1.00 1.00 1.00 1588
SSH-Patator 1.00 1.00 1.00 1179
accuracy 1.00 89182
macro avg 1.00 1.00 1.00 89182
weighted avg 1.00 1.00 1.00 89182
[[86415 0 0]
[ 0 1588 0]
[ 2 0 1177]]
| Predicted BENIGN | Predicted FTP-Patator | Predicted SSH-Patator | |
|---|---|---|---|
| Actual BENIGN | 86,415 | 0 | 0 |
| Actual FTP-Patator | 0 | 1,588 | 0 |
| Actual SSH-Patator | 2 | 0 | 1,177 |
Only 2 rows were misclassified out of 89,182. Both were SSH-Patator flows predicted as BENIGN.
Why these 2 rows were missed:
SSH-Patator and BENIGN traffic both use port 22 for legitimate SSH connections. Among the 1,179 SSH-Patator test flows, 2 had feature values so similar to normal SSH traffic that even the Random Forest could not distinguish them. This is expected — no model achieves perfect recall on every possible variant of a real attack. A 99.83% recall on SSH-Patator is an excellent result.
| Metric | Value | Plain meaning |
|---|---|---|
| BENIGN precision 1.00 | Every flow flagged as BENIGN was actually BENIGN | Zero false alarms |
| FTP-Patator recall 1.00 | Every FTP brute force flow was correctly identified | Zero FTP attacks missed |
| SSH-Patator recall 1.00* | 1,177 of 1,179 SSH brute force flows correctly identified | 2 SSH attacks missed |
| Overall accuracy 1.00 | 89,180 of 89,182 rows correctly classified | Near-perfect performance |
*Reported as 1.00 due to rounding at 2 decimal places. Exact value is 0.9983.
Random Forest measures each feature's contribution to reducing impurity across all 100 trees. A feature with high importance consistently produces splits that cleanly separate classes. A feature with low importance produces splits that barely help.
Importance values sum to 1.0 across all 79 features.
| Rank | Feature | Importance | Why it matters |
|---|---|---|---|
| 1 | Destination Port |
0.1684 | FTP=21, SSH=22 — single strongest separator |
| 2 | Init_Win_bytes_backward |
0.0788 | Server TCP window size is service-specific |
| 3 | Init_Win_bytes_forward |
0.0520 | Brute force tool TCP window differs from browser |
| 4 | Packet Length Mean |
0.0483 | Login packets have consistent small sizes |
| 5 | Average Packet Size |
0.0379 | Correlated with packet length mean |
| 6 | min_seg_size_forward |
0.0366 | Minimum TCP segment size in forward direction |
| 7 | Fwd Packet Length Max |
0.0330 | Maximum forward packet size |
| 8 | Flow Bytes/s |
0.0284 | Throughput differs between brute force and normal |
| 9 | Fwd Packet Length Mean |
0.0270 | Forward packet length average |
| 10 | Bwd Packets/s |
0.0266 | Backward packet rate |
| 11 | Fwd Packet Length Std |
0.0256 | Variance in forward packet sizes |
| 12 | Packet Length Std |
0.0255 | Overall packet size variance |
| 13 | Avg Fwd Segment Size |
0.0241 | Average TCP segment size forward |
| 14 | Fwd Header Length |
0.0238 | Forward TCP header length |
| 15 | Packet Length Variance |
0.0225 | Packet size variance |
This is the key insight from the feature importance analysis. FTP-Patator attacks exclusively target port 21 (the FTP service). SSH-Patator attacks exclusively target port 22 (the SSH service). BENIGN traffic is distributed across many ports.
A single split on Destination Port:
- All flows to port 21 → very likely FTP-Patator
- All flows to port 22 → likely SSH-Patator or legitimate SSH
- All other ports → very likely BENIGN
This one feature reduces uncertainty so dramatically that the remaining 78 features only need to handle the ambiguous cases (legitimate SSH vs SSH brute force on port 22, and the small amount of legitimate FTP traffic on port 21).
Init_Win_bytes_backward and Init_Win_bytes_forward capture the TCP window size negotiated at the start of the connection. Brute force tools typically use non-standard TCP stack implementations with distinctive window size values. Legitimate operating systems (Windows, Linux, macOS) use well-known default window sizes. This makes TCP window size a reliable secondary discriminator for automated attack traffic.
is_zero_duration rank: 67 of 79
is_zero_duration importance: 0.000001
is_zero_duration ranked 67th out of 79 features with effectively zero contribution to the Random Forest's decisions.
The feature was engineered to provide additional signal for distinguishing attack flows from BENIGN flows. However, Destination Port alone creates near-perfect class separation in this dataset. By the time the model considers is_zero_duration, the class decision has already been made with very high confidence based on port and TCP window features.
When a feature with stronger discriminative power already exists, additional features that encode overlapping information become redundant. is_zero_duration is correlated with attack behavior, but Destination Port captures the same separation more directly and powerfully.
Low importance in one model on one dataset does not invalidate the engineering reasoning. Three points:
Point 1 — The reasoning is sound and defensible. Instantaneous zero-duration flows are a real signature of scanning and brute force reconnaissance. The decision to flag them was based on network security domain knowledge, not on guesswork. That reasoning stands regardless of the feature's importance score.
Point 2 — The feature will have higher importance in other contexts.
When BlackTrace is extended with Wednesday (DoS) and Friday (DDoS, PortScan) data, Destination Port becomes less discriminative — DoS attacks can target any port. is_zero_duration may rise significantly in the rankings when the dataset includes port scan data, where zero-duration flows are particularly concentrated.
Point 3 — The Isolation Forest context is different.
The feature was always more architecturally relevant to the Isolation Forest, which cannot use port labels to identify attacks. In an unsupervised context, is_zero_duration provides directional signal that Destination Port cannot provide. That the supervised Random Forest found a more powerful substitute is expected behavior, not a failure.
Data leakage occurs when information about the target variable (what you are trying to predict) is accidentally included in the input features. The model learns to use this leaked information instead of real patterns. It achieves perfect metrics during evaluation but fails completely in production where the leaked information does not exist.
After training, this check was run:
print("is_zero_duration in test features:", "is_zero_duration" in X_test.columns)
print("y_binary in test features:", "y_binary" in X_test.columns)
print("y_multiclass in test features:", "y_multiclass" in X_test.columns)
print("Label in test features:", "Label" in X_test.columns)Output:
is_zero_duration in test features: True
y_binary in test features: False
y_multiclass in test features: False
Label in test features: False
| Column | In features | Correct? | Why |
|---|---|---|---|
is_zero_duration |
True | Yes | Engineered feature — should be in X |
y_binary |
False | Yes | Label — must not be in X |
y_multiclass |
False | Yes | Label — must not be in X |
Label |
False | Yes | Raw string label — must not be in X |
No label leakage exists. The near-perfect results are genuine.
Several properties confirm legitimacy:
The confusion matrix shows 2 errors. A perfectly leaky model would show zero errors. Two SSH-Patator flows were misclassified as BENIGN — exactly the kind of boundary-case errors a real model makes on genuinely ambiguous flows.
The feature importances are domain-coherent. Destination Port at 16.8% makes complete security sense. FTP attacks go to port 21, SSH attacks go to port 22. If the model had learned from leaked labels, feature importances would be distributed differently — the label columns would absorb all importance leaving legitimate features with near-zero scores.
The train/test split used stratification and a fixed seed. The test set contains real rows that were never modified, never resampled, and never seen during training or SMOTE.
FTP-Patator and SSH-Patator attacks generate flows that are statistically indistinguishable from normal traffic at the per-flow level in terms of general statistics — but they target specific ports with specific TCP stack signatures. Supervised learning can exploit this directly. The Random Forest discovered Destination Port as the primary discriminator and achieved near-perfect classification.
The 96.9%/3.1% class imbalance that would have caused a naive model to predict only BENIGN was completely neutralised. The model achieves 1.00 recall on FTP-Patator and 0.9983 recall on SSH-Patator. Both techniques were necessary — SMOTE alone or class weighting alone would have produced inferior results.
random_forest.joblib is saved, trained, and evaluated. The model can be loaded by the FastAPI inference endpoint to classify incoming network flows in real time. No scaler is required at inference time — raw (preprocessed) feature values are passed directly.
| Attack type | Isolation Forest recall | Random Forest recall |
|---|---|---|
| FTP-Patator | 0.00 | 1.00 |
| SSH-Patator | 0.00 | 1.00 |
The Random Forest compensates exactly for the Isolation Forest's limitation. Neither model alone is sufficient. Together they provide complete coverage of the Tuesday dataset attack types, with the Isolation Forest contributing anomaly detection capability for attack types beyond the training distribution.
is_zero_duration ranked 67th in this specific model on this specific dataset because stronger discriminators existed. The reasoning behind the feature is sound and it will contribute more meaningfully when the dataset is extended to include PortScan data where zero-duration flows are concentrated and Destination Port is less discriminative.
