Guide

Samples we have (in onedrive/datasets)

~4000 MalwareBazaar ~560 Rearmed MOTIF (Features not extracted yet) ~8000 Benign ~4000 BenchMFC Base ~4000 BenchMFC Evolving ~4000 BenchMFC Unseen

0. Build the images natively in Docker

docker compose -f docker-compose.finetune.yml build

1. Extract Stage 2 Features (MOTIF Top 10 Families) NOT DONE

Isolate top 10 families, run static triage, extract FCGs, and merge to JSONL.

# Isolate Top 10 PE Files PROVIDED IN ONEDRIVE
python hpc-malware/experiments/motif/isolate_top10_pe.py --manifest datasets/MOTIF/motif_manifest.json --output-dir datasets/MOTIF/top10_samples

# Stage 1: Static Triage
python scripts/pipeline/s1_static_triage.py datasets/MOTIF/top10_samples datasets/MOTIF/top10_stage1

# Stage 2: Ghidra Semantic Extraction (2 workers)
python scripts/pipeline/s2_semantic_restoration.py datasets/MOTIF/top10_samples datasets/MOTIF/top10_stage2 --stage1-dir datasets/MOTIF/top10_stage1 --workers 2

# Merge to JSONL 
python hpc-malware/experiments/motif/build_motif_top10.py --manifest datasets/MOTIF/motif_manifest.json --stage1-dir datasets/MOTIF/top10_stage1 --stage2-dir datasets/MOTIF/top10_stage2 --output datasets/MOTIF/motif_top10_no_stage3.jsonl

2. Stage 3 Consensus (Percentage Backfills) DONE

Generate datasets for 10%, 20%, 50%, and 80% consensus.

# 10% Consensus
python scripts/pipeline/backfill_stage3_custom.py scripts/pipeline/datasets_to_backfill.json --consensus-threshold 0.10 --refit-memory

# 20%, 50%, 80% Consensus (Run sequentially)
python scripts/pipeline/backfill_stage3_custom.py scripts/pipeline/datasets_to_backfill.json --consensus-threshold 0.20 --refit-memory
python scripts/pipeline/backfill_stage3_custom.py scripts/pipeline/datasets_to_backfill.json --consensus-threshold 0.50 --refit-memory
python scripts/pipeline/backfill_stage3_custom.py scripts/pipeline/datasets_to_backfill.json --consensus-threshold 0.80 --refit-memory

3. Build Fine-Tuning Datasets

Generate Train/Val splits utilizing the chosen reasoning modes. Supported modes are cot, nocot, posthoc, and posthoc_cot.

# Motif Top 10 @ 20% (Post-Hoc CoT)
python hpc-malware/universal/universal_finetune_dataset.py --mode posthoc_cot --malicious datasets/MOTIF/motif_top10_20pct.jsonl --benign datasets/malwarebazaar/benign_features_20pct.jsonl --output-dir splits/motif_20pct_cot

# MalwareBazaar @ 20% (Post-Hoc CoT)
python hpc-malware/universal/universal_finetune_dataset.py --mode posthoc_cot --malicious datasets/malwarebazaar/malwarebazaar_malicious_refined_20pct.jsonl --benign datasets/malwarebazaar/benign_features_20pct.jsonl --output-dir splits/mb_20pct_cot

(Repeat for 10%, 20%, 50%, 80%, malware, benign varying --mode to nocot, cot, or posthoc by changing paths).

4. Fine-Tune Models

Run QLoRA in Docker. Checkpoints save automatically.

# Fine-tune Motif 20% CoT
docker compose -f docker-compose.finetune.yml run --rm finetune --data-dir /app/splits/motif_20pct_cot --output-dir /app/models/llm_motif_20pct_cot --model-name HuggingFaceTB/SmolLM-135M --use-4bit --epochs 1 --max-steps 2 --save-steps 1

(Repeat for other dataset splits).

5. Evaluate Models

Test models against reserved test.jsonl.

# Evaluate Motif 20% CoT Model
python hpc-malware/experiments/motif/evaluate_finetuned_family_llm.py --model-dir models/llm_motif_20pct_cot/merged_model_fp16 --test-file splits/motif_20pct_cot/test.jsonl --repo-dir datasets/MOTIF/repo --output datasets/eval_motif_20pct_cot.json