SheepHunter

Data

Download the datasets first:

• E3: https://github.com/darpa-i2o/Transparent-Computing/blob/master/README-E3.md
• Arena / simulated data: https://github.com/PKU-ASAL/Simulated-Data

Expected datasets:

• E3: cadets, theia, trace
• Arena-style: linux, win10, wins12

Expected layout:

logs/e3/<dataset>/
logs/Nodlink/<dataset>/

Each dataset directory should contain:

• raw log files
• <dataset>.txt with malicious process UUIDs

Environment

conda create -n fpreduce python=3.9 -y
conda activate fpreduce
pip install -r requirements.txt

requirements.txt has been updated to match the current code dependencies.

Prepare Directories

Create the working directories before running:

mkdir -p t results

t/ stores intermediate caches such as:

• overfit-<dataset>.pkl
• magic-overfit-<dataset>.pkl
• idf-overfit-<dataset>.pkl
• semantic-overfit-<dataset>.pkl

Run Overfit Detector

E3:

python e3_overfit_detect.py --dataset cadets
python e3_overfit_detect.py --dataset theia
python e3_overfit_detect.py --dataset trace

Arena-style:

python nodlink_overfit_detect.py --dataset linux
python nodlink_overfit_detect.py --dataset win10
python nodlink_overfit_detect.py --dataset wins12

This step creates t/overfit-<dataset>.pkl.

Build Embeddings

E3 example:

python e3_parse.py --dataset cadets

python e3_embedding_load.py --dataset cadets --detect_method overfit --embedding_method magic --embedding_dir results/magic_cadets_embedding.npz --only_positive
python e3_embedding_load.py --dataset cadets --detect_method overfit --embedding_method idf --embedding_dir results/magic_cadets_embedding.npz --only_positive
python e3_embedding_load.py --dataset cadets --detect_method overfit --embedding_method semantic --embedding_dir results/magic_cadets_embedding.npz --only_positive

Arena-style example:

python nodlink_parse.py --dataset linux

python nodlink_embedding_load.py --dataset linux --detect_method overfit --embedding_method magic --embedding_dir results/magic_linux_embedding.npz --only_positive
python nodlink_embedding_load.py --dataset linux --detect_method overfit --embedding_method idf --embedding_dir results/magic_linux_embedding.npz --only_positive
python nodlink_embedding_load.py --dataset linux --detect_method overfit --embedding_method semantic --embedding_dir results/magic_linux_embedding.npz --only_positive

Run FP Reduction

python fp_detect.py --dataset cadets --detect_method overfit --embedding_method magic
python fp_detect.py --dataset cadets --detect_method overfit --embedding_method idf
python fp_detect.py --dataset cadets --detect_method overfit --embedding_method semantic

python get_result.py --dataset cadets --detect_method overfit

Apply the same pattern to:

• cadets
• theia
• trace
• linux
• win10
• wins12

Provided Cached Results

Cached results for validating the overfit pipeline can be downloaded from:

• https://drive.google.com/file/d/1njqlbm1A-Yr69e5vYHiNWm6actf73HJl/view?usp=sharing

The archive also includes precomputed embeddings.

The archive includes files such as:

• magic_<dataset>_embedding.npz
• community_result-<embedding>-overfit-<dataset>.pkl
• overfit_<dataset>_summary.txt

If you reuse cached files, make sure the dataset name matches the file name.