collect-outputs util

[jprokos26]

Can be utilized to generate evaluation tables automatically utilizing various flags.
Organize by performer, attack type, or any other config field.
Additionally clean up your output directory automatically.

• Remove folders with just armory-log.txt and colored-log.txt
• Remove pairs of duplicate png images from saved_samples
• Removed files are placed in ~/.armory/outputs/.cleaned

Example:
After running:

cd ~/git/twosixlabs/gard-evaluatsion/MITM_sleeper_agent/jprokos26
python3 -m armory utils collect-outputs --collate --glob "*Poison*.json" --sort --filter "diffusion"
cp -Lr ~/.armory/results/SleeperAgentAttack* .
git commit -am "Adding diffusion sleeper results"

Note, cwd affects href link for config paths
See https://github.com/twosixlabs/gard-evaluations/tree/eval7-jp-MITM_sleeper/MITM_sleeper_agent/jprokos26 for output

[christopherwoodall]

The suggestion below does not prohibit the inclusion of this branch.

[jprokos26]

Need to add flag to change where it is looking for results files instead of just ARMORY_OUTPUT_DIR

[jprokos26]

Addressed by 723c810.

Stale:
Add runtime field. Can be extracted from armory-log.txt

for file in SleeperAgentAttack/*/armory-log.txt; do echo -n $(basename $(find $(dirname $file) -name "*.json")); echo -n " - "; tail -n 1 $file | cut -d" " -f3; done

[jprokos26]

If defense is null, should grab other kwargs
https://github.com/twosixlabs/armory/blob/master/scenario_configs/eval6/poisoning/sleeper_agent/baseline_defenses/cifar10_sleeper_agent_p10_perfect_filter.json#L58-L64

[swsuggs]

Acknowledging that this tool is not comprehensive but will still be useful for eval 8. The JATIC armory-library will accomplish similar goals via MLFlow.