Skip to content

Latest commit

 

History

History
40 lines (34 loc) · 2.95 KB

guideline.md

File metadata and controls

40 lines (34 loc) · 2.95 KB
Raw

Scripts

As we are applying our method Run-off Election on two deterministic defense methods Deep Partiton Aggregation and Finite Aggregation, we generally use the same code of work done by this paper (Improved Certified Defenses against Data Poisoning with (Deterministic) Finite Aggregation) in this repo.

Assigning samples to training sets of different base learners

cd train
python3 FiniteAggregation_data_norm_hash.py --dataset=cifar --k=50 --d=16

Here --dataset can be mnist, cifar and gtsrb, which are benchmarks evaluated in our paper; --k and --d corresponds to the hyper-parameters for our FA+ROE. d=1 corresponds to DPA+ROE. For details, please refer to Section 3 of our paper.

Training the base learners

cd train
python3 FiniteAggregation_train_cifar_nin_baseline.py --k=50 --d=16 --start=0 --range=800 --version 1

Here --k and --d are the same as above, and a total of $k\cdot d$ base learners will be trained independently. --start and --range specify which base learners are trained with this script. Also, --version shows the version of base classifiers trained, version plays role in the initial seed of classifiers. For instance, when one uses --k=50 and --d=16, one can use --start=0 and --range=800 to train all base learners sequentially, or one can use two separate runs with repsectively --start=0 and --start=400 (both with --range=400) to train in parallel the first 400 and the last 400 base learners. To train on MNIST and GTSRB, run FiniteAggregation_train_mnist_nin_baseline.py and FiniteAggregation_train_gtsrb_nin_baseline.py respectively.

Collecting predictions of base learners on test sets

python3 prediction/FiniteAggregation_evaluate_cifar_nin_baseline.py --models=cifar_nin_baseline_FiniteAggregation_k50_d16 --version 1

For MNIST and GTSRB, run FiniteAggregation_evaluate_mnist_nin_baseline.py and FiniteAggregation_evaluate_gtsrb_nin_baseline.py instead. We note that --version refers to the version of classifiers that you're looking for their predictions.

Computing the certified radius using the collected predictions

These three lines of codes, find the certified radius based on methods:

  • DPA+ROE
  • FA+ROE
  • FA.
python3 dpa_roe_cerfity.py --evaluations=cifar_nin_baseline_FiniteAggregation_k50_d1 --num_classes=10 --version 1
python3 fa_roe_cerfity.py --evaluations=cifar_nin_baseline_FiniteAggregation_k50_d16 --k=50 --d=16 --num_classes=10 --version 1
python3 fa_cerfity.py --evaluations=cifar_nin_baseline_FiniteAggregation_k50_d16 --k=50 --d=16 --num_classes=10 --version 1

Here --num_classes is the size of the label set on the evalauted dataset (i.e. --num_classes=10 for MNIST and CIFAR-10 and --num_classes=43 for GTSRB) and --version shows the version of classifiers you're looking for their certificates.