Config parameter on time-out

[JacksonZyy]

Hi,

I tried to run MNIST_ERAN dataset with network mnist_convSmallRELU__Point.
And I tried to increase the execution time with the hope that it would return more conclusive images.
I update the config as follows:
bab: timeout: 2000 branching: reduceop: max
But the program still terminates very quickly, with average execution time 3.06 seconds for each image.
I wonder why the verification process still terminates early with 2000 seconds timeout?
Thanks for your help!

Best regards,
Veronica

[huanzhang12]

@JacksonZyy Can you provide an exact command and config file so we can try to replicate the problem?

[JacksonZyy]

Hi,

I used the previous version of your system like 3 months ago.
The execution command is:

conda activate alpha-beta-crown
cd complete_verifier
python robustness_verifier.py --config exp_configs/mnist_conv_small.yaml

where I simply modified your original robustness_verifier.py

1. to load_model_onnx() (as defined in util.py), as I used the onnx model from ERAN website
2. Only select the first 100 images from the ERAN dataset instead of all of them

The .ymal file is updated to be

general:
  mode: verified-acc
model:
  name: mnist_conv_small
  path: models/eran/mnist_convSmallRELU__Point.onnx
data:
  dataset: MNIST_ERAN
  std: [0.30810001492500305]
  mean: [0.1307000070810318]
specification:
  epsilon: 0.11
attack:
  pgd_restarts: 100
solver:
  beta-crown:
    batch_size: 2048
    iteration: 20
bab:
  timeout: 2000
  branching:
    reduceop: max

The system printout gives me:

number of correctly classified examples: 100
incorrectly classified idx (total 0): []
attack success idx (total 16): [6, 8, 9, 11, 18, 20, 24, 33, 38, 61, 62, 65, 66, 73, 78, 92]
attack_success rate: 0.16
verification success idx (total 39): [0, 1, 2, 3, 5, 10, 13, 14, 17, 23, 25, 27, 28, 30, 31, 32, 34, 37, 39, 51, 56, 57, 60, 64, 67, 68, 70, 71, 74, 75, 79, 81, 82, 85, 86, 88, 90, 91, 97]
verification failure idx (total 0): []
final verified acc: 84.0%[100]
verifier is called on 39 examples.
total verified: 84
mean time [total:100]: 2.4678870487213134
mean time [cnt:39] (excluding attack success): 5.120762268702189
mean time [cnt:55] (including attack success): 4.487067361311479

[JacksonZyy]

The timeout works fine with the new system, even though the version before Sep release doesn't work