Contest Proposal and infrastructure for the Unrestricted Adversarial Examples Challenge
Switch branches/tags
Nothing to show
Clone or download
Latest commit 5bc84d4 Sep 20, 2018

README.md

Unrestricted Adversarial Examples Challenge Build Status

In the Unrestricted Adversarial Examples Challenge, attackers submit arbitrary adversarial inputs, and defenders are expected to assign low confidence to difficult inputs while retaining high confidence and accuracy on a clean, unambiguous test set. You can learn more about the motivation and structure of the contest in our recent paper

This repository contains code for the warm-up to the challenge, as well as the public proposal for the contest. We are currently accepting defenses for the warm-up.

image

Leaderboard for the warm-up to the contest

We include three attacks in the warm-up to the contest:

The top few distinct models for each dataset are shown below. You can see all submissions in the full scoreboard.

Two-Class MNIST dataset

Defense Submitted by Clean data Spatial grid attack SPSA attack Boundary attack Submission Date
MadryPGD LeNet Baseline Google Brain 100.0% 0% 0% 0% Sept 14th, 2018
Undefended LeNet Baseline Google Brain 100.0% 0% 19.6% 0% Sept 14th, 2018

All percentages above correspond to the model's accuracy at 80% coverage.

Bird or Bicycle dataset

Defense Submitted by Clean data Spatial grid attack SPSA attack Boundary attack Submission Date
Keras ResNet
(via ImageNet)
Google Brain 100.0% ?? ?? ?? In progress
Pytorch ResNet
(via bird-or-bicycle extras)
Google Brain 99.5% 45.2% 12.8% ?? Sept 13th, 2018

All percentages above correspond to the model's accuracy at 80% coverage.

Submitting a defense for the warm-up

The warm-up before the contest is currently underway and is accepting submissions. If you have additional questions, feel free to submit a new GitHub issue with the "question" tag and we will respond shortly.

The contest

The contest phase will begin after the warm-up attacks have been conclusively solved. We have published the contest proposal and are soliciting feedback from the community.

Paper

You can learn more about the motivation and structure of the contest in our recent paper:

Unrestricted Adversarial Examples
Tom B. Brown, Nicholas Carlini, Chiyuan Zhang, Catherine Olsson, Paul Christiano and Ian Goodfellow
[Arxiv paper preprint]