Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incorrect preprocessing for ImageNet-C evaluation #59

Closed
imrahulr opened this issue Jan 18, 2022 · 2 comments
Closed

Incorrect preprocessing for ImageNet-C evaluation #59

imrahulr opened this issue Jan 18, 2022 · 2 comments

Comments

@imrahulr
Copy link

imrahulr commented Jan 18, 2022

I see that the ImageNet-C evaluation uses the preprocessing: Resize(256)+CenterCrop(224)+ToTensor().

def load_imagenetc(
n_examples: Optional[int] = 5000,
severity: int = 5,
data_dir: str = './data',
shuffle: bool = False,
corruptions: Sequence[str] = CORRUPTIONS,
prepr: str = 'Res256Crop224'
) -> Tuple[torch.Tensor, torch.Tensor]:
transforms_test = PREPROCESSINGS[prepr]

This causes discrepancies with the scores reported in the original papers (DeepAugment, AugMix, Standard RN-50). The ImageNet-C dataset already contains 224x224 images and hence only ToTensor() should be used for consistency.

Fixing prepr='none' in load_imagenetc should solve the issue (assuming all the models are capable of handling 224x224 images as input).

@max-andr
Copy link
Member

The suggestion of fixing prepr='none' in load_imagenetc definitely makes sense for the current models! We'll look into this!

@max-andr
Copy link
Member

max-andr commented May 9, 2022

Sorry, took us a long time. Should be fixed with this PR: #85.

@max-andr max-andr closed this as completed May 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants