Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A potential issue that might affect comparison fairness #21

Open
aiyolo opened this issue Dec 20, 2020 · 2 comments
Open

A potential issue that might affect comparison fairness #21

aiyolo opened this issue Dec 20, 2020 · 2 comments

Comments

@aiyolo
Copy link

aiyolo commented Dec 20, 2020

rfs/eval_fewshot.py

Lines 126 to 131 in f8c837b

meta_testloader = DataLoader(MetaCIFAR100(args=opt, partition='test',
train_transform=train_trans,
test_transform=test_trans,
fix_seed=False),
batch_size=opt.test_batch_size, shuffle=False, drop_last=False,
num_workers=opt.num_workers)

In experiment, the flag of num_workers=3 in meta_testloader, it will assign 3 threads to fetch data, however, these threads have the same seed for np.random, which means it will sample 3 similar task, the 3 task has the same class, the same support images and the same query images, so it might be unfair to other methods although it is irrelevant to the performance.
By the way, setting worker_it_fn for each thread or num_workers<=1 might be a remedy.

@Jiawen-huang
Copy link

I solve it.
Just delete this sentence:
lambda x: np.asarray(x),

@aiyolo
Copy link
Author

aiyolo commented Dec 20, 2020

I don't think so, the line you commented maps x to a np.Array, which has nothing to with a random seed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants