-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Estimated standard deviations calculation #4
Comments
Hey Tal, I dug into the logs of my previous experiments, and I found that I was using 1 model for each subrun. To replicate my calculation of standard deviation, Those arguments were added while I was cleaning the code after the experiment have finished, and I apologize for the inconsistency. To calculate the standard deviation of the estimated mean, I used the following formula |
Hi, |
Yes, that is correct. |
I am closing this issue since it has been resolved. |
Hi, Thanks! |
Hi,
At table 2 in the paper you say: " We also show the estimated standard deviations of the averages computed over 175 random data split and training seeds ".
But 'mnist_guess.yaml' parameters are: target_model_count=200 target_model_count_subrun=10 , which means that for each combination of (cur_num_samples, cur_loss_bin) there are 20 records in the Data Base (each record contain test_acc that was calculated over 10 models accuracies) that is used for the standard deviation calculation. So for my understanding, the standard deviation is calculated over 10 random data split and training seeds.
I would appreciate if you could clarify this issue for me, to make sure that i understand the way you calculated the standard deviations.
Thanks,
Tal
The text was updated successfully, but these errors were encountered: