Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Important data leakage in resnet example. #21

Closed
Cam-B04 opened this issue Aug 22, 2022 · 1 comment
Closed

Important data leakage in resnet example. #21

Cam-B04 opened this issue Aug 22, 2022 · 1 comment

Comments

@Cam-B04
Copy link
Contributor

Cam-B04 commented Aug 22, 2022

Hi !

First thank you very much for this repo, it is very helpful for people who are new to BNNs like me.

I have started to use TyXe for a convolutional BNN, starting from your resnet.py example.
After getting some unexpected behavior during training I have noticed that in the callback function the network was not set into evalutation mode. It is resulting in an important data leakage by training the network on the test dataset as well if I am correct.

I would recommend to start and finish the callback function with respectively b.eval() and b.train().

Hope this helps,
Regards!

Cam-B04 added a commit to Cam-B04/TyXe that referenced this issue Aug 22, 2022
…uring

callback and consequently avoid dataleakage.
@hpplyt
Copy link
Collaborator

hpplyt commented Sep 8, 2022

Thanks spotting this and the PR! I don't think it's overkill at all, I'll merge it right away, sorry for only getting back to you now.

Technically I wouldn't really call this data leakage since training won't be affected by the test data, but it does break independence between the test predictions, so I definitely should have included the eval and train calls :)

@hpplyt hpplyt closed this as completed in 368bf62 Sep 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants