Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixing random seeds does not yield deterministic results #412

Closed
mufeili opened this Issue Feb 25, 2019 · 8 comments

Comments

Projects
None yet
4 participants
@mufeili
Copy link
Member

mufeili commented Feb 25, 2019

It has been observed by Erutan-pku, @hbsun2113 , me and @yzh119 that after fixing NumPy random seed, PyTorch random seed and cudnn.deterministic=True we still get different results across runs. There can be two possibilities:

  1. This is a bug.
  2. Some system optimization involves randomness, in which case we should provide APIs for users to fix results.
@BarclayII

This comment has been minimized.

Copy link
Collaborator

BarclayII commented Feb 25, 2019

In the case of SGC, the function python/dgl/data/citation_graph.py:_encode_onehot does not necessarily return the same output from the same input, since it is enumerating a set.

Changing from classes = set(labels) to classes = list(sorted(set(labels))) solves the problem.

@mufeili

This comment has been minimized.

Copy link
Member Author

mufeili commented Feb 26, 2019

@BarclayII In that case we may want to add an option for our API to decide if a deterministic split is to be employed. Could you also please check GraphSAGE as mentioned by @hbsun2113 ?

@BarclayII

This comment has been minimized.

Copy link
Collaborator

BarclayII commented Feb 26, 2019

Works for me.

@BarclayII

This comment has been minimized.

Copy link
Collaborator

BarclayII commented Feb 26, 2019

@BarclayII In that case we may want to add an option for our API to decide if a deterministic split is to be employed. Could you also please check GraphSAGE as mentioned by @hbsun2113 ?

Also it's not related to splitting the dataset, but encoding the labels into one-hot vectors.

@jermainewang jermainewang added the bug label Feb 26, 2019

@hbsun2113

This comment has been minimized.

Copy link
Contributor

hbsun2113 commented Feb 26, 2019

Changing from classes = set(labels) to classes = list(sorted(set(labels))) solves the problem.

BarclayII hi! I wonder why we need sort the labels?

@mufeili

This comment has been minimized.

Copy link
Member Author

mufeili commented Feb 26, 2019

@hbsun2113 Sets are unordered and you can get different results across tries. To sort it can solve the issue.

@hbsun2113

This comment has been minimized.

Copy link
Contributor

hbsun2113 commented Feb 26, 2019

Sets are unordered and you can get different results across tries. To sort it can solve the issue.

@mufeili Encoding label to different onehot leads to different results?
I think we only need call _encode_onehot once during training.

@BarclayII

This comment has been minimized.

Copy link
Collaborator

BarclayII commented Feb 26, 2019

@mufeili Encoding label to different onehot leads to different results?
I think we only need call _encode_onehot once during training.

Yes to both.

Think of a linear logistic regression. If we swap the positive and negative labels, the optimal parameters would switch their signs as well. This implies that, given everything the same (including initial parameters) except that we permute the labels, the trajectory of parameter updates would still be different.

Now that we have a complicated non-convex model, we know that a lot of local minima exist. Since the trajectory of parameter updates differ if we permute the labels, it is natural that the model performance would be (slightly) different between runs. This also naturally extends to multi-class classification.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.