Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.Sign up
Fixing random seeds does not yield deterministic results #412
It has been observed by Erutan-pku, @hbsun2113 , me and @yzh119 that after fixing NumPy random seed, PyTorch random seed and
In the case of SGC, the function
referenced this issue
Feb 25, 2019
Yes to both.
Think of a linear logistic regression. If we swap the positive and negative labels, the optimal parameters would switch their signs as well. This implies that, given everything the same (including initial parameters) except that we permute the labels, the trajectory of parameter updates would still be different.
Now that we have a complicated non-convex model, we know that a lot of local minima exist. Since the trajectory of parameter updates differ if we permute the labels, it is natural that the model performance would be (slightly) different between runs. This also naturally extends to multi-class classification.