-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix negative sampling RNGs. #39
Conversation
I'll wait with merging until the fix is verified through trained models. |
During refactoring, the ZipfRangeGenerator was unintentionally replaced by an RNG. This commit undoes this unintended change.
Well, there was yet another bug to be squashed. I squashed and pushed the fix already. let negative = match self.skipgram_config.model {
ModelType::StructuredSkipGram => {
let context_size = self.skipgram_config.context_size as usize;
let offset = output % (context_size * 2);
let rand_type = self.range_gen.next().unwrap();
// in structured skipgram the offset into the output matrix is calculated as:
// (vocab_idx * context_size * 2) + offset
rand_type * context_size * 2 + offset
}
ModelType::SkipGram => self.range_gen.next().unwrap(),
}; But I guess that would've drawn a negative sample with the same offset as the original context. |
Weird, that seems to originate from before |
I'm suspecting that I copy-pasted that from one of my working branches. The skipgram model is done and is now only |
Structgram is also done now, the newly trained model is 0.47% below the pretrained model at 59.53% vs. 60.10%. Do you think this difference can be due to random factors? |
Hard to tell. I did see some variance between runs. I think 0.57% is within the realm of possibilities. Maybe you could train a basic topo model with both structgram models to verify that the results are the same for a downstream task? (Of course, it may not give the best result as with zipf distribution parameter hacking, but it's a good extra sanity check.) |
New embeddings on val 98.19 vs. old embeddings on val 98.08. Which is actually a new over-all highscore for our topo experiments. Anyways, I think this still confirms that we are getting comparable embeddings out of the new release. The .1 difference could be due to random effects in toponn or due to random effects in finalfrontier, I don't think it's possible to figure that out. |
On Thu, Apr 25, 2019, at 15:39, Sebastian Pütz wrote:
New embeddings on val 98.19 vs. old embeddings on val 98.08.
Which is actually a new over-all highscore for our topo experiments.
Anyways, I think this still confirms that we are getting comparable
embeddings out of the new release. The .1 difference could be due to
random effects in toponn or due to random effects in finalfrontier, I
don't think it's possible to figure that out.
Ship it! Erm, I mean, merge it!
|
During refactoring, the ZipfRangeGenerator was unintentionally
replaced by an RNG. This commit undoes this change.