Fix negative sampling RNGs.

[sebpuetz]

During refactoring, the ZipfRangeGenerator was unintentionally
replaced by an RNG. This commit undoes this change.

[sebpuetz]

I'll wait with merging until the fix is verified through trained models.

[sebpuetz]

Well, there was yet another bug to be squashed. I squashed and pushed the fix already.
Structgram panicked because of out of bounds indices from negative samples. I can't remember why I put this code there:

            let negative = match self.skipgram_config.model {
                ModelType::StructuredSkipGram => {
                    let context_size = self.skipgram_config.context_size as usize;
                    let offset = output % (context_size * 2);
                    let rand_type = self.range_gen.next().unwrap();
                    // in structured skipgram the offset into the output matrix is calculated as:
                    // (vocab_idx * context_size * 2) + offset
                    rand_type * context_size * 2 + offset
                }
                ModelType::SkipGram => self.range_gen.next().unwrap(),
            };

But I guess that would've drawn a negative sample with the same offset as the original context.

[danieldk]

Weird, that seems to originate from before BandedRangeGenerator.

[sebpuetz]

Weird, that seems to originate from before BandedRangeGenerator.

I'm suspecting that I copy-pasted that from one of my working branches.

The skipgram model is done and is now only 0.2% below the pretrained model. 56.94% vs. 57.20%. Structgram will take a while, I didn't see that it crashed until last night.

[sebpuetz]

Structgram is also done now, the newly trained model is 0.47% below the pretrained model at 59.53% vs. 60.10%. Do you think this difference can be due to random factors?
I won't merge this PR until you green light that.

[danieldk]

Structgram is also done now, the newly trained model is 0.47% below the pretrained model at 59.53% vs. 60.10%. Do you think this difference can be due to random factors?

Hard to tell. I did see some variance between runs. I think 0.57% is within the realm of possibilities. Maybe you could train a basic topo model with both structgram models to verify that the results are the same for a downstream task?

(Of course, it may not give the best result as with zipf distribution parameter hacking, but it's a good extra sanity check.)

[sebpuetz]

New embeddings on val 98.19 vs. old embeddings on val 98.08.

Which is actually a new over-all highscore for our topo experiments.

Anyways, I think this still confirms that we are getting comparable embeddings out of the new release. The .1 difference could be due to random effects in toponn or due to random effects in finalfrontier, I don't think it's possible to figure that out.

[danieldk]

On Thu, Apr 25, 2019, at 15:39, Sebastian Pütz wrote: New embeddings on val 98.19 vs. old embeddings on val 98.08. Which is actually a new over-all highscore for our topo experiments. Anyways, I think this still confirms that we are getting comparable embeddings out of the new release. The .1 difference could be due to random effects in toponn or due to random effects in finalfrontier, I don't think it's possible to figure that out.

Ship it! Erm, I mean, merge it!