Fixed validation set

[svenvanderburg]

For the validation set we currently use the same data generator as for training. This means the validation set is different every epoch. A fixed validation set would make the validation loss progress during training smoother.

[florian-huber]

We could add an option to the generator. Maybe to get a random subset according to the reference score (Tanimoto) bins once in the beginning?

[svenvanderburg]

A benefit of a fixed validation set would be that the val_loss decreases more smoothly during training. Therefore the early-stopping criteria is more robust (i.e. now sometimes early stopping doesn't stop because one random set of validation data has a lower loss then the previous ones)

[svenvanderburg]

I created #77 to make provide the option of using a fixed dataset throughout a training session, but that doesn't solve the full problem.

To fully solve the problem we need to create a fixed validation set that is generated by going through the validation InChiKeys using the same data generation algorithm, but for one cycle only. I am willing to do that, but I don't have the data yet.

How to store and use the fixed validation set?

We could store the spectrum ids and the corresponding inchikeys. Assuming we keep the list with BinnedSpectrums constant we can then retreive the binned spectrum. The similarity score can be retrieved from the reference_df using the inchikeys. This could be stored in a simple .csv file.

It would be safer to store the binned spectrums themselves, with their inchikeys. I guess @florian-huber knows best how to store this?

We can create a new data generator specific for reading in this fixed dataset (should be very simple to implement)

@florian-huber let's discuss this!

[svenvanderburg]

Update: I tried to set a fixed random seed just before starting data generation in #80, but I was unsuccesful.

[florian-huber]

I think I could fix the random seed issue with #83! I believe the random seed needed to be initialized in the generator init otherwise it will only be triggered correctly when importing the generator classs.

[svenvanderburg]

Merged #83 and #80 in #77, can you review #77 @florian-huber ?