When is dictionaries.bin created?

[svkeerthy]

On downloading the trained model, I get dictionaries.bin. However, on training the network with preprocessed data from https://s3.amazonaws.com/code2vec/data/java14m_data.tar.gz, I'm unable to generate dictionaries.bin at the end of the training.

Because of this, I am unable to use Interactive predict for manual examination of the model. Am I missing something here? Please let me know at which phase, dictionaries.bin would be generated.

[urialon]

Hi,
The file dictionaries.bin is supposed to be saved every time we save the model, i.e., at the end of every epoch. We have recently made some major refactoring which might have broken it.
As an immediate step, you can try going back to commit: 9789cfa

@eladn - if I understand correctly, there is no call to self.save from tensorflow_model.py nor from model_base.py. The function save_inner_model(..) is only saving the model weights but not the dictionaries. Can you take a look?

[svkeerthy]

Hi,
Cool! Thanks.
Then, calling self.save, after the call to _save_inner_model in tensorflow_model.py, hopefully would be a quick fix. Please correct me if I am wrong.

[urialon]

Right,
but it is even better saving the dictionaries after every epoch here, because maybe you would want to stop training manually before the model stops itself.
Maybe it will be best to save the dictionaries even before starting training, right after creating the vocabularies here.

You can change that locally as a quick fix. I am waiting for @eladn 's response and we'll push a fix.
Sorry for the inconvenience and thanks for letting us know.

[haldos]

Hi,
Cool! Thanks.
Then, calling self.save, after the call to _save_inner_model in tensorflow_model.py, hopefully would be a quick fix. Please correct me if I am wrong.

This option works fine for me.

[urialon]

Great! I'm glad to hear. Let me know if you have any other questions.

[eladn]

Thanks for letting us know. I made a PR fixing this issue as suggested above.