How to fine tune the existing weights on new data ?

[asiddhant]

I converted the hdf5 file back as a ckpt file (using the custom_getter method in bilm/model.py) and tried to use it with architecture in bilm/training.py but the loaded weights give very bad perplexity on heldout data when I do run_test.py. Are the architectures in bilm/model.py and bilm/training.py compatible. If you feel I m doing something wrong, is it possible for you to share the ckpt file of the given hdf5 file.

Thanks

[matt-peters]

The architectures are the same. However, the hdf5 file doesn't contain the softmax weights as they aren't needed to compute the ELMo representations, and they significantly increase the size of the file. I'll make the checkpoint file for the pre-trained model with the softmax weights available shortly.

[asiddhant]

Thanks a lot. I missed that part. Yeah it would be great to have ckpt file as well but since vocab will anyways be new, pretrained softmax would not be required for fine tuning as well. So I am closing this issue.

[asiddhant]

Hi would it be possible for you to share the checkpoint file as well?

[matt-peters]

The checkpoint file is available: see https://github.com/allenai/bilm-tf/blob/master/README.md#can-you-provide-the-tensorflow-checkpoint-from-training