How to continue train ALBERT from the modelreleased on the tfhub ?

[wxp16]

Training from scratch is very expensive. Anybody know how to continue train ALBERT from the exported model..

Thanks

[0x0539]

Note that the TF-Hub checkpoints do not contain LAMB optimizer momentum estimates, so it won't be exactly the same as resuming from a training checkpoint (which we do plan to release at some point). In particular, the momentum estimates will be reset to zero.

That said, it's still probably faster than starting from a randomly-initialized ALBERT :)

If you want to try it, I can think of two ways:

1. Extract the checkpoint files manually from the TF-Hub module. Look inside the variables/ folder. These files can be copied into your training dir as the .data and .index files of a checkpoint. You may need to create a file called checkpoint that points to them (look inside an ALBERT training directory for the format).
2. Add a flag to run_pretraining.py for initializing from TF-Hub. It should be similar to this recent commit, which did the same thing but for run_classifier.py. If you want to put together a PR for this, that'd really be helpful.

[0x0539]

There's an easier way to get these checkpoints now. See the [Tar file] links in the readme.