-
Notifications
You must be signed in to change notification settings - Fork 969
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Crazy idea about reusing pre-trained char-based RNNLM #747
Comments
Yes, using pretrained language model embeddings is a very standard practice. |
Mind shedding a light on how to implement it with torch and/or nn.LookupTable? :) Ps. After a quick scan I didn't see any api to read out or load weights into LookupTable. |
@soumith I hacked together a modified version of The goal here is to later use that
|
@coodoo one usually simply copy the weights from a pre-trained model. For example, old_lookup = torch.load(...) -- can be a full model
lookup = nn.LookupTable(...)
lookup.weight:copy(old_lookup.weight)
-- add lookup in your network Another way (if you only want to finetune the last layer for example) is to load the full network, removing the last layer and adding a new randomly-initialized layer on top of the network. net = torch.load(...)
net:remove() -- remove last layer
net:add(nn.Linear(4096,21)) -- add new layer replacing the old one But your approach also seems to work, but would require adding an extra entry to every module that one wants to reuse the weights. |
@fmassa Thanks for the detailed instructions, obviously those are much better ways to do the job, appreciate it! 😀 |
Verified the approach is working and uploaded an updated version here, @fmass thanks again for your kind instructions, those were very helpful! |
This might be a crazy idea, normally we have limited amount of labelled data set to train a char-based
RNN LM
for sentiment analysis, I'm wondering would it be possible to train aRNNLM
with large data set like Penn Tree Bank or billion words from wikipedia first, then base on that model (possibly as ann.LookupTable
) to further train with our labelled data for sentiment analysis task?Has anyone done this before?
The text was updated successfully, but these errors were encountered: