Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider providing a BERT weight download link? #990

Closed
fierceX opened this issue Oct 28, 2019 · 6 comments
Closed

Consider providing a BERT weight download link? #990

fierceX opened this issue Oct 28, 2019 · 6 comments

Comments

@fierceX
Copy link
Member

@fierceX fierceX commented Oct 28, 2019

Now the BERT weight is downloaded via gluonnlp, but in some cases the network may not be too good or the server does not have an external network. This requires running gluonnlp to download the BERT weight, which is a bit inconvenient, can provide a BERT weight download link, or more Multiple models and dataset download pages, you can download the required weights and datasets without running gluonnlp
@dmlc/gluon-nlp-team

References

  • list reference and related literature
  • list known implementations
@fierceX fierceX added the enhancement label Oct 28, 2019
@leezu leezu added the help wanted label Oct 28, 2019
@szha

This comment has been minimized.

Copy link
Member

@szha szha commented Oct 28, 2019

The current code should still work in the described scenario because:

  • The model download logic avoids repeatedly downloading the model by checking the checksum of the model weight file at target location.
  • By specifying root argument in get_model, you can auto-download the model parameters into a location of your choosing.
@fierceX

This comment has been minimized.

Copy link
Member Author

@fierceX fierceX commented Nov 1, 2019

@szha
I know how it works now. I mean, on some servers that don't have an external network, it's a little unfriendly, mainly because you can't download weight files directly from a web page.
This requires running the code downloading weight file repeatedly on a machine with a network.

@sergeiudris

This comment has been minimized.

Copy link

@sergeiudris sergeiudris commented Nov 11, 2019

download links would allow to use models with other languages e.g. clojure/scala

@sergeiudris

This comment has been minimized.

Copy link

@sergeiudris sergeiudris commented Nov 12, 2019

[EDIT 2019-11-13]

  • downloading via get_model is not enough
  • use this guide https://gluon-nlp.mxnet.io/model_zoo/bert/index.html#export-bert-for-deployment
    • there a HINT block The pre-training, fine-tunining and export scripts are available here.
    • download export scripts, use bert/export/export.py
    • for example
      • python3 bert/export/export.py --task regression --seq_length 384
        • outputs -0000.params and -symbol.json files to output_dir

ok, a solution :

import gluonnlp as nlp; import mxnet as mx;
model, vocab = nlp.model.get_model('bert_12_768_12', dataset_name='book_corpus_wiki_en_uncased', use_classifier=False, use_decoder=False);
tokenizer = nlp.data.BERTTokenizer(vocab, lower=True);
transform = nlp.data.BERTSentenceTransform(tokenizer, max_seq_length=512, pair=False, pad=False);
sample = transform(['Hello world!']);
words, valid_len, segments = mx.nd.array([sample[0]]), mx.nd.array([sample[1]]), mx.nd.array([sample[2]]);
seq_encoding, cls_encoding = model(words, segments, valid_len);
  • create a script, e.g. bert.py with the code from above
  • install dependencies and run
# install dependencies
sudo apt-get update
sudo apt-get install -y python3 python3-pip
pip3 install --user gluonnlp mxnet

# run the script
python3 bert.py
  • the BERT model will be downloaded to ~/.mxnet/models
    • for example /root/.mxnet/models or /home/user/mxnet/models
    • files
      • bert_12_768_12_book_corpus_wiki_en_uncased-75cc780f.params
      • book_corpus_wiki_en_uncased-a6607397.vocab
  • files can be used with clojure/scala or other mxnet languages

it is obvious, of course. nevertheless..

@eric-haibin-lin

This comment has been minimized.

Copy link
Member

@eric-haibin-lin eric-haibin-lin commented Nov 13, 2019

So the use case is about easier ways to download / distribute model weights without setting up gluon-nlp? That sounds reasonable.

Adding links to those parameter weight on the website is not hard. We need exact effort to maintain the documentation everytime the link changes but since model weight does not change frequently, it should be fine. @szha maybe we should include the link in the upcoming model catalog page?

@eric-haibin-lin

This comment has been minimized.

Copy link
Member

@eric-haibin-lin eric-haibin-lin commented Dec 18, 2019

will track the issue in #990

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.