-
Notifications
You must be signed in to change notification settings - Fork 25.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
how to use extracted features in extract_features.py? #493
Comments
Without fine-tuning, BERT features are usually less useful than plain GloVe or wrd2vec indeed. See the recent study by Matthew Peters, Sebastian Ruder, Noah A. Smith (To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks) for some practical tips on that. |
thank you so much~ |
@heslowen could you please share the code for extracting features in order to use them for learning a classifier? Thanks. |
@joistick11 you can find a demo in extract_features.py |
Could you please help me?
Thanks you very much! |
@joistick11 you want to embed a sentence to a vector? |
@heslowen Hello, would you please help me? For a sequence like [cls I have a dog.sep], when I input this to Bert and get the last hidden layer of sequence out, let’s say the output is “vector”, is the vector[0] embedding of cls, vector[1] embedding of I, etc. vector[-1] embedding of sep? |
@heslowen How did you extract features after training a classifier on top of BERT? I've been trying to do the same, but I'm unable to do so. |
@rvoak I use pytorch. I did it as the demo in extract_featrues.py. it is easy to do that, you just need to load a tokenizer, a bert model, then tokenize your sentences, and then run the model to get the encoded_layers |
@RomanShen yes you're right |
@heslowen Thanks for your reply! |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
@heslowen sorry about my english, now i doing embedding for sentence task, i tuned with my corpus with this library, and i received config.json, vocab.txt and model.bin file, but in bert's original doc, can extract feature when load from ckpt tensorflow checkpoint. according to your answer, i must write feature_extraction for torch, that's right ? please help me |
@hungph-dev-ict Do you mind opening a new issue with your problem? I'll try and help you out. |
@LysandreJik Thank you for your help, I will find solution for my problem, it's use last hidden layer in bert mechanism, but if you have a better solution, can you help me ? |
How long should the extract_features.py take to complete? when using 'bert-large-uncased' it takes seconds however it writes a blank file. any advice? the code I used: !python extract_features.py |
You can look at what the BertForSequenceClassification model https://github.com/huggingface/transformers/blob/3ba5470eb85464df62f324bea88e20da234c423f/pytorch_pretrained_bert/modeling.py#L867 does in it’s forward 139. |
I extract features like examples in extarct_features.py. But went I used these features (the last encoded_layers) as word embeddings in a text classification task, I got a worse result than using 300D Glove(any other parameters are the same). I also used these features to compute the cos similarity for each word in sentences, I found that all values were around 0.6. So are these features can be used as Glove or word2vec embeddings? What exactly these features are?
The text was updated successfully, but these errors were encountered: