-
Notifications
You must be signed in to change notification settings - Fork 26.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
how to get word embedding vector in GPT-2 #1458
Comments
Hi, indeed GPT-2 has a slightly different implementation than BERT. In order to have access to the embeddings, you would have to do the following: from transformers import GPT2LMHeadModel
model = GPT2LMHeadModel.from_pretrained('gpt2') # or any other checkpoint
word_embeddings = model.transformer.wte.weight # Word Token Embeddings
position_embeddings = model.transformer.wpe.weight # Word Position Embeddings |
Hi, Thank you for your reply! So if I want to get the vector for 'man', it would be like this:
Is it correct? |
Just wondering, how to transform word_vector to word? Imagine a word vector and change a few elements, how can I find closest word from gpt2 model? |
So for each token in dictionary there is a static embedding(on layer 0). You can use cosine similarity to find the closet static embedding to the transformed vector. That should help you find the word. |
Thanks. It means that for every word_vector I have to calculate vocab_size (~50K) cosine_sim manipulation. Is that right? |
I guess so. Unless you can use some property to first tighten the range. |
Ok. Three more questions, 1) is there any resource on how to generate fixed length sentence (a sentence with N words that ends with "." or "!" )? 2) what is the most effective underlying parameter for hyper-parameter tuning (eg. Temperature)? 3) Is there any slack channel to discuss these types of questions? |
about 1) I don't think that there is any. You can use Web Scraping for such specified sentences. Also, you can download a corpus and use Regex to extract desired sentences.
|
Did you succeed? I'm pursuing the same goal and I don't know how to validate my findings. I have tested some king - man + woman stuff, but it didn't work. |
How did it go? I am stuck here too. |
How did it go? |
Well, it is working. However, these weights/embeddings are "context-dependent" so one should not expect "king-queen+woman" lead to anything. |
The code already posted here is correct:
where
You can validate that this is correct by passing the embeds into the model and checking that you get the same thing as when passing in the inputs:
or even
|
it's a bit late, but might help someone, despite not being static, contextual embeddings still gave me reasonable results here
gave me:
and regarding @fqassemi 's question:
gave me reasonable result for the nearest vectors:
|
Thanks for your code @ish3lan . I assume you can extend that to include sentences not just words correct? On another note, I've been using that first code snippet and trying a bunch of different words to get different cosine similarities but all the similarities ended up being very high (>0.95). Is this normal/expected? |
❓ Questions & Help
How can we get the word embedding vector in gpt-2? I follow the guidance in bert(model.embeddings.word_embeddings.weight). But it shows that ''GPT2LMHeadModel' object has no attribute 'embeddings''.
Please help me with that. Thank you in advance.
The text was updated successfully, but these errors were encountered: