New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
using BERT as a language Model #37
Comments
I don't think you can do that with Bert. The masked LM loss is not a Language Modeling loss, it doesn't work nicely with the chain rule like the usual Language Modeling loss. |
Hello @thomwolf I can see it is possible to assign score by using BERT . By masking each word sequentially. Then score sentence by summary of word score. Here is how people were doing it for Tensorflow. I am trying to do following
Is it the right way to assign score using BERT? |
no, you masked word but not restore. |
update default adversarial training config for roberta question answe…
@mdasadul Did you managed to do it? |
Yes please check my tweet on this @mdasaduluofa
…On Wed, May 27, 2020, 1:37 PM orko19 ***@***.***> wrote:
@mdasadul <https://github.com/mdasadul> Did you managed to do it?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#37 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB5DO5N2MGF6QCTAZ3L3NITRTS7J3ANCNFSM4GFFKJJA>
.
|
@mdasadul Do you mean this one? |
It should be similar. Following code is for distilBert
|
@mdasadul I get the error:
|
The score is equivalent to perplexity. Hence lower the score better the sentence, right? |
Yes that is right
Md Asadul Islam
Machine Learning Engineer
Scribendi Inc
…On Mon, Jul 6, 2020 at 11:54 PM nlp-sudo ***@***.***> wrote:
The score is equivalent to perplexity. Hence lower the score better the
sentence, right?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#37 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB5DO5KTBQJEEM7J72TCH2LR2K2AVANCNFSM4GFFKJJA>
.
|
@mdasadul I get the error:
Any idea why? |
Yes, your sentence needs to be longer than 1 word. PPL of 1 word sentence
doesn't mean anything. Please try with longer sentences
Md Asadul Islam
Machine Learning Engineer
Scribendi Inc
…On Sun, Mar 14, 2021 at 7:48 AM orenschonlab ***@***.***> wrote:
@mdasadul <https://github.com/mdasadul> I get the error:
return math.exp(loss.item() / len(tokenize_input))
ValueError: only one element tensors can be converted to Python scalars
Any idea why?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#37 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB5DO5ITYG7M6TG2XV5NZ6LTDSPARANCNFSM4GFFKJJA>
.
|
@mdasadul I have a sentence with more than 1 word and still get the error |
Below is an example from the official docs on how to implement GPT2 to determine perplexity. |
@EricFillion But how can it be used for a sentence, not for a dataset? |
@orenschonlab Try below
|
I just played around with the code @mdasadul posted above. It works perfectly and is nice and concise. It outputted the same scores from the official documentation for short inputs. If you're still interested in using the method from the official documentation, then you can replace "'\n\n'.join(test['text'])" with the text you wish to determine the perplexity of. You'll also want to add ".item()" to ppl to convert the tensor to a float. |
This repo is quite useful. It supports Huggingface models. |
* Add support for `Blenderbot` models Closes huggingface#37 References huggingface#29 * Add support for `BlenderbotTokenizer` * Add blenderbot to supported models * Add support for `BlenderbotSmallTokenizer` * Add custom tests for blenderbot-small * Add support for `BlenderbotSmall` models * Update list of supported models * Improve `addPastKeyValues` function * Allow skipping of adding encoder past key values
I was trying to use BERT as a language model to assign a score(could be PPL score) of a given sentence. Something like
P("He is go to school")=0.008
P("He is going to school")=0.08
Which is indicating that the probability of second sentence is higher than first sentence. Is there a way to get a score like this?
Thanks
The text was updated successfully, but these errors were encountered: