Getting Sentence level log probabilities using this model

[Shashi456]

So I was trying to implement the SLOR score, using the transformer-XL, mostly avoiding training. But given the XL model and the sentence, how could i go about getting the sentence level log probability?

Attached is the formula I'm trying to implement.

Thanks a lot for your help.

[thomwolf]

So p_M(S) is just the output of the model right?

For p_u(S), I think the easiest is probably to use the empirical probabilities.
TransfoXLTokenizer has a counter to store words frequencies here which should be populated in the "pretrained" tokenizer so I would use and normalize this to get unconditional probabilities for each word and then compute SLOR.

[Shashi456]

@thomwolf So the code would look something like :

s_ids = convert_sentence_to_ids(sentence)
config = TransfoXLConfig()
model = TransfoXLModel(config)
last_hidden_state, new_mems = model(input_ids)
for i in sentence : 
     uni_prob =  freq(token)/no_of_tokens
     sent_uni_prob * = uni_prob 
SLOR = (1/len(sentence))* last_hidden_state - sent_uni_prob 

or am i mistaken about the idea that the last_hidden_state is the sentence probability i'm looking for

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[davebulaval]

In case someone found this, I found an implementation that seems legit here sgondala/GoogleConceptualCaptioning@981e1b6.