How do you calculate Efficacy Score for multi-token words in your counterfact dataset?

[Zce1112zslx]

Efficacy score can be calculated by P[o] and P[o*]. The object o can sometimes be tokenized to several sub-words (tokens) and in this case how do you calculate P[o]?

[kmeng01]

Hi! You can see how we do this in the eval code. tl;dr we multiply the probabilities, which is equivalent to summing logprobs: $$p(o) = \prod_{i \in o} p(w_i \mid w_{j < i}) = \exp \left( \log \prod_i p(w_i \mid w_{j < i}) \right) = \exp \left( \sum_i \log p(w_i \mid w_{j < i}) \right).$$

You'll notice that, in practice, we use negative log probabilities: $$p(o) = \exp \left( -\sum_i -\log p(w_i \mid w_{j < i}) \right)$$

Given $p(o), p(o^*)$, we can perform a direct comparison.