You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Thanks for sharing the code. I have a question about score calculation.
In doc function, the representation of the document is multiplied with the mask.
However, could the max(2) in the score function accidentally choose the padding token score (which score is 0) if all the non-padding token scores are negative?
I asked because I check the implementation here and found they assign a large negative value score[~exp_mask] = - 10000 before search for the max score.
The text was updated successfully, but these errors were encountered:
That's indeed a good edge case to keep in mind — thanks! Since we care about the largest/max similarities per query token for the high-scoring documents, these are practically always positive. But we might want to enforce this more directly with a ReLU before the max sim.
It is based on your similarity function. L2 is always positive, but cosine similarity could be negative. I agree with you that all negative similarities are definitely a concern case.
Hi
Thanks for sharing the code. I have a question about score calculation.
In doc function, the representation of the document is multiplied with the mask.
However, could the
max(2)
in the score function accidentally choose the padding token score (which score is 0) if all the non-padding token scores are negative?I asked because I check the implementation here and found they assign a large negative value
score[~exp_mask] = - 10000
before search for the max score.The text was updated successfully, but these errors were encountered: