Preset functions for True/False summatives?

[jmdetect]

Fantastic idea from the insight on surface form competition, and it does solve the problem well. It's impressive indeed.

Have you considered adding preset functions, for the case of "Does this make sense?" / "Is this truthful?" calls with True/False 1-token responses and so forth?

Right now we maintain a list of FALSE, false ... etc ... this seems much better. It seems like how we as humans psychologically process the world and are affected by similar words/multiple word senses, fascinating...

I was sad to see this must use restricted pre=specified set of tokens to rebalance, but that makes sense by necessity of reducing complexity.

[ari-holtzman]

Interesting suggestion!

I could definitely imagine this doing well, though the hope would be that PMI_DC already largely compensates for this effect by alleviating surface form competition.

Regardless, this repo in particular is maintained to reproduce precisely what we did in the paper, so we wouldn't want to diverge from the experiments we reported here.