How to measure the effectiveness of tag "controls" (e.g. sentiment)?

[enzoampil]

I was thinking that simple methodology would be to generate sequences for sentiment spans and measure accuracy based on some overlap measure, e.g. Jaccard, or rougue.

While imperfect, this would be an initial approach to communicating the effectiveness of utilizing text generation controls, derived from pre-trained supervised models.

Tweets would be a good start.

Potential dataset

https://www.kaggle.com/c/tweet-sentiment-extraction/data

[enzoampil]

Can also try to get a pooled version of each predicted token vector based on the regular Tito Joker model, and another model that generates only positive statements. This new model should be a language model fine tuned on positive examples of a sentiment analysis dataset.

[enzoampil]

OpenAI published exactly how to do the above with IMDB reviews!!

Interesting to see that the tag controls are quite similar to ours, the main difference seems to be the use of an additional reward function for the sentiment special tokens.

https://lvwerra.github.io/trl/05-gpt2-sentiment-control/