Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add app_reviews templates #148

Merged

Conversation

gchhablani
Copy link
Contributor

@gchhablani gchhablani commented Jun 9, 2021

Notes:

  • I am not sure about convert_to_star_rating. Should a model be able to generate emojis? 🤔

Edit
This is from bert-base-uncased:

>>> bt('[UNK]')
{'input_ids': [101, 100, 102], 'token_type_ids': [0, 0, 0], 'attention_mask': [1, 1, 1]}
>>> bt('🤔')
{'input_ids': [101, 100, 102], 'token_type_ids': [0, 0, 0], 'attention_mask': [1, 1, 1]}

@craffel craffel self-assigned this Jun 10, 2021
templates/app_reviews/templates.yaml Outdated Show resolved Hide resolved
reference: Given the review, generate a star rating.
d34e1413-2699-4701-baa2-05d931d012ba: !Template
id: d34e1413-2699-4701-baa2-05d931d012ba
jinja: "What would be the star rating of this review: {{review}}?
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This prompt and the one above are ambiguous - the model won't know whether you mean for it to generate the number of stars, a certain number of emojis, etc. I think you need to give the model more instructions as to which you want.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've updated the rating information for both cases. I've added emojis for the star-case, hope that is fine?

f923aeeb-1150-486d-99cd-c4ab93ed5130: !Template
id: f923aeeb-1150-486d-99cd-c4ab93ed5130
jinja:
"Would you recommend an app to a friend which has a {{star}}-star rating on a scale of 1 to 5 (1 being lowest and 5 being highest)?
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This prompt will only result in 5 unique examples - I'm not sure that's enough, I think the model would just memorize the mapping right away.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense. I will remove this template then.

9746ce4b-ac58-4dfb-9783-d77c95cb62cf: !Template
id: 9746ce4b-ac58-4dfb-9783-d77c95cb62cf
jinja:
"What would be the \u2B50-rating of this review (\u2B50 being lowest and \u2B50\u2B50\u2B50\u2B50\u2B50 being highest) : {{review}}?\n|||{% if star==5 %}\n\u2B50\
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking in to it, T5's vocab can't tokenize the star emoji:
https://colab.research.google.com/drive/1uXD0f2QdRdk9hMTIW9wF2S_R5tvDperz?usp=sharing
Why not just use * instead?

@craffel craffel merged commit 847712a into bigscience-workshop:main Jun 11, 2021
@craffel
Copy link
Contributor

craffel commented Jun 11, 2021

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants