Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

user scoring and question selection #57

Merged
merged 7 commits into from
Dec 26, 2022

Conversation

MattAlexMiracle
Copy link
Collaborator

This implements an early version of #37 using the following systems:

  • We use "information gain" to model how much a prompt's rating could benefit from new votes
  • We score votes based on how close they were to the optimal vote, i.e. if the consensus produces [100,300,200] and you vote for index 2 (counting from 0), you would get 1 point, since the ranking is [0,2,1] as 300 is the biggest, 200 the second, and 100 the smallest value.
  • We score prompts using "positive edge": i.e. if your votes are [200,300,100,500], you would multiply it with the (signed) distance to the middle, i.e. [-1,0,1,2] and get $-1\cdot 200+0\cdot 300+1\cdot 100+2\cdot 500$ points
  • We then score rankings using the kendall-tau correlation (or "normalized bubble sort distance"): this compares the combined ranking from all rankers to the ranking an individual user produced, and gives points based on how similar they are.

All three point scoring methods (voting, ranking, and prompting) have their own point counts, since the scales are very different. How we weight them between each other is going to have to be experimental, as it also depends on many external factors, such as "how many votes per question do we target".

Copy link
Collaborator

@yk yk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you

if I understand correctly, a "vote" is a best-one-of-N judgement. Is there also a variant if each user provides a complete ranking of all N choices? I'm thinking if we already make the user read all of the options, they might as well rank all of them.

@andreaskoepf andreaskoepf merged commit f8c3008 into LAION-AI:main Dec 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants