-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add 'Both Bad' button to judging #13
Comments
The question is how we should alter the score when someone clicks the On Tue, Sep 2, 2014 at 3:48 PM, Ondrej Bojar notifications@github.com
|
We should reduce the score for both and log separately, e.g. in one more column in the database, that each of the candidates got one more 'bad mark'. In the end, we should know how many such disqualifying marks did each sentence get. ----- Original Message -----
Ondrej Bojar (mailto:obo@cuni.cz / bojar@ufal.mff.cuni.cz) |
We have a table for individual scorings with a result column that is either 'a' or 'b'. We could just make it 'x' when both translations are unacceptable. Or do you mean to create a bad mark counter in the translations table? Do we also want to increment it when the other translation gets selected? |
This is a bit unfortunate since as already mentioned the Elo scoring is On Thu, Sep 4, 2014 at 9:58 AM, Ondřej Cífka notifications@github.com
|
Yes, adding 'x' for 'Both Bad' would be great. (And actually, I would consider one more button, 'Both Acceptable', e.g. 'o' for OK in the database.) As for the counters, I'd actually want to know exactly: how many wins (a), losses (b), bads (x) and possibly OKs (o) did each candidate get. If we don't add Both OK, I hope the scorers will tell us both relative knowledge (this one is better) as well as absolute knowledge (this candidate is bad, full stop). ----- Original Message -----
Ondrej Bojar (mailto:obo@cuni.cz / bojar@ufal.mff.cuni.cz) |
I dont know anything about Elo. The main reason for adding Both Bad is that when scoring, I really did not want to pick any of two bad outputs. The second reason is that (as written in another comment here) I believe this would give us also some absolute information. I suggest adding Both Bad and -- if there is no better way -- completely ignore it in Elo scoring and just use it elsewhere. E.g. I'd make use of it in the manual shutter page. ----- Original Message -----
Ondrej Bojar (mailto:obo@cuni.cz / bojar@ufal.mff.cuni.cz) |
I am not arguing against both bad button. The question is how to solve it I will hoverer argue against both OK button. We are not looking for the
|
Agreed. On the other hand, I find it essential that we still somewhere record the actual comparisons and answers we got, in the most detailed and original form, so that we can reinterpret them if we find something better. On September 4, 2014 10:41:49 AM CEST, edasubert notifications@github.com wrote:
Ondrej Bojar (mailto:obo@cuni.cz / bojar@ufal.mff.cuni.cz) |
I'm not very much in favor of having a 'virtual translation' entry in the database because I would have to change the way scoring works. :-) An entry in the scorings table, identified by a hash, is created every time a user accesses the Scorings page, and updated when the user makes a choice. This is to prevent users from resubmitting their choice. If we used the virtual translation, we would actually have to remove the entry from the database and create two new entries (with different hashes etc.)... Not a very clean solution. So I think the best solution is not to actually make the translations lose against a virtual translation, just change their score as if they did. We could also make the two translations lose against each other. That is, just calculate the 'losing' part of the Elo rating and subtract that from either translation's score. (I'm not sure how this would behave.) |
OK let's do it your way but I am in favor of pretending they lost to
|
Added a 'Both wrong' button (sounds better I think). When clicked, we make both candidates lose to a notional translation with score 1400 (this is stored in the settings as Scoring.both_bad_winner_score). |
I do not think 'both wrong' is better I would suggest something along
|
How about 'both junk'? |
It does have the proper meaning, but I would wish for something a bit more
|
Both Junk sounds best. On September 5, 2014 10:52:01 AM CEST, "Ondřej Cífka" notifications@github.com wrote:
Ondrej Bojar (mailto:obo@cuni.cz / bojar@ufal.mff.cuni.cz) |
Both Inacceptable On September 5, 2014 10:54:18 AM CEST, edasubert notifications@github.com wrote:
Ondrej Bojar (mailto:obo@cuni.cz / bojar@ufal.mff.cuni.cz) |
I would rather use ridiculous or something
|
Next to the Skip, we should also have 'Both Bad' button. This absolute scoring is very important (and different from plain pick better) because MT will be almost always bad but humans can make killer errors. If the judge understands he has to pick a better one, he would pick the better (but bad) human translation. And we would not know anything about the fact that it is suspicious.
The text was updated successfully, but these errors were encountered: