Evaluation script for task 1 of WMT2016 QE shared task.

[kepler]

• Includes a much more efficient DeltaAvg method (the Perl version takes 16s, while this Python version takes less than 1s).
• Accepts files in either the tab-separated submission format or just with HTER scores.
• Various predicted/submitted files can be passed at once, and the script will print ordered results for both scoring and ranking metrics.

[varvara-l]

Thanks for the contribution!
I'm afraid though that this script is a bit irrelevant for this particular project -- Marmot is targeted at subsentence-level QE (task 2 of WMT16), not sentence-level QE.
I think this script should be more relevant for QuEst++: https://github.com/ghpaetzold/questplusplus

[kepler]

No problem. I had sent the script as a gist to Lucia and Chris just for you
guys to think about using it in next year's shared task. Chris then
suggested a pull request to Marmot.

Either way, I'll leave the script here:
https://gist.github.com/kepler/6043a41ed8f3ed0be1e68c5942b99734

A qua, 8/06/2016, 09:52, varvara-l notifications@github.com escreveu:

Thanks for the contribution!
I'm afraid though that this script is a bit irrelevant for this particular
project -- Marmot is targeted at subsentence-level QE (task 2 of WMT16),
not sentence-level QE.
I think this script should be more relevant for QuEst++:
https://github.com/ghpaetzold/questplusplus

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#34 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AAEgHXDg7FR59RhPRYOQv07jdClpzWUOks5qJoKwgaJpZM4IwZS1
.

[chrishokamp]

I'm merging because our original ideas was to support all levels of qe, and
the evaluation script itself doesn't break any other code.
On Jun 8, 2016 2:04 AM, "Fabio Natanael Kepler" notifications@github.com
wrote:

No problem. I had sent the script as a gist to Lucia and Chris just for you
guys to think about using it in next year's shared task. Chris then
suggested a pull request to Marmot.

Either way, I'll leave the script here:
https://gist.github.com/kepler/6043a41ed8f3ed0be1e68c5942b99734

A qua, 8/06/2016, 09:52, varvara-l notifications@github.com escreveu:

Thanks for the contribution!
I'm afraid though that this script is a bit irrelevant for this
particular
project -- Marmot is targeted at subsentence-level QE (task 2 of WMT16),
not sentence-level QE.
I think this script should be more relevant for QuEst++:
https://github.com/ghpaetzold/questplusplus

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#34 (comment), or
mute
the thread
<
https://github.com/notifications/unsubscribe/AAEgHXDg7FR59RhPRYOQv07jdClpzWUOks5qJoKwgaJpZM4IwZS1

.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#34 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/ABicP8LBwijwbaRhDdP9yas729Lp_Vk4ks5qJoWIgaJpZM4IwZS1
.