Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Article: Quality evaluation #86

Open
cefoo opened this issue Nov 30, 2021 · 7 comments
Open

Article: Quality evaluation #86

cefoo opened this issue Nov 30, 2021 · 7 comments
Labels
article core good first issue Good for newcomers help wanted Extra attention is needed quality

Comments

@cefoo
Copy link
Collaborator

cefoo commented Nov 30, 2021

No description provided.

@cefoo cefoo added good first issue Good for newcomers help wanted Extra attention is needed quality labels Nov 30, 2021
@bittlingmayer
Copy link
Collaborator

Automated

With reference translations

  • BLEU
  • LEPOR
  • F-score
  • METEOR
  • NIST
  • Word error rate
  • COMET

Without reference translations

  • Round-trip translation
  • Vector similarity (LASER, BERT etc)
  • ModelFront

Human

  • Labelling (a.k.a. direct assessment)
  • Post-editing distance
  • Post-editing rate (a.k.a. approval rate)

@bittlingmayer
Copy link
Collaborator

COMET is a special case, it's like quality estimation AND using reference translation as a hint.

@cefoo cefoo added the core label Dec 6, 2021
@cefoo
Copy link
Collaborator Author

cefoo commented Dec 6, 2021

These are worth turning into individual pages each, right?

@cefoo cefoo changed the title Quality evaluation Article: Quality evaluation Jan 17, 2022
@cefoo cefoo added the article label Jan 18, 2022
@cefoo cefoo mentioned this issue Mar 4, 2022
Closed
@lena-voita
Copy link
Contributor

Is anyone working on this? I could take this one

@cefoo
Copy link
Collaborator Author

cefoo commented Apr 7, 2022

Sure, you can work on this!
Please check out our Style Guide for reference at the time of writing.
Thanks so much! 😄

@lena-voita
Copy link
Contributor

  1. Is there an empty page for this?
  2. It's supposed to be two separate pages: Quality Evaluation and Quality Estimation, right?

@cefoo
Copy link
Collaborator Author

cefoo commented Apr 7, 2022

1- We recommend that you follow the structure of other articles within the same directory. In this case, it would be Quality estimation.

2- Yes, exactly.

Tip: Keep it simple and short, and focus on the target audience - the marketplace/real cases. 👩‍💻

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
article core good first issue Good for newcomers help wanted Extra attention is needed quality
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@bittlingmayer @lena-voita @cefoo