Change Review Limit from 3 to 5 (was: Better reviewing experience)

[lissyx]

Currently, reviewing scheme is validation of three sentences. While I can understand the need to avoid pushing too much in the face of """basic""" users, when you have lots of sentences and lot of time (e.g., hackathon going on?) it's a waste of time.

At best, it should be possible to have that configured per-profile.

[MohammedBelkacem]

You mean you would like display more sentences within the view?
For instance I can review more than 3 at once by browsing to the next pages.
Is that what you mean?

[lissyx]

Yes, being able to display more than three at once.

[dabinat]

I agree, there’s a lot of unnecessary clicking involved.

But is there any particular reason why it defaults to 3? Voice clip validation is in groups of 5.

[MichaelNMaggs]

I agree. Three does make it extremely slow, especially as the ‘next page’ button is so far distant from the location of the third vote. It all seems very tiring. Allowing many more sentences per page would help a lot.

[nukeador]

Since having the number of sentences displayed variable might require more coding, we can probably get to an agreement about how many to display and then have in the backlog the possibility to change this in your profile.

@MichaelKohler what do you think? I want to make sure we do reduce our focus on the February roadmap P1s.

[dabinat]

A better long-term solution would be to simply not have the Finish Review button at all and have them submit and disappear as you press the vote buttons, with the next one appearing at the bottom automatically. Then it doesn’t matter how many there are per page.

Or make it one per page with the ability to skip, like voice validation.

[lissyx]

and have them submit and disappear as you press the vote buttons,

And if you misclick, then you can't correct :/

[dabinat]

I don’t think that’s difficult to solve. When you click a button it could say at the top “Sentence approved (undo)”. Click undo and it adds it back to the list again.

[nukeador]

Note that we are also trying to make minimal UX decisions for this tool because we hope in the future it will be integrated with the main voice-web app and we'll have proper UX resources taking care of this.

So for now what I suggest is that we stick with a number by default (it seems 5 is a good one to keep alignment with voice-web) and revisit it down the road once more people have used the tool.

Cheers.

[MichaelKohler]

Yes, let's go woth 5 for now. From a technical perspective, a patch for this should be straightforward.

[lissyx]

I think 5 looks like a better compromise for now

[MeryllEssig]

At least, some keyboard shortcuts would be cool for changing current page (left/right), and validating next unreviewed sentence (y/n).

[lissyx]

My two cents, but someone submitted ~3600 french sentences. Validation right now is painful. I think an efficient validation scheme would be:

• 10-20 sentences per page
• checkbox and not two buttons 👍 / 👎
• a checkbox at the top that would tick all

This way, it's much more efficient:

• I can read fastly and check if it's okay
• if it's the case, then I use the one-checkbox-for-all and submit

It's a two click + scroll step VS 5 clicks moving around half the screen.

[nukeador]

Thanks everyone for your feedback, it's going to be super useful once we have time to prioritize this issue.

For context, this is where we are currently focused: https://discourse.mozilla.org/t/sentence-collection-tool-february-2019-milestone-discussion/35166/13

[MeryllEssig]

My two cents, but someone submitted ~3600 french sentences. Validation right now is painful. I think an efficient validation scheme would be:

• 10-20 sentences per page
• checkbox and not two buttons 👍 / 👎
• a checkbox at the top that would tick all

This way, it's much more efficient:

• I can read fastly and check if it's okay
• if it's the case, then I use the one-checkbox-for-all and submit

It's a two click + scroll step VS 5 clicks moving around half the screen.

@lissyx I'm the one who collected the 3600 sentences, I created a little extraction tool in node.js that extract sentences from a txt book from project gutenberg, and preprocess it to remove sentences with more than 14 words, less than 3 words, and more than 100 characters (among other things). I can open source the tool if someone wants to use it.

But yeah, It's hard to review now with that much sentences. But I'm not sure checkboxes are a good way. For example, if we don't want to review a sentence because we're not sure it would be a good one, we can't. Speed of reviewing has a limit in my opinion.

[lissyx]

My two cents, but someone submitted ~3600 french sentences. Validation right now is painful. I think an efficient validation scheme would be:

• 10-20 sentences per page
• checkbox and not two buttons +1 / -1
• a checkbox at the top that would tick all

This way, it's much more efficient:

• I can read fastly and check if it's okay
• if it's the case, then I use the one-checkbox-for-all and submit

It's a two click + scroll step VS 5 clicks moving around half the screen.

@lissyx I'm the one who collected the 3600 sentences, I created a little extraction tool in node.js that extract sentences from a txt book from project gutenberg, and preprocess it to remove sentences with more than 14 words, less than 3 words, and more than 100 characters (among other things). I can open source the tool if someone wants to use it.

I don't want to hijack, but this is something we already had ... https://github.com/Common-Voice/commonvoice-fr/blob/master/project-gutenberg.py

But yeah, It's hard to review now with that much sentences. But I'm not sure checkboxes are a good way. For example, if we don't want to review a sentence because we're not sure it would be a good one, we can't. Speed of reviewing has a limit in my opinion.

You can always untick some sentences.

[lissyx]

@MeryllEssig To avoid hijacking the discussion here, I'd suggest to have a look at https://github.com/mozfr/besogne/wiki/Common-Voice-fr and join us on Telegram.

[nukeador]

Can you all please share any work/scripts you have on discourse? I think this is extremely helpful for other communities, we can maintain a topic there about these things :-)

Thanks!

[klasrocket]

Me and a friend also realized how much work it is involved in reviewing sentences and put together a GreaseMonkey script to auto set all sentences to 👍 on load, which, as I understand should not violate any rule or purpose. Only tested on Firefox/Linux sofar.
https://gist.github.com/TimGremalm/0b865f14f7e8be23c068bba666ef7a07

[nukeador]

@klasrocket I would like to ask you not to use scripts to bypass the review process to mass-vote sentences. The whole purpose of this tool is to make sure at least a few individuals have read the sentences, this is really important because if wrong sentences end up in the voice collection they will be useless for the machine learning algorithm, wasting everyone's time.

Please, let us know how to improve the process and we will try our best (or write a PR is you have the skills) :-)

Thanks.

[klasrocket]

Ok, I see your point @nukeador even if I don't totally agree. Unfortunately I'm a noob so don't expect any major code improvements from me, eager to learn though. But I'll try to contribute with some thoughts :) I agree with previous comments that it's really hard to get things done with the current setup, which I think is more about messing with the mouse than reading/evaluating, I probably won't bother to review at all if 4 clicks and 4 mouse movements/3 sentences are necessary and I believe that people that actually found the tool and put time into doing the reviewing are motivated enough to not misuse the tool, if they understand how it works and why, maybe the "criterias" could be clarified/explained a bit more and placed on the review page so you can't miss them. Another way might be to evaluate the sentence one by one with one button, preferably a keyboard shortcut.
Another thought is that it seems like the sentences keeps a specific order, wouldn't it be better to randomize the order to make more sentences get reviewed, or maybe that happens when the first few sentences gets 3 votes?

[MichaelNMaggs]

The sentences appear for review in the same order that they were submitted (grouped by the number of previous votes, I understand). Where they come from a public domain source, such as a book, that tends to result in similar words and phrases appearing close together as the sentences sequentially track though the story. That makes them rather boring to review, and it would be give reviewers a better experience if all the review sentences could be randomized - not only within a single upload, but also across uploads.

[nukeador]

Thanks for the feedback, please keep it coming. We can probably identify what are the smaller improvements we can implement to keep improving the experience.

I'll check this with @MichaelKohler which is the only regular contributor we have for this project (but please, note we are still trying to finish our February milestone)

[nukeador]

Let me suggest something here, since this issue has turned into a big conversation that might be difficult to follow for other people, I think it's better if we move it to the project discourse with a nice summary and keep individual ideas we come with as individual issues here (but only create them once we have agreement).

I could try to find sometime to write the summary but I'm not sure if I'll have time this week (in case someone else wants to take the lead here)

[nukeador]

(And sorry for the noise, I was just testing the github-discourse importer)

[MichaelKohler]

Agreed, let's create a Discourse topic, I've also indicated that at https://discourse.mozilla.org/t/common-voice-sentence-collection-tool-launch/35427/18?u=mkohler. For now let's put the limit per page from 3 to 5 as discussed further above as first step.

[MichaelKohler]

Here is the new topic: https://discourse.mozilla.org/t/suggestions-for-the-sentence-collector/37734/3

[Mte90]

I saw now 5200~ Italian sentences to review and 3 for every view is like a big job also because loading is also not very fast.

I think that we can copy the UI from localization tools where the sentences are like 20-50 so it is more fast.

[MichaelKohler]

🎉 This issue has been resolved in version 1.8.8 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀