Quality control lets you get more accurate responses and restrict access to tasks for cheating Tolokers. Quality control consists of rules. All rules work independently.
{% note warning %}
Quality control settings in a project are applied to all project pools, so you can't change them in just one of the pools.
When you clone a project, its quality control settings aren't transferred.
{% endnote %}
To set up quality control:
-
Open the project page.
-
Go to the Quality control tab.
-
Click Set quality control.
-
Click Add Quality Control Rule.
-
In the list that appears, select the appropriate rules. If you aren't sure what quality control rules you need, select a ready-made protection set of rules with default settings.
-
Make settings for the rules you added. Below is a list of rules with links to detailed information about the rule settings.
-
To keep track of how often Tolokers make mistakes:
-
Control tasks: Use them to assign a skill to Tolokers based on their control task responses and ban Tolokers who submit incorrect responses.
-
Majority vote: Quality is based on matching the response from the majority of Tolokers who complete the same task.
-
Results of checking: Evaluate Tolokers based on the number of accepted and rejected responses.
-
-
To protect your project from robots and cheaters:
-
Fast responses: Control the minimum time that must be spent per task suite.
-
Skipped assignments: Restrict access to your pool tasks for Tolokers who skip multiple assignments in a row.
-
-
To attract a variety of Tolokers:
-
Earnings: Limit the amount each Toloker can earn in the pool per day.
-
Submitted assignments: Limit how many assignments each Toloker can submit in the pool per day.
-
-
To allow recompletion of certain assignments:
-
Recompletion of assignments from banned users: Send completed assignments to other Tolokers to redo them if the Toloker was banned.
-
Processing of rejected and accepted assignments: Send rejected assignments to other Tolokers to redo them.
-
-
Create a task pool in the project.
-
Learn more about how to set up a project:
{% include faq-set-quality-control %}
{% cut "How many control tasks do I need to add?" %}
We recommend adding at least 1% of control tasks in the pool. And for small pools — 5–10%.
{% cut "Why's that?" %}
Each control task is shown to the Toloker only once. If you use smart mixing, you determine how many control tasks should be in a suite. If each suite contains one control task, then the maximum number of suites the Toloker can complete is equal to the number of control tasks in the pool. If you increase the number of control tasks in a suite, the number of suites available to the Toloker decreases by the same number.
There shouldn't be too few pages available. Otherwise:
- You won't be able to correctly evaluate the quality of the Toloker's responses.
- The Toloker won't be interested in completing such tasks because they'll spend a lot of time studying instructions but won't earn much.
{% cut "Example" %}
There are 10,000 tasks in the pool, and 100 of them are control tasks (1%). Each suite contains 10 tasks, and 1 of them is a control task. Hence, a user can complete up to 100 suites.
There are 100 tasks in the pool, and 1 of them is a control task (1%). Each suite contains 10 tasks, and 1 of them is a control task. Hence, each user can only complete 1 suite.
There are 100 tasks in the pool, and 10 of them are control tasks (10%). Each suite contains 10 tasks, and 1 of them is a control task. Hence, each user can complete up to 100 suites.
{% endcut %}
{% endcut %}
If there are few control tasks in the open pool, add new control tasks.
{% cut "What for" %}
In a large pool with few control tasks, a situation might occur when users who have completed a lot of tasks in the project stop getting new task suites. This happens when the Toloker completes all control tasks in the pool.
{% endcut %}
{% note info %}
To filter out Tolokers, use the Control tasks quality control rule. To rank Tolokers by the quality of responses in control tasks, use a skill.
{% endnote %}
{% endcut %}
{% include faq-correct-responses-counted %}
{% include faq-skill-for-every-pool %}
{% include faq-skill-for-different-projects %}
{% include faq-disable-tasks %}
{% include contact-support %}