examples/0.getting_started/0.learn_the_basics/learn_the_basics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Toloka and Toloka API\n",
    "\n",
    "Toloka is a crowdsourcing platform that helps to analyze large volumes of data in a short period of time.\n",
    "\n",
    "Examples of common tasks:\n",
    "* Group the wide variety of items in your online store into categories.\n",
    "* Find or verify information.\n",
    "* Translate texts.\n",
    "\n",
    "[Toloka-Kit](https://github.com/Toloka/toloka-kit) is an open-source library, integrated into Toloka API functionality.\n",
    "\n",
    "**Useful links:**\n",
    "\n",
    "- [Toloka Kit documentation](https://toloka.ai/en/docs/toloka-kit/?utm_source=github&utm_medium=site&utm_campaign=tolokakit)\n",
    "- [Toloka homepage](https://toloka.ai/?utm_source=github&utm_medium=site&utm_campaign=tolokakit)\n",
    "- [Toloka requester's guide](https://toloka.ai/en/docs/?utm_source=github&utm_medium=site&utm_campaign=tolokakit)\n",
    "- [Toloka API documentation](https://toloka.ai/en/docs/api/?utm_source=github&utm_medium=site&utm_campaign=tolokakit)\n",
    "\n",
    "The best way to start is to test Toloka web interface by trying out [one of the tutorials](https://toloka.ai/en/docs/guide/concepts/usecases?utm_source=github&utm_medium=site&utm_campaign=tolokakit).\n",
    "\n",
    "## Registration\n",
    "\n",
    "1. [Register](https://toloka.ai/en/docs/guide/concepts/access?utm_source=github&utm_medium=site&utm_campaign=tolokakit) in Toloka as a requester.\n",
    "2. Choose the backend:\n",
    "  * The [production backend](https://platform.toloka.ai/for-requesters/?utm_source=github&utm_medium=site&utm_campaign=tolokakit) is used by default in this example.\n",
    "  * The [sandbox backend](https://platform.sandbox.toloka.ai/for-requesters/?utm_source=github&utm_medium=site&utm_campaign=tolokakit) is a testing environment for Toloka. [Learn more](https://toloka.ai/en/docs/guide/concepts/sandbox?utm_source=github&utm_medium=site&utm_campaign=tolokakit).\n",
    "3. [Add funds](https://toloka.ai/en/docs/guide/concepts/refill?utm_source=github&utm_medium=site&utm_campaign=tolokakit) to your Toloka account, if you're going to use the production version.\n",
    "4. [Get an OAuth token](https://toloka.ai/en/docs/api/concepts/access#token?utm_source=github&utm_medium=site&utm_campaign=tolokakit) for your version. Go to **Profile** → **Integrations** → **Get OAuth Token**.\n",
    "\n",
    "<table  align=\"center\">\n",
    "  <tr><td>\n",
    "    <img src=\"./img/OAuth.png\"\n",
    "         alt=\"OAuth token\"  width=\"1000\">\n",
    "  </td></tr>\n",
    "  <tr><td align=\"center\">\n",
    "    <b>Figure 1.</b> How to get an OAuth token.\n",
    "  </td></tr>\n",
    "</table>\n",
    "\n",
    "### Call to action\n",
    "If you found some bugs or have a new feature idea, don't hesitate to [open a new issue on Github](https://github.com/Toloka/toloka-kit/issues/new/choose).\n",
    "Like our library and examples? Star [our repo on Github](https://github.com/Toloka/toloka-kit)\n",
    "\n",
    "## Getting started with Toloka-Kit\n",
    "Install Toloka-Kit, import the necessary libraries into your Python script and set up logging."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "!pip install toloka-kit==1.0.2\n",
    "!pip install pandas\n",
    "!pip install ipyplot\n",
    "\n",
    "import datetime\n",
    "import time\n",
    "import logging\n",
    "import sys\n",
    "import getpass\n",
    "\n",
    "import pandas\n",
    "import ipyplot\n",
    "\n",
    "import toloka.client as toloka\n",
    "import toloka.client.project.template_builder as tb\n",
    "\n",
    "logging.basicConfig(\n",
    "    format='[%(levelname)s] %(name)s: %(message)s',\n",
    "    level=logging.INFO,\n",
    "    stream=sys.stdout,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Сreate a Toloka client instance. All API calls will go through it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "toloka_client = toloka.TolokaClient(getpass.getpass('Enter your OAuth token: '), 'PRODUCTION') # Or switch to 'SANDBOX'\n",
    "# Lines below check that the OAuth token is correct and print your account's name\n",
    "print(toloka_client.get_requester())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cells below can help you learn more about an object or a method you are interested in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "toloka.TolokaClient?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "toloka.TolokaClient.get_requester?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "toloka.requester.Requester?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Toloka entities and how to manage them with Toloka-Kit\n",
    "\n",
    "### Project\n",
    "A [project](https://toloka.ai/en/docs/guide/concepts/overview?utm_source=github&utm_medium=site&utm_campaign=tolokakit#project) is a top-level object. It contains instructions, task interface settings, input and output data specification, and default quality control rules for this project pools. Projects make it easier for you to post similar tasks in the future, because you don't have to re-configure the interface.\n",
    "\n",
    "The easier the task, the better the results. If your task contains more than one question, you should divide it into several projects.\n",
    "\n",
    "In this tutorial you will create a project with tasks that ask Tolokers to specify the type of animal depicted in a photo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_project = toloka.Project(\n",
    "    public_name='Cat or Dog?',\n",
    "    public_description='Specify the type of animal depicted in a photo.',\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The cell above creates an object in your device memory. This is not all, the project must also contain:\n",
    "* [Input and output data specification](https://toloka.ai/en/docs/guide/concepts/incoming?utm_source=github&utm_medium=site&utm_campaign=tolokakit)\n",
    "* [Task interface settings](https://toloka.ai/en/docs/guide/concepts/spec?utm_source=github&utm_medium=site&utm_campaign=tolokakit)\n",
    "* [An instruction](ttps://toloka.ai/en/docs/guide/concepts/instruction?utm_source=github&utm_medium=site&utm_campaign=tolokakit)\n",
    "\n",
    "**Important:** Several cells below will add changes to the object stored in your device memory. The data will only be sent to the server after calling one of the `toloka_client` methods.\n",
    "\n",
    "#### Input and output data\n",
    "\n",
    "The `image` input field contains URLs of images that need to be labeled.\n",
    "\n",
    "The `result` output field will receive `cat` and `dog` labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_specification = {'image': toloka.project.UrlSpec()}\n",
    "output_specification = {'result': toloka.project.StringSpec()}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Task interface\n",
    "\n",
    "The task interface displays the main task elements to Tolokers. It's important because it is how Tolokers see your tasks. If it's too complicated and unclear, the labeling results might be poor.\n",
    "\n",
    "There are two editors available in Toloka:\n",
    "* [HTML/CSS/JS editor](https://toloka.ai/en/docs/guide/concepts/spec?utm_source=github&utm_medium=site&utm_campaign=tolokakit#interface-section)\n",
    "* [Template Builder](https://toloka.ai/en/docs/template-builder/?utm_source=github&utm_medium=site&utm_campaign=tolokakit)\n",
    "\n",
    "Template Builder configures task interface at the entity level. We recommend it for your projects, especially for the first ones.\n",
    "\n",
    "The cell below creates a task interface for our project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This component shows images\n",
    "image_viewer = tb.ImageViewV1(tb.InputData('image'), ratio=[1, 1])\n",
    "\n",
    "# This component allows to select a label\n",
    "radio_group_field = tb.RadioGroupFieldV1(\n",
    "    tb.OutputData('result'),\n",
    "    [\n",
    "        tb.fields.GroupFieldOption('cat', 'Cat'),\n",
    "        tb.fields.GroupFieldOption('dog', 'Dog')\n",
    "    ],\n",
    "    validation=tb.RequiredConditionV1(),\n",
    ")\n",
    "\n",
    "# Allows to set a width limit when displaying a task\n",
    "task_width_plugin = tb.TolokaPluginV1(\n",
    "    'scroll',\n",
    "    task_width=400,\n",
    ")\n",
    "\n",
    "# How performers will see the task\n",
    "project_interface = toloka.project.TemplateBuilderViewSpec(\n",
    "    view=tb.ListViewV1([image_viewer, radio_group_field]),\n",
    "    plugins=[task_width_plugin],\n",
    ")\n",
    "\n",
    "# This block assigns task interface and input/output data specification to the project\n",
    "# Note that this is done via the task specification class\n",
    "new_project.task_spec = toloka.project.task_spec.TaskSpec(\n",
    "    input_spec=input_specification,\n",
    "    output_spec=output_specification,\n",
    "    view_spec=project_interface,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Task instructions\n",
    "\n",
    "The first thing the Tolokers see when they select a task are the [instructions](https://toloka.ai/en/docs/guide/concepts/instruction?utm_source=github&utm_medium=site&utm_campaign=tolokakit) that you wrote. Describe what needs to be done in simple and clear language, and give examples.\n",
    "\n",
    "Good instructions help the Tolokers complete the task correctly. The clarity and completeness of the instructions affect the response quality and the project rating. Unclear or too complex instructions, on the contrary, will scare off Tolokers.\n",
    "\n",
    "Create the instructions for your project with the following cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_project.public_instructions = 'Look at the picture. Determine what is on it: a <b>cat</b> or a <b>dog</b>. Choose the correct option.'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create a project\n",
    "\n",
    "Now, use `toloka_client` defined at the beginning to create the project.\n",
    "\n",
    "The data is only sent to the server after calling one of the `toloka_client` methods, the cell below actually creates a project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_project = toloka_client.create_project(new_project)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Project preview\n",
    "\n",
    "1. Preview project\n",
    "\n",
    "2. Go to the project page to make sure the task interface works correctly. To do this, click the link in the output of the cell above.\n",
    "\n",
    "3. In the upper-right corner of the project page click ![Project actions](./img/drop-down.svg) → **![Preview](./img/preview.svg) Preview**:\n",
    "\n",
    "<table  align=\"center\">\n",
    "  <tr><td>\n",
    "    <img src=\"./img/project_look.png\"\n",
    "         alt=\"Project interface\"  width=\"1000\">\n",
    "  </td></tr>\n",
    "  <tr><td align=\"center\">\n",
    "    <b>Figure 2.</b> What the project interface might look like.\n",
    "  </td></tr>\n",
    "</table>\n",
    "\n",
    "4. In the upper part of the preview page click **Change input data**, and insert an image URL into the `image` field, then click **Apply**.\n",
    "\n",
    "<table  align=\"center\">\n",
    "  <tr><td>\n",
    "    <img src=\"./img/task_look.png\"\n",
    "         alt=\"Task interface\"  width=\"1000\">\n",
    "  </td></tr>\n",
    "  <tr><td align=\"center\">\n",
    "    <b>Figure 3.</b> What the task interface might look like and how to insert images in the preview.\n",
    "  </td></tr>\n",
    "</table>\n",
    "\n",
    "5. In the upper part of the preview demonstration click **Instructions**. Make sure the instructions are shown and that they say what you want them to.\n",
    "\n",
    "6. Select an option in your task. In the lower-left corner of the preview demonstration click **Submit**, then **View responses**. In the appeared result window, check that your results are written in expected format and that the entered data is correct.\n",
    "\n",
    "<table  align=\"center\">\n",
    "  <tr><td>\n",
    "    <img src=\"./img/results_preview.png\"\n",
    "         alt=\"Result priview\"  width=\"1000\">\n",
    "  </td></tr>\n",
    "  <tr><td align=\"center\">\n",
    "    <b>Figure 4.</b> What the results might look like.\n",
    "  </td></tr>\n",
    "</table>\n",
    "\n",
    "Tips:\n",
    "* We strongly recommend **checking the task interface and instructions** every time you create a project. This will help you to ensure that the Tolokers will complete the task and that your results will be useful.\n",
    "* Do a **trial run** with a small amount of data. Make sure that after running the entire pipeline you get the data in the expected format and quality."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pool\n",
    "A [pool](https://toloka.ai/en/docs/guide/concepts/overview?utm_source=github&utm_medium=site&utm_campaign=tolokakit#pool) is a set of tasks that share common pricing, start date, selection of Tolokers, overlap, and quality control configurations. All task in a pool are processed in parallel. One project can have several pools. You can add new tasks to a pool at any time, as well as open or stop it.\n",
    "\n",
    "The cell below will create a pool as an object in your device memory. You will send it to Toloka with `toloka_client` method a bit later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_pool = toloka.Pool(\n",
    "    project_id=new_project.id,\n",
    "    private_name='Pool 1',  # Only you can see this information\n",
    "    may_contain_adult_content=False,\n",
    "    reward_per_assignment=0.01,  # Sets the minimum payment amount for one task suite in USD\n",
    "    assignment_max_duration_seconds=60*5,  # Gives performers 5 minutes to complete one task suite\n",
    "    will_expire=datetime.datetime.utcnow() + datetime.timedelta(days=365),  # Sets that the pool will close after one year\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overlap\n",
    "\n",
    "To minimize the risk of getting wrong answers, you can ask several Tolokers to complete the same task. This is called _overlap_.\n",
    "\n",
    "In this example we set the overlap to `3`. This means that **every** task will be completed by **three** different Tolokers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_pool.defaults = toloka.pool.Pool.Defaults(\n",
    "    default_overlap_for_new_tasks=3,\n",
    "    default_overlap_for_new_task_suites=0,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task suite\n",
    "\n",
    "A [task suite](https://toloka.ai/en/docs/guide/concepts/overview?utm_source=github&utm_medium=site&utm_campaign=tolokakit#tasks-page) is a set of tasks that are shown on a single page.\n",
    "\n",
    "An important part of configuring pools is to decide how many tasks will be issued to a Toloker at once. For example, if you set 3 tasks for a task suite, a Toloker will see three images at once on one page.\n",
    "\n",
    "Note that the `reward_per_assignment` and `assignment_max_duration_seconds` fields in pool settings set the price and time for one **task suite**, not task.\n",
    "\n",
    "Why you should combine tasks in a task suite:\n",
    "\n",
    "* To set a more precise price for a single task.\n",
    "* To calculate a Toloker's skill and use it to determine the correct answer more accurately. Learn more in the [Aggregation](#aggregation) section.\n",
    "* To better apply quality control settings that improve the final quality of the response. Learn more in the [Quality control rules](#quality_control_rules) section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_pool.set_mixer_config(\n",
    "    real_tasks_count=10,  # The number of tasks per page.\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Filters\n",
    "\n",
    "[Filters](https://toloka.ai/en/docs/guide/concepts/filters?utm_source=github&utm_medium=site&utm_campaign=tolokakit) help you select Tolokers for your project.\n",
    "\n",
    "There may be different reasons to use filters, for example:\n",
    "\n",
    "* You require some specific group of Tolokers for your pool.\n",
    "* You want to exclude a certain group of Tolokers.\n",
    "\n",
    "Tasks will only be shown to matching Tolokers, rather than to all of them.\n",
    "\n",
    "This example requires English-speaking Tolokers, because the project instructions are in English."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_pool.filter = toloka.filter.Languages.in_('EN')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Quality control rules<a id='quality_control_rules'></a>\n",
    "\n",
    "[Quality control rules](https://toloka.ai/en/docs/guide/concepts/check-performers?utm_source=github&utm_medium=site&utm_campaign=tolokakit) regulate task completion and Toloker access.\n",
    "\n",
    "Quality control lets you get more accurate responses and restrict access to tasks for cheating users. All rules work independently. Learn more about [settting up quality control](https://toloka.ai/en/docs/guide/concepts/qa-pool-settings?utm_source=github&utm_medium=site&utm_campaign=tolokakit).\n",
    "\n",
    "This example uses the captcha rule. It is the simplest way to exclude fake users (robots) and cheaters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Turns on captchas\n",
    "new_pool.set_captcha_frequency('MEDIUM')\n",
    "# Bans performers by captcha criteria\n",
    "new_pool.quality_control.add_action(\n",
    "    # Type of quality control rule\n",
    "    collector=toloka.collectors.Captcha(history_size=5),\n",
    "    # This condition triggers the action below\n",
    "    # Here overridden comparison operator actually returns a Condition object\n",
    "    conditions=[toloka.conditions.FailRate > 20],\n",
    "    # What exactly should the rule do when the condition is met\n",
    "    # It bans the performer for 1 day\n",
    "    action=toloka.actions.RestrictionV2(\n",
    "        scope='PROJECT',\n",
    "        duration=1,\n",
    "        duration_unit='DAYS',\n",
    "        private_comment='Captcha',\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create pool\n",
    "\n",
    "The cell below creates a pool with all the information above which was stored in your device memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_pool = toloka_client.create_pool(new_pool)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Open your project page. You will see your new pool.\n",
    "\n",
    "<table  align=\"center\">\n",
    "  <tr><td>\n",
    "    <img src=\"./img/project_with_pool.png\"\n",
    "         alt=\"Project interface with a pool\"  width=\"1000\">\n",
    "  </td></tr>\n",
    "  <tr><td align=\"center\">\n",
    "    <b>Figure 5.</b> Project interface with a pool.\n",
    "  </td></tr>\n",
    "</table>\n",
    "\n",
    "The pool interface looks like this.\n",
    "\n",
    "<table  align=\"center\">\n",
    "  <tr><td>\n",
    "    <img src=\"./img/pool_preview.png\"\n",
    "         alt=\"Pool interface\"  width=\"1000\">\n",
    "  </td></tr>\n",
    "  <tr><td align=\"center\">\n",
    "    <b>Figure 6.</b> Pool interface.\n",
    "  </td></tr>\n",
    "</table>\n",
    "\n",
    "Right now the pool is empty and closed. It has no tasks or task suites."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Upload tasks\n",
    "\n",
    "A [task](https://toloka.ai/en/docs/guide/concepts/overview?utm_source=github&utm_medium=site&utm_campaign=tolokakit#task) is the smallest portion of data you need to mark up.\n",
    "\n",
    "This example uses a small data set with images. This dataset is collected by the Toloka team and distributed under a\n",
    "[![License: CC BY 4.0](https://img.shields.io/badge/License-CC%20BY%204.0-lightgrey.svg)](https://creativecommons.org/licenses/by/4.0/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download the data set\n",
    "!curl https://tlk.s3.yandex.net/dataset/cats_vs_dogs/toy_dataset.tsv --output dataset.tsv\n",
    "\n",
    "dataset = pandas.read_csv('dataset.tsv', sep='\\t')\n",
    "\n",
    "print(f'Dataset contains {len(dataset)} rows\\n')\n",
    "\n",
    "dataset = dataset.sample(frac=1).reset_index(drop=True)\n",
    "\n",
    "ipyplot.plot_images(\n",
    "    images=[row['url'] for _, row in dataset.iterrows()],\n",
    "    labels=[row['label'] for _, row in dataset.iterrows()],\n",
    "    max_images=12,\n",
    "    img_width=300,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create tasks. One task will be created from one image.\n",
    "\n",
    "Toloka will automatically create task suites and show the tasks depending on a project overlap:\n",
    "\n",
    "1. One task suite will consist of 10 tasks.\n",
    "2. Toloka will let 3 different Tolokers to complete the tasks.\n",
    "\n",
    "We configured these settings while creating the pool."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tasks = [\n",
    "    toloka.Task(input_values={'image': url}, pool_id=new_pool.id)\n",
    "    for url in dataset['url']\n",
    "]\n",
    "# Add tasks to a pool\n",
    "toloka_client.create_tasks(tasks, allow_defaults=True)\n",
    "print(f'Populated pool with {len(tasks)} tasks')\n",
    "print(f'To view this pool, go to https://toloka.dev/requester/project/{new_project.id}/pool/{new_pool.id}')\n",
    "# print(f'To view this pool, go to https://sandbox.toloka.dev/requester/project/{new_project.id}/pool/{new_pool.id}') # Print a sandbox version link\n",
    "\n",
    "# Opens the pool\n",
    "new_pool = toloka_client.open_pool(new_pool.id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you open your pool, Tolokers will see your tasks in their mobile app or in Toloka web version, and start working on them.\n",
    "\n",
    "In small pools like this, it usually takes up to 10 minutes for all the tasks to be performed.\n",
    "\n",
    "With big pools, we recommend that you set up automatic waiting. See the example in the cell below.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pool_id = new_pool.id\n",
    "\n",
    "def wait_pool_for_close(pool_id, minutes_to_wait=1):\n",
    "    sleep_time = 60 * minutes_to_wait\n",
    "    pool = toloka_client.get_pool(pool_id)\n",
    "    while not pool.is_closed():\n",
    "        op = toloka_client.get_analytics([toloka.analytics_request.CompletionPercentagePoolAnalytics(subject_id=pool.id)])\n",
    "        op = toloka_client.wait_operation(op)\n",
    "        percentage = op.details['value'][0]['result']['value']\n",
    "        print(\n",
    "            f'   {datetime.datetime.now().strftime(\"%H:%M:%S\")}\\t'\n",
    "            f'Pool {pool.id} - {percentage}%'\n",
    "        )\n",
    "        time.sleep(sleep_time)\n",
    "        pool = toloka_client.get_pool(pool.id)\n",
    "    print('Pool was closed.')\n",
    "\n",
    "wait_pool_for_close(pool_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get responses\n",
    "\n",
    "When all the tasks are completed, look at the responses from Tolokers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "answers_df = toloka_client.get_assignments_df(pool_id)\n",
    "# Prepare dataframe for aggregation\n",
    "answers_df = answers_df.rename(columns={\n",
    "    'INPUT:image': 'task',\n",
    "    'OUTPUT:result': 'label',\n",
    "    'ASSIGNMENT:worker_id': 'worker',\n",
    "})\n",
    "\n",
    "print(f'answers count: {len(answers_df)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An `assignment` value is one Toloker's responses to all the tasks on a task suite.\n",
    "\n",
    "If a Toloker completed several task suites, then `toloka_client.get_assignments_df` will contain several `assignment` values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Aggregation <a id='aggregation'></a>\n",
    "\n",
    "You should run the [results aggregation](https://toloka.ai/en/docs/guide/concepts/result-aggregation?utm_source=github&utm_medium=site&utm_campaign=tolokakit) only if you set the overlap for the tasks to 2 or higher.\n",
    "\n",
    "The [majority vote](https://toloka.ai/en/docs/guide/concepts/mvote?utm_source=github&utm_medium=site&utm_campaign=tolokakit) method is a quality control method based on matching responses from the majority of Tolokers who complete the same task. For example, if 2 out of 3 Tolokers selected the `cat` label, then the final label for this task will be `cat`.\n",
    "\n",
    "Majority vote is easily implemented, but you can also use our crowdsourcing [Crowd-Kit library](https://github.com/Toloka/crowd-kit?utm_source=github&utm_medium=site&utm_campaign=tolokakit). It contains a lot of new aggregation methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install crowd-kit==1.1.0\n",
    "from crowdkit.aggregation import MajorityVote"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "MajorityVote?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run majority vote aggregation\n",
    "predicted_answers = MajorityVote().fit_predict(answers_df)\n",
    "\n",
    "print(predicted_answers)\n",
    "\n",
    "# Some preparations for displaying the results\n",
    "predicted_answers = predicted_answers.sample(frac=1)\n",
    "images = predicted_answers.index.values\n",
    "labels = predicted_answers.values\n",
    "start_with = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look at the results.\n",
    "\n",
    "Note: The cell below can be run several times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if start_with >= len(predicted_answers):\n",
    "    print('no more images')\n",
    "else:\n",
    "    ipyplot.plot_images(\n",
    "        images=images[start_with:],\n",
    "        labels=labels[start_with:],\n",
    "        max_images=12,\n",
    "        img_width=300,\n",
    "    )\n",
    "\n",
    "    start_with += 12"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see the labeled images. Some possible results are shown in figure 7 below.\n",
    "\n",
    "<table  align=\"center\">\n",
    "  <tr><td>\n",
    "    <img src=\"./img/possible_results.png\"\n",
    "         alt=\"Possible results\"  width=\"1000\">\n",
    "  </td></tr>\n",
    "  <tr><td align=\"center\">\n",
    "    <b>Figure 7.</b> Possible results.\n",
    "  </td></tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "This example explained basic Toloka entities and how Toloka-Kit can work with them.\n",
    "\n",
    "The described project (classification) is very useful for:\n",
    "* Accurate evaluation.\n",
    "* Checking the results of a complex project, as in [object detection example](https://github.com/Toloka/toloka-kit/tree/main/examples/1.computer_vision/object_detection) [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Toloka/toloka-kit/blob/main/examples/1.computer_vision/object_detection/object_detection.ipynb)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10.6 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "vscode": {
   "interpreter": {
    "hash": "e7370f93d1d0cde622a1f8e1c04877d8463912d04d973331ad4851f04de6915a"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}