Improve UX of reviewing task results

[meta-paul]

Need to write a simple review server with customizable front-end, to replace current command-line code.

[meta-paul]

API v0

---

GET /tasks

Get all available tasks (to select one for review)

{
    "tasks": [
        {
            "id": <int>,
            "name": <str>,
            "is_reviewed": <bool>,
            "unit_count": <int>,
            "created_at": <datetime>
        },
        ...  // more tasks
    ]
}

---

GET /qualifications

Get all available qualifications (to select "approve" and "reject" qualifications)

{
    "qualifications": [
        {
            "id": <int>,
            "name": <str>,
        },
        ...  // more qualifications
    ]
}

---

POST /qualifications

Create a new qualification

{
    "name": <str>,
}

---

GET /task-units?{task_id=}{qualification_id=}{worker_id=}{limit=}{offset=}

Get all workers results (filtered by task_id and/or qualification_id, etc)

{
    "units": [
        {
            "id": <int>,
            "worker_id": <int>,
            "task_id": <int>,
            "pay_amount": <int>,
            "status": <str>,
            "creation_date": <int>,
            "results": {
                "start": <timestamp>,
                "end": <timestamp>,
                "input": <json str>,
                "tips": <int>,  // optional
                "feedback": <str>,  // optional
            }
        },
        ...  // more task units
    ]
}

---

GET /granted-qualifications?{task_id=}{qualification_id=}{worker_id=}{limit=}{offset=}

Get all granted qualifications (filtered by task_id and/or qualification_id, etc)

{
    "granted_qualifications": [
        {
            "id": <int>,
            "worker_id": <int>,
            "qualification_id": <int>,
            "value": <int>,
            // "feedback": <str>,
            "creation_date": <int>,
        },
        ...  // more granted qualifications
    ]
}

---

POST /granted-qualifications/reject

Reject worker's input

{
    "qualification_id": <int>,
    "worker_id": <int>,
    "block_permanently": <bool>,
    "feedback": <str>,  // optional
}

---

POST /granted-qualifications/approve

Approve worker's input

{
    "qualification_id": <int>,
    "worker_id": <int>,
    "value": <int>,
    "feedback": <str>,  // optional
    "tip": <int>,  // optional
}

[JackUrb]

Overall these basic endpoints make sense, but I want to clarify a bit on the design.

I don't know where we'd use "feedback" for approve, and we don't really have metadata about qualifications (like if it's a block) at the moment. On granted-qualifications in general, we tend to use grant and revoke, and we separate these out from approving/rejecting units.

Some key endpoints will be /approve, /soft-reject, and /reject, and they should be able to take multiple unit ids as an argument such that we can do approve-all/soft-reject-all/reject-all queries. Units can also end up having large quantities of data, so we'd likely only want to load the data for a specific unit being actively viewed (though we can pull metadata for chunks).

An aside on the expected review flow:

When going through review, the expectation is that we'll be able to review all of the units in one pass, ideally batched by worker. This behavior in the mock operates through the following steps.

1. Aggregate all of the units for the given task_name.
2. Group these units by worker, and compute previous approval statistics for reviewed work while bucketing the unreviewed work. We then shuffle the unreviewed work to lower the chance of missing quality degradation or ramp up phases if only sampling a few units per worker.
3. Show the work to the reviewer in worker-grouped batches, with additional previous approval statistics as well as overall counts (how many workers and total units are left to be reviewed).
4. Reviewer is able to look at as many per worker as they like, individually actioning. Then, they can bulk apply for the rest.
5. Run until all units are reviewed.

This type of setup requires constructing and maintaining some kind of local state on the server, though the setup I've used so far is really not the right way to do it.

[meta-paul]

API v1

---

GET /tasks

Get all available tasks (to select one for review)

{
    "tasks": [
        {
            "id": <int>,
            "name": <str>,
            "is_reviewed": <bool>,
            "unit_count": <int>,
            "created_at": <datetime>
        },
        ...  // more tasks
    ]
}

---

GET /qualifications

Get all available qualifications (to select "approve" and "reject" qualifications)

{
    "qualifications": [
        {
            "id": <int>,
            "name": <str>,
        },
        ...  // more qualifications
    ]
}

---

POST /qualifications

Create a new qualification

{
    "name": <str>,
}

---

GET /task-units?{task_id=}{qualification_id=}{worker_id=}{limit=}{offset=}

Get all workers results (filtered by task_id and/or qualification_id, etc) - without the details of worker's output

{
    "workers": [
        "id": <int>,
        "stats": <json str>,
        "units": [
            {
                "id": <int>,
                "worker_id": <int>,
                "task_id": <int>,
                "pay_amount": <int>,
                "status": <str>,
                "creation_date": <int>,
                "results": {
                    "start": <timestamp>,
                    "end": <timestamp>,
                    "input_preview": <json str>,  // optional
                    "output_preview": <json str>,  // optional
                },
                "review": {
                    "tips": <int>,
                    "feedback": <str>,
                }
            },
            ...  // more units
        ]
     ],
     ...  // more workers
}

GET /task-units-details?{unit_ids=}

Get full input for specified workers results

{
    "units": [
        {
            "id": <int>,
            "input": <json str>,  // instructions for worker
            "output": <json str>,  // resposne from worker
        },
        ...  // more units
    ]
}

---

POST /task-units/reject

Reject worker's input

{
    "unit_ids": [<int>, ...],
    "feedback": <str>,  // optional
}

---

POST /task-units/soft-reject

Soft-reject worker's input

{
    "unit_ids": [<int>, ...],
    "feedback": <str>,  // optional
}

---

POST /task-units/approve

Approve worker's input

{
    "unit_ids": [<int>, ...],
    "feedback": <str>,  // optional
    "tip": <int>,  // optional
}

---

GET /granted-qualifications?{task_id=}{qualification_id=}{worker_id=}{limit=}{offset=}

Get all granted qualifications (filtered by task_id and/or qualification_id, etc)

{
    "granted_qualifications": [
        {
            "id": <int>,
            "worker_id": <int>,
            "qualification_id": <int>,
            "value": <int>,
            "creation_date": <int>,
        },
        ...  // more granted qualifications
    ]
}

---

POST /granted-qualifications/grant

Grant qualification to worker

{
    "qualification_id": <int>,
    "worker_id": <int>,
    "value": <int>,
}

---

POST /granted-qualifications/revoke

Revoke qualification from worker

{
    "qualification_id": <int>,
    "worker_id": <int>,
    "block_permanently": <bool>,
}

---

POST /workers/{id}/block

Permanently block a worker

{
    "feedback": <str>,
}

[JackUrb]

Note, /task-units-input should be getting the full task data including outputs, not just inputs. We'd be feeding this directly to the review component.

Also, I imagine it makes sense to just move block to its own endpoint.

[meta-paul]

Makes sense, I've just edited API v1.

[meta-paul]

Review flow

• we get list of available tasks from GET /tasks
• User selects a task
• we get list of available qualifications from GET /qualifications
• (optional) User selects "approve" and "reject" qualifications
• We pull all unit ids from GET /tasks/{id}/worker-units-ids
  • Due to the need to randomly shuffle units grouped by a worker (to mitigate reviewers bias, etc) we're implementing client-side pagination - client gets full list of all ids, creates a page of unit ids, and then pulls data for those specific units.
• We group units by worker, sort workers by number of their units, and pick them for review one-by-one
• For each worker:
  • we pull worker stats from GET /workers/{id}/stats
  • we pull units by ids from GET /task-units?unit_ids=[...]
  • we sort units by creation_date and pick them for review one-by-one
    • for each reviewed unit we pull its details from GET /task-units-details?unit_ids=[...]
    • User can choose to reject/accept unit, grant/revoke qualification, and block the worker

API v2

---

GET /tasks

Get all available tasks (to select one for review)

{
    "tasks": [
        {
            "id": <int>,
            "name": <str>,
            "is_reviewed": <bool>,
            "unit_count": <int>,
            "created_at": <datetime>
        },
        ...  // more tasks
    ]
}

---

GET /qualifications

Get all available qualifications (to select "approve" and "reject" qualifications)

{
    "qualifications": [
        {
            "id": <int>,
            "name": <str>,
        },
        ...  // more qualifications
    ]
}

---

POST /qualifications

Create a new qualification

{
    "name": <str>,
}

---

GET /tasks/{id}/worker-units-ids

Get full, unpaginated list of unit IDs within a task (for subsequent client-side grouping by worker_id and GET /task-units pagination)

{
    "worker_units_ids": [
        {
            "worker_id": <int>,
            "unit_id": <int>,
        },
        ...  // more ids
    ]
}

---

GET /task-units?{task_id=}{unit_ids=}{qualification_id=}{worker_id=}

Get workers' results (filtered by task_id and/or unit_ids, etc) - without full details of input/output

{
	"units": [
		{
			"id": <int>,
			"worker_id": <int>,
			"task_id": <int>,
			"pay_amount": <int>,
			"status": <str>,
			"creation_date": <int>,
			"results": {
				"start": <timestamp>,
				"end": <timestamp>,
				"input_preview": <json str>,  // optional
				"output_preview": <json str>,  // optional
			},
			"review": {
				"tips": <int>,
				"feedback": <str>,
			}
		},
		...  // more units
	]
}

GET /task-units-details?{unit_ids=}

Get full input for specified workers results (units_ids is mandatory)

{
    "units": [
        {
            "id": <int>,
            "input": <json str>,  // instructions for worker
            "output": <json str>,  // resposne from worker
        },
        ...  // more units
    ]
}

---

POST /task-units/reject

Reject worker's input

{
    "unit_ids": [<int>, ...],
    "feedback": <str>,  // optional
}

---

POST /task-units/soft-reject

Soft-reject worker's input

{
    "unit_ids": [<int>, ...],
    "feedback": <str>,  // optional
}

---

POST /task-units/approve

Approve worker's input

{
    "unit_ids": [<int>, ...],
    "feedback": <str>,  // optional
    "tip": <int>,  // optional
}

---

GET /granted-qualifications?{task_id=}{qualification_id=}{worker_id=}{limit=}{offset=}

Get all granted qualifications (filtered by task_id and/or qualification_id, etc)

{
    "granted_qualifications": [
        {
            "id": <int>,
            "worker_id": <int>,
            "qualification_id": <int>,
            "value": <int>,
            "creation_date": <int>,
        },
        ...  // more granted qualifications
    ]
}

---

POST /granted-qualifications/grant

Grant qualification to worker

{
    "qualification_id": <int>,
    "worker_id": <int>,
    "value": <int>,
}

---

POST /granted-qualifications/revoke

Revoke qualification from worker

{
    "qualification_id": <int>,
    "worker_id": <int>,
    "block_permanently": <bool>,
}

---

POST /workers/{id}/block

Permanently block a worker

{
    "feedback": <str>,
}

---

GET /workers/{id}/stats?{task_id=}{requester_id=}{since=}{limit=}

Get stats of recent work for the worker

    "worker_stats": [
        {
            "id": <int>,
	        "stats": {
		        "total_count": <int>,  // within the scope of the filters
		        "approved_count": <int>,
		        "rejected_count": <int>,
		        "soft_rejected_count": <int>,
	        },
        },
    ]

[meta-paul]

This has been implemented in v1.2.0