External API for sensitive content detection

[obulat]

Summary

There are some external APIs that can label images with semantic labels and detect whether the image is sensitive. This can be highly relevant to search relevancy and content safety.

This project will need to select the optimal service to use and need to make a performance determination about implementing this as ‘on the fly’ vs. as a job queue.

Description

The project would have three basic parts:

1. Working through a queue of works that need scanning

2. Auto-moderating works based on high-confidence scan results and integrating others into the overall moderation queue (whether as part of the same queue as user reports or a secondary one)

3. Indicating on a work when its sensitivity designation is a result of auto or manual moderation based on machine labels

The last one also like increases the need for a "moderation challenge" queue so that auto-moderated works in particular have an easy avenue for users to challenge the moderation result

Best guess at list of implementation plans:

• Groundwork: Investigate the performance characteristics of various approaches to building and working through the queue; propose the most-likely-to-succeed version

• Groundwork: Choose a tool to use for machine labelling and identify confidence characteristics of the output; make a proposal for auto-moderation, if it can happen at all on both sides (i.e., how confident is the output that a work is not sensitivity or that it is sensitive: both are basically auto-moderation in one direction or another)

• Groundwork: Work with moderators to decide whether to integrate machine labelled works into the moderation queue and how to prioritise review

• Groundwork: Propose an approach for communicating machine-labelling based moderation decisions on works designated as sensitive

• IP: The to-be-labelled queue and integration with the labelling tool

• IP: Surface results on the API

• IP: Integrate results into the moderation queue

• IP: Frontend presentation of machine-labelling based auto-moderation

Documents

• Project Proposal
• Implementation Plan

Issues

Prior Art