title | titleSuffix | description | author | manager | ms.service | ms.custom | ms.topic | ms.date | ms.author |
---|---|---|---|---|---|---|---|---|---|
Custom categories (rapid) in Azure AI Content Safety |
Azure AI services |
Learn about content incidents and how you can use Azure AI Content Safety to handle them on your platform. |
PatrickFarley |
nitinme |
azure-ai-content-safety |
build-2024 |
conceptual |
04/11/2024 |
pafarley |
In content moderation scenarios, custom categories (rapid) is the process of identifying, analyzing, containing, eradicating, and recovering from cyber incidents that involve inappropriate or harmful content on online platforms.
An incident may involve a set of emerging content patterns (text, image, or other modalities) that violate Microsoft community guidelines or the customers' own policies and expectations. These incidents need to be mitigated quickly and accurately to avoid potential live site issues or harm to users and communities.
One way to deal with emerging content incidents is to use Blocklists, but that only allows exact text matching and no image matching. The Azure AI Content Safety custom categories (rapid) API offers the following advanced capabilities:
- semantic text matching using embedding search with a lightweight classifier
- image matching with a lightweight object-tracking model and embedding search.
First, you use the API to create an incident object with a description. Then you add any number of image or text samples to the incident. No training step is needed.
Then, you can include your defined incident in a regular text analysis or image analysis request. The service will indicate whether the submitted content is an instance of your incident. The service can still do other content moderation tasks in the same API call.
The text custom categories (rapid) API supports all languages that are supported by Content Safety text moderation. See Language support.
See the following table for the input limitations of the custom categories (rapid) API:
Object | Limitation |
---|---|
Maximum length of an incident name | 100 characters |
Maximum number of text/image samples per incident | 1000 |
Maximum size of each sample | Text: 500 characters Image: 4 MB |
Maximum number of text or image incidents per resource | 100 |
Supported Image formats | BMP, GIF, JPEG, PNG, TIF, WEBP |
To use this API, you must create your Azure AI Content Safety resource in one of the supported regions. See Region availability.
Follow the how-to guide to use the Azure AI Content Safety custom categories (rapid) API.