Prompt Categories

In this section: Prompt Category Evaluation and Operations

Availability: optional prompt categorisation is available, off by default, since 1852. 1854 adds bounded source-tree evaluation tooling and distinct failure outcomes; it does not establish real-model accuracy. This administrator taxonomy is not completion of #652's O*NET-specific classifier, and #686's acceptance remains open.

See Work Trend Index → What work are people asking AI to do? for the operator interpretation: sampled goal counts and administrator annotations, not Microsoft's O*NET percentages or inferred human/agent working modes.

Governance: a separate, explicit prompt data flow

Local synthetic fixture, not live consent, model output, accuracy evidence or persistence proof.

Categorisation is off by default. A configured model alone does not enable it. An administrator must explicitly opt in under Administration → Copilot & reporting → Prompt categories after confirming their organisation permits this data flow.

The feature is an optional prompt-only enrichment, with aggregate-only reporting and an administrator-defined taxonomy. You can use an existing deployment or explicitly enable the optional service in the installer. It does not perform response analysis, create per-person intent profiles, or implement the complete four-mode model proposed in #651.

The existing opt-in interaction-history import reads Microsoft Graph user prompts and responses. Categorisation sends user prompts only to the administrator's Azure AI Foundry model deployment while those prompts are in importer memory. Copilot responses are never sent. The audit feed cannot supply prompt text; it cannot provide categorisation coverage.

Categorisation stores a single allowlisted category id, the immutable taxonomy version and an optional administrator-assigned human-mode annotation. It stores no prompt, response, rationale or model response in SQL, Azure Tables or logs; the existing statistical/AI Language enrichments remain unchanged. Prompt contents can still contain names and sensitive information: this is not a PII-redaction service. The Azure service receives that prompt: assess the deployment's service-side content filtering, abuse monitoring, retention and applicable Microsoft data-processing terms separately. The categorisation no-text-retention rule is not a claim about the model service's own policies.

Global deployment types may process requests across Azure regions. Data Zone deployment types use their advertised EU or US boundary; regional deployment types use their advertised regional processing boundary. Choose and verify the actual deployment type and data boundary with your Azure administrator. A hostname, storage location or private endpoint does not prove the model's processing region.

The installer groups Cognitive Services and Azure OpenAI in its Azure AI tab, separate from the other Azure PaaS resources. The OpenAI section is specifically for the prompt-categorisation service: it provisions an optional Azure OpenAI account and model deployment, not a general-purpose deployment for other tasks or a full Foundry hub/project or agent platform. It also grants inference access and, when private networking is selected, creates the account private endpoint and the optional privatelink.openai.azure.com DNS zone. All tenant/resource values in any example below are synthetic.

Optional installer-managed service

The installer option is off by default. Select Enable prompt categorisation service, supply a unique resource name, and review the deployment settings. Browse available models... picks a model that Azure offers in your region; see Choose the model and where prompts are processed. The default deployment is prompt-categories, using gpt-4o-mini, the Azure default model version, regional Standard deployment SKU and capacity 1. Capacity units and throughput depend on the model; this is not a provisioned-throughput/PTU deployment. Regional availability, model availability, subscription access and quota can prevent deployment; the installer surfaces the Azure failure rather than substituting a different model or data boundary.

Provisioning creates or updates the configured account/deployment, grants the existing runtime service principal Cognitive Services OpenAI User, and sets FoundryPromptCategorisationEndpoint and FoundryPromptCategorisationDeployment on the web app (which also hosts the importer WebJob). It clears FoundryPromptCategorisationKey so authentication uses the runtime Entra credential. These settings identify this task-specific deployment, leaving future deployments for other tasks separate. Existing accounts with the configured name must be OpenAI accounts.

Disabling this installer option and applying an upgrade clears the three installer-managed prompt-categorisation runtime settings and their former FoundryPromptEndpoint / FoundryPromptDeployment / FoundryPromptKey aliases; it does not delete the account, deployment or historical taxonomy/classification data. This also clears manually supplied values for those settings, so reapply bring-your-own settings after an installer upgrade if that is your chosen configuration.

Provisioning is not consent to send prompts. The portal opt-in remains off on first use. Review governance, the taxonomy, sampling and costs, then enable it separately in Administration. Disabling/removing backend settings stops future model calls but does not rewrite a previously saved opt-in.

For an installer-managed deployment, no manual Foundry endpoint, key or web-app configuration is needed. Select Azure AI → Enable prompt categorisation service, enable Targets → Copilot AI interaction history, and after installation enable and save Administration → Prompt categories → Enable Foundry prompt categorisation. The Targets checkbox Copilot interactions is a separate audit-log import; it does not provide prompt text and is not enough for categorisation. The interaction-history import's Graph permission and other prerequisites still apply.

Choose the model and where prompts are processed

On Azure AI, Browse available models... lists the models that Azure offers in your subscription and region and that work with the categoriser. Fill in the installer identity, subscription and Azure region on the other pages first: the browser uses them to read Azure's model catalogue for that region. It creates nothing and sends no prompts, but the installer identity needs permission to read the catalogue at subscription scope, even if its other rights are limited to a resource group. If the lookup fails, or takes more than 90 seconds, the browser says so and offers Error details; you can still type the settings in by hand.

• The list opens on the deployment type already configured (Regional Standard unless you've changed it), with Small models for categorisation ticked. A small model is enough to classify prompts. Where Azure lists them for the region, the small models are GPT-4.1 nano, GPT-4.1 mini and GPT-4o mini; untick the box to add the compatible larger models, GPT-4.1 and GPT-4o. Mini and nano are a size guide, not a price quote, so check Azure OpenAI pricing and your subscription quota for the model you choose.
• Only model versions known to work with the categoriser's request (Chat Completions with a strict JSON schema) are listed. Reasoning models, deprecated models and versions whose compatibility isn't known are left out, so the list is deliberately not every model Azure offers.
• Use selected model fills in the model name, version and deployment type together, with a capacity that Azure accepts for that model. Cancel changes nothing. Capacity is a model-specific throughput (rate-limit) allowance, not provisioned throughput (PTUs) or a fixed monthly charge: you pay per token. The browser shows the capacity values Azure allows, when Azure reports them.

The deployment type decides where prompts are processed, so choose it deliberately:

Deployment type	Saved value	Where prompts and responses may be processed
Regional Standard (default)	Standard	In the resource's Azure geography. This can span regions within that geography.
Data Zone Standard	DataZoneStandard	Anywhere in the resource's Microsoft-defined data zone, not just its region.
Global Standard	GlobalStandard	In any Azure region worldwide.

Stored data stays in the resource's Azure geography whichever type you choose. If Azure lists no compatible model for the selected type, the browser says so and leaves the type alone: it never switches you to Data Zone or Global Standard. Check your data-residency requirements before choosing either, and see Microsoft's deployment-type guidance.

Deployment checks the choice again. Before it creates or changes the Azure OpenAI account, the installer re-reads Azure's catalogue and checks that the configured model, version, deployment type and capacity are offered. It checks the existing account's actual region if the account already exists, otherwise the configured region. Settings typed in by hand get the same check. If the combination isn't offered, the deployment stops with Azure does not list this model/version/deployment type in the account's region. Use Browse available models on Azure AI; do not change processing geography without approval., or with a message that the capacity isn't supported. Being listed isn't a guarantee: Azure can still refuse the deployment because of quota, current capacity or policy.

Prerequisites and configuration

1. Upgrade the analytics database. 202610080900001_AddPromptCategories follows 202610071400001_CopilotTurnPairing, which follows 202610071200001_Agent365PackageCatalog after 202610021200001_LicenceHistory. Run the matching manual scripts in that exact order if not using the installer. All three scripts are attached to Stable build 1852. See the release upgrade checklist. Each script applies/stamps or safely no-ops on rerun using its immediate predecessor's unchanged EF model snapshot.

2. Enable Targets → Copilot AI interaction history and grant its separate Graph application permission AiEnterpriseInteraction.Read.All. This is not the audit-log Copilot interactions checkbox. Configure UserGroupsFilter if a pilot is appropriate; user rotation and the existing per-cycle user cap still apply.

3. Either use the optional installer service above, or configure these runtime importer App Service settings for an existing compatible deployment:

   Setting	Meaning
   FoundryPromptCategorisationEndpoint	Azure OpenAI-compatible resource origin for prompt categorisation, for example https://contoso.openai.azure.com/. HTTPS port 443 only; no path, credentials, query or fragment. Azure *.cognitiveservices.azure.com resource origins are also supported when their deployments expose the compatible API.
   FoundryPromptCategorisationDeployment	Prompt-categorisation model deployment name, for example prompt-categories. The model must support strict JSON-schema structured output and temperature: 0 on API version 2024-10-21. Models that reject those parameters fail enrichment, not import.
   FoundryPromptCategorisationKey	Optional resource key for that endpoint, stored only as a protected runtime setting. Never put it in a taxonomy, source file or screenshot.

   The web app and importer read the APPSETTING_-prefixed environment value first, then the plain environment value, then XML configuration. For existing feature deployments, the former FoundryPromptEndpoint / FoundryPromptDeployment / FoundryPromptKey namespace is accepted only if all three new settings are absent. Namespaces are never mixed: a partially migrated new configuration does not borrow an old endpoint or key. Installer upgrades write the task-specific settings and clear the aliases. Saved installer JSON uses the FoundryPrompt* properties below; runtime App Service names are a separate namespace.

   The importer uses the solution's existing runtime service principal through ClientSecretCredential, not managed identity or DefaultAzureCredential. Give it Cognitive Services OpenAI User on the model resource. A configured key is tried first; any 401/403 switches that backend instance to Entra RBAC. DNS, timeout and service failures are not retried as authentication failures. Redirects are refused so credentials cannot be forwarded to another host.

4. The installer configures the web application's endpoint/deployment settings automatically for its managed service. For an existing deployment configured manually, set the corresponding values on the web application to allow the editor's initial opt-in. No key is sent to or returned by the portal. Portal readiness describes configuration, not a live connectivity or model-accuracy test; an existing opt-in can still be disabled if those settings are later removed.

5. Configure the solution's existing Storage connection string and give its runtime identity Storage Table Data Contributor. Configuration lives in the AnalyticsState Azure Table, partition PromptCategories, current row current and historical taxonomy-<version> rows. A missing storage connection has a process-wide memory fallback, but that fallback cannot authorise categorisation or save an opt-in. An unreadable or invalid configuration fails closed: no prompts are sent.

6. Open the administration editor, review the taxonomy and spending cap, then explicitly enable and save. Unsaved edits are not active. Saved changes take effect at the next import cycle: the current cycle keeps its captured opt-in, taxonomy and cap. Stop the Office365ActivityImporter WebJob to halt a current cycle immediately. Requests already submitted to the model service may still complete there. Administration app-role checks and the same-origin XHR requirement protect writes under the normal portal role-enforcement policy; leave EnforcePortalRoles enabled for governance. Administration does not imply See PII; this feature creates no per-person category API.

   Since Stable 1856 (PR #768): a save or Reset applies only to the settings the page loaded. If another administrator saved or reset them in the meantime, nothing is saved, the page shows The prompt categorisation settings were changed after this page loaded them, so nothing was saved. Reload to see the current settings, then make your changes again., and Reload shows the settings now in force. Builds before 1856 let a page left open overwrite a later change, which could switch categorisation, and with it the sending of prompts to the model, back on, or undo a spending cap.

The optional service adds persisted installer settings for enablement, resource name, deployment name, model name and version, deployment type and capacity. Schema 3.2.0 added the Foundry fields and 3.3.0 (Stable build 1852) added the deployment type; files saved without one keep regional Standard. Older files load with the service off. Stable 1856 and 1857 save schema 3.5.0, and Stable 1859 and later 3.6.0, which changes only how the secrets are encrypted; see Installer Configuration Workspace. The hosting requirement remains Windows / .NET Framework 4.8, not .NET 10.

Saved installer property	Default
FoundryPromptEnabled	false
FoundryPromptResourceName	empty
FoundryPromptDeploymentName	prompt-categories
FoundryPromptModelName	gpt-4o-mini
FoundryPromptModelVersion	empty — Azure default
FoundryPromptDeploymentType	Standard (Regional Standard); also DataZoneStandard or GlobalStandard
FoundryPromptCapacity	1

Categories and versioning

The default synthetic preset includes examples of both human-mode annotations:

Default categories	Illustrative human mode
Meeting summaries, document editing, information lookup, content drafting, analysis	Directing
Delegated research, delegated problem solving	Supervising
Other	Unassigned

Directing means specifying a task and steering the output, such as drafting an email and revising its tone. Supervising means delegating work with more autonomy and reviewing the result, such as asking for an investigation, a checked fix and results for approval. The delegated categories explicitly describe this intent; broad task categories can still occur in either workflow.

These are illustrative category-wide administrator labels, not evidence that the user actually worked in that mode. A single prompt may not establish the distinction. The Human mode info tooltip explains both modes and their limitations; choose Unassigned where unsure. Names and descriptions for the built-in preset are translated in English and Spanish. Customer-edited names and descriptions remain exactly as entered, including Unicode. Existing saved taxonomies are not overwritten: the new preset appears on first use or after Reset defaults and disable, which also turns categorisation off.

An administrator can define 2–20 categories, including the reserved stable id other and product label Other. Each id is a short lowercase ASCII identifier; each Unicode name is at most 100 characters and its goal description at most 500. Define goals, not named people or personal profiles. The optional human-mode metadata is directing, supervising, or unassigned. The model never infers that metadata: the importer copies the administrator's category annotation.

These labels are not the Work Trend Index's four modes. There is no agent-intensity model, conversation-level human-intensity calculation or Asking/Delegation/Exploration/Collaboration score. This is the deliberately limited relationship to #651. The preset is also not a claim of implementing O*NET or reproducing Microsoft's WTI methodology.

Changing an id, label, description or human annotation produces a content-addressed taxonomy version. Changing the opt-in or spending cap alone does not. Labels used by old facts are immutable; reports require one version at a time and never silently mix them. Simultaneous editor saves are last-completed-save-wins, but cannot rewrite an old version's meaning. Reload before editing if another administrator may be working on it.

Historical prompts cannot be recategorised. Their text was discarded. A failed or capped classification is not retried automatically from stored text, and the ordinary watermark overlap de-duplicates an already-imported interaction before enrichment. Do not interpret a taxonomy edit as a backfill.

Safety, cost and throughput controls

Each model request contains at most ten prompts, truncated to at most 5,000 UTF-16 code units without splitting a surrogate pair. Input is explicitly treated as untrusted data. The model receives no tools, no separately attached user identity, and no requested rationale. Strict JSON schema limits output to an allowlisted category-id array; the importer independently checks the exact input length and allowlist and never stores an unrecognised string. The schema follows Azure's supported structured-output subset, which does not support array-length keywords.

• Other means no goal matched, or the returned category/JSON failed the allowlist validation.
• Not classified is operational: missing text, the per-cycle cap, service refusal/content filtering, timeout, rate limiting or backend failure. It is not a goal category and has no human-mode annotation.
• Disabling the feature preserves the existing interaction import and creates no classification facts.
• The cycle defaults to at most 1,000 prompts, configurable from 1–10,000. This cap spans user chunks; it does not reset for every user.
• At most two model batches execute concurrently. Work is scheduled in bounded waves rather than creating one task per tenant user or prompt.
• A model request has a 20-second timeout. A 429 stops new model batches for the rest of that cycle; at most the other in-flight batch may complete. There are no rate-limit retry loops. New interaction statistics still import; unsent prompts are marked Not classified and their text is discarded.
• A timeout, access denial or service failure also opens a per-cycle circuit breaker. A dead backend therefore delays an import by at most the already in-flight requests, rather than repeatedly spending the timeout on every batch. A refusal or content-filter finish reason inside a successful completion remains batch-specific. HTTP error bodies are never read, so a prompt blocked with HTTP 400 cannot be distinguished from another HTTP 400 failure: it is recorded as service failure and stops new batches for that cycle.
• A key-authentication refusal can cause one additional HTTP attempt for the same prompt batch. Sent counts unique submitted documents; HTTP-attempt counts and reported input/output tokens are separate counters. Failed service calls do not always return token usage, so reported tokens are a lower bound, not a billing ledger.
• SQL fact writes are batched through a JSON parameter in slices of 500, not per-prompt EF queries or Add calls. Configuration is read once per cycle.
• Health's classified/token counts describe model processing, not SQL commit confirmation. A storage-failure means interaction statistics remain but some category facts were not saved; those prompts are not replayed.

At the 200,000-user design point the existing Graph user cap remains the first boundary, and the prompt cap is independent of directory size. The sample is neither a census nor a statistically random sample: first eligible new prompts within a cycle consume the category budget.