Skip to content

Add ApplyGuardrailScope for security guardrail evaluation tracing#259

Merged
nikhilNava merged 2 commits into
mainfrom
nikhilc/apply-guardrail-scope
Jun 1, 2026
Merged

Add ApplyGuardrailScope for security guardrail evaluation tracing#259
nikhilNava merged 2 commits into
mainfrom
nikhilc/apply-guardrail-scope

Conversation

@nikhilNava
Copy link
Copy Markdown
Contributor

Summary

Introduces a new OpenTelemetry tracing scope (\ApplyGuardrailScope) that captures security guardrail evaluations as spans. This enables observability into content safety, policy enforcement, and risk assessment decisions made during agent operations.

Port of microsoft/opentelemetry-distro-dotnet#109 (excluding ETW support, which is not applicable to the Python SDK).

Changes

New Contracts

  • *\GuardrailDetails* — Frozen dataclass capturing guardian evaluation metadata: target type, decision, provider info, policy details, and content hash for forensic correlation.
  • *\GuardrailFinding* — Represents an individual risk finding with severity, category, and confidence score.
  • *\GuardrailDecisionType* — Enum for well-known guardian decisions (Allow, Audit, Deny, Modify, Warn).
  • *\GuardrailRiskSeverity* — String constants for risk severity levels (None, Low, Medium, High, Critical).
  • *\GuardrailTargetType* — Constants for guardrail targets (LlmInput, LlmOutput, ToolCall, etc.).

New Tracing Scope

  • *\ApplyGuardrailScope* — Context manager scope following the same pattern as \InvokeAgentScope, \InferenceScope, and \ExecuteToolScope. Supports recording findings post-creation via
    ecord_finding(), updating decisions via
    ecord_decision(), and recording content output via
    ecord_content_output().

OpenTelemetry Constants

  • Added \microsoft.security.\ and \microsoft.guardian.\ attribute keys to constants.
  • Added \�pply_guardrail\ to the exporter operation name filter set.

Tests

  • 14 unit tests covering scope creation, finding recording, attribute emission, and edge cases.

Design Notes

  • Guardian spans are children of the operation they protect (e.g., inference or tool execution spans) via \SpanKind.INTERNAL.
  • Multiple guardian spans may exist under a single operation when guardrails are chained.
  • Follows the existing Core + Extensions pattern and context manager scope lifecycle.
  • All 250 existing observability core tests continue to pass.

Introduces a new OpenTelemetry tracing scope (ApplyGuardrailScope) that captures
security guardrail evaluations as spans. This enables observability into content
safety, policy enforcement, and risk assessment decisions made during agent
operations.

Port of microsoft/opentelemetry-distro-dotnet#109 (excluding ETW support).

New contracts:
- GuardrailDetails: Immutable dataclass capturing guardian evaluation metadata
- GuardrailFinding: Represents an individual risk finding with severity and score
- GuardrailDecisionType: Enum for guardian decisions (Allow, Audit, Deny, Modify, Warn)
- GuardrailRiskSeverity: Constants for risk severity levels
- GuardrailTargetType: Constants for guardrail targets (LlmInput, LlmOutput, etc.)

New tracing scope:
- ApplyGuardrailScope: Context manager scope with record_decision(),
  record_content_output(), and record_finding() methods

Infrastructure updates:
- Added microsoft.security.* and microsoft.guardian.* attribute keys to constants
- Added apply_guardrail to the exporter operation name filter set
- 14 unit tests covering scope creation, finding recording, and edge cases

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings May 28, 2026 17:06
@nikhilNava nikhilNava requested a review from a team as a code owner May 28, 2026 17:06
@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 28, 2026

⚠️ Deprecation Warning: The deny-licenses option is deprecated for possible removal in the next major release. For more information, see issue 997.

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds guardrail evaluation tracing to the observability core package, exposing new guardrail contracts and an ApplyGuardrailScope for OpenTelemetry spans around security/policy checks.

Changes:

  • Adds guardrail metadata/finding models and decision/severity/target constants.
  • Implements ApplyGuardrailScope with span attributes, finding events, decision updates, and output recording.
  • Updates exports, operation filtering, constants, and unit coverage for the new scope.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
libraries/microsoft-agents-a365-observability-core/microsoft_agents_a365/observability/core/apply_guardrail_scope.py Implements guardrail tracing scope and event recording.
libraries/microsoft-agents-a365-observability-core/microsoft_agents_a365/observability/core/constants.py Adds operation and guardrail/security attribute constants.
libraries/microsoft-agents-a365-observability-core/microsoft_agents_a365/observability/core/__init__.py Exports the new guardrail API surface.
libraries/microsoft-agents-a365-observability-core/microsoft_agents_a365/observability/core/exporters/utils.py Includes guardrail spans in exporter operation filtering.
libraries/microsoft-agents-a365-observability-core/microsoft_agents_a365/observability/core/guardrail_decision_type.py Adds guardrail decision enum.
libraries/microsoft-agents-a365-observability-core/microsoft_agents_a365/observability/core/guardrail_details.py Adds guardrail evaluation metadata dataclass.
libraries/microsoft-agents-a365-observability-core/microsoft_agents_a365/observability/core/guardrail_finding.py Adds guardrail finding dataclass.
libraries/microsoft-agents-a365-observability-core/microsoft_agents_a365/observability/core/guardrail_risk_severity.py Adds common risk severity values.
libraries/microsoft-agents-a365-observability-core/microsoft_agents_a365/observability/core/guardrail_target_type.py Adds common guardrail target values.
tests/observability/core/test_apply_guardrail_scope.py Adds unit tests for guardrail scope behavior and attributes.

The request.content field (InputMessagesParam) can be a string, list[str],
or InputMessages object. OTel attributes only accept primitive values or
primitive sequences, so structured objects would be silently dropped.

Now uses safe_json_dumps for non-string content, matching the pattern used
by InferenceScope for input message serialization.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@nikhilNava nikhilNava enabled auto-merge (squash) May 28, 2026 20:07
@nikhilNava nikhilNava merged commit 8f0b601 into main Jun 1, 2026
9 checks passed
@nikhilNava nikhilNava deleted the nikhilc/apply-guardrail-scope branch June 1, 2026 16:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants