Description
When AgentCore Observability enables CloudWatch Transaction Search, it routes X-Ray spans to /aws/application-signals/data. CloudWatch Application Signals then automatically generates metric time series. Because it pre-generates metrics for every combination of API call made x 4 metric names (Latency, Error, Fault, Throttle), this can create many metrics, which are charged per month, and can be costly for agents that use many AWS APIs. There is no documented way to suppress metrics while leaving tracing, although removing the permission from the CloudWatch Logs resource for xray.amazonaws.com to write to /aws/application-signals/data while keeping aws/spans for tracing appears to work as an effective workaround.
Steps to Reproduce
The included zip file contains a reproduction that creates a new agent using the agentcore CLI, replaces its code with calls to list DynamoDB tables and S3 buckets and a second routine that lists only EventBridge subscriptions, runs the first, examines the created metrics (waiting for them to propagate), applies the workaround by removing the CloudWatch Logs resource for xray.amazonaws.com to write to /aws/application-signals/data , then invokes the agent's second routine and shows that the second set of metrics are not logged (but the traces are available).
metrics-repro.zip
Expected Behavior
Ideally the user does not want to create the expanded metrics (EMF), they only need the traces for AgentCore observability to work. Note this is an issue in X-Ray/CloudWatch Application Signals that requires additional coordination between the AgentCore runtime and X-Ray (X-Ray invokes CloudWatch Application Signals to form enriched metrics itself; AgentCore does not have, or does not invoke, a mechanism to suppress this. Since this can be costly, the AgentCore documentation should warn about Application Signals metric costs when enabling observability and the AgentCore CLI should provide without cost disclosure and require user permission when enabling Application Signals.
Actual Behavior
The AgentCore CLI automatically configures the AgentCore runtime for observability, which exports OTEL metrics, but also results in those traces being expanded combinatorily to a large set of metrics (currently billed at $.30/month each) and increases the cost to run the agent.
CLI Version
0.15.0
Operating System
macOS
Additional Context
No response
Description
When AgentCore Observability enables CloudWatch Transaction Search, it routes X-Ray spans to /aws/application-signals/data. CloudWatch Application Signals then automatically generates metric time series. Because it pre-generates metrics for every combination of API call made x 4 metric names (Latency, Error, Fault, Throttle), this can create many metrics, which are charged per month, and can be costly for agents that use many AWS APIs. There is no documented way to suppress metrics while leaving tracing, although removing the permission from the CloudWatch Logs resource for xray.amazonaws.com to write to
/aws/application-signals/datawhile keepingaws/spansfor tracing appears to work as an effective workaround.Steps to Reproduce
The included zip file contains a reproduction that creates a new agent using the agentcore CLI, replaces its code with calls to list DynamoDB tables and S3 buckets and a second routine that lists only EventBridge subscriptions, runs the first, examines the created metrics (waiting for them to propagate), applies the workaround by removing the CloudWatch Logs resource for xray.amazonaws.com to write to
/aws/application-signals/data, then invokes the agent's second routine and shows that the second set of metrics are not logged (but the traces are available).metrics-repro.zip
Expected Behavior
Ideally the user does not want to create the expanded metrics (EMF), they only need the traces for AgentCore observability to work. Note this is an issue in X-Ray/CloudWatch Application Signals that requires additional coordination between the AgentCore runtime and X-Ray (X-Ray invokes CloudWatch Application Signals to form enriched metrics itself; AgentCore does not have, or does not invoke, a mechanism to suppress this. Since this can be costly, the AgentCore documentation should warn about Application Signals metric costs when enabling observability and the AgentCore CLI should provide without cost disclosure and require user permission when enabling Application Signals.
Actual Behavior
The AgentCore CLI automatically configures the AgentCore runtime for observability, which exports OTEL metrics, but also results in those traces being expanded combinatorily to a large set of metrics (currently billed at $.30/month each) and increases the cost to run the agent.
CLI Version
0.15.0
Operating System
macOS
Additional Context
No response