You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This is a patch release on the previous 0.8.15 release, fixing the issue(s) below.
feat(test_tools): a failed scenario test leads with a short reason and the turn it failed at, short enough for pytest's one-line summary, followed by the judge's reasoning and the conversation with each turn's trace id; the harness's own frames are hidden from the traceback. --monocle-scenario-results (or MONOCLE_SCENARIO_RESULTS=true) also logs every result of a run to .monocle/test_results/scenarios_<time>.log. Off by default (#840)
fix(instrumentation): the LangChain inference span records every tool the model asked for, not just the first. A model can return several tool calls in one response, and entity.3.name and data.output both kept only tool_calls[0] — so a turn that booked a flight and a hotel together named one of them, even though both tools ran and both have their own agentic.tool.invocation span. One tool call still reads exactly as before; several are listed, so no existing trace changes shape. agent_inference_type likewise reports a delegation wherever the handoff sits in the list (#261, #841)
feat(instrumentation): the workflow span sets parent.is_remote = true when its parent came from another process, such as an A2A caller. An exported span keeps only parent_id, so a reader could not tell the root of this app's part of a shared trace from a span whose parent went missing. Spans with a local parent or no parent are unchanged (#841)
fix(instrumentation): the FastAPI http span records a usable url. It was built from the socket the server is bound to, so an app listening on IPv6 reported the unparseable http://::1:10333/. The Host header the client sent is used when present (http://localhost:10333/), and the socket fallback brackets an IPv6 host (#837)
feat(test_tools): scenario tests — describe an agent in a sentence and the goals people bring to it, and @MonocleValidator().monocle_scenario_testcase(seed) generates multi-turn test conversations from it: different kinds of users, users who hold details back until asked, requests the agent should refuse, and red-team attacks adapted from DeepTeam's templates. test_scenario_async(agent, agent_type, scenario) runs each one live against the agent as its own pytest case, with a simulated user playing the customer and an LLM judge deciding after each reply whether the scenario's success criteria are met — checked against the turn's actual tool calls, not the agent's claims. Generated scenarios are cached in scenarios.json beside the test, so later runs make no model call, and every conversation is traced as its own session. See test_tools/README.md (#830, #842)
chore(deps): monocle_test_tools now installs langchain, langchain-openai and pyyaml by default, for the scenario tests' simulated user, judge and seed files. The test_agent extra is kept, empty, so existing installs keep working (#830)
feat(instrumentation): while the MONOCLE_SUPPRESS_SPANS context value is True, instrumented calls still run but emit no spans — unlike MONOCLE_SKIP_EXECUTIONS, which skips the call. Scenario tests use it to keep the simulated user's and judge's own LLM calls out of the agent's trace (#830)
fix(instrumentation): LlamaIndex, LangChain and Haystack retrieval spans record the full retrieved context instead of its first 100 characters, so an eval of a RAG answer sees what the model was actually given; LlamaIndex also kept only the first document. All documents are joined and truncated only past MONOCLE_RETRIEVAL_OUTPUT_MAX_CHARS characters (default 65536). Retrieval spans are correspondingly larger; lower the cap to trim them (#829)
feat(instrumentation): detect Vertex AI Agent Engine as the app host. The workflow span reports entity.2.type = app_hosting.gcp_agent_engine and, as entity.2.name, the engine's full resource name projects/<project>/locations/<location>/reasoningEngines/<id> (the bare engine id when project or location is unset), built from the GOOGLE_CLOUD_AGENT_ENGINE_ID, GOOGLE_CLOUD_AGENT_ENGINE_LOCATION and GOOGLE_CLOUD_PROJECT the runtime sets. It is checked before any other host, since Agent Engine runs on Cloud Run and also sets K_SERVICE. Call setup_monocle_telemetry() in the agent module: adk deploy agent_engine --otel_to_cloud imports it after installing Google's tracer provider, so Monocle and Cloud Trace share that provider and its per-request flush (#828)
fix(instrumentation): a call in an unsampled trace no longer fails. Under the default ParentBased sampler an unsampled parent (for example an incoming traceparent with flags 00, which Agent Engine can send) makes every span a NonRecordingSpan, and span processing read its missing status, attributes and parent and raised, failing the application's own request. The sync, async and streaming wrappers now just run the call, and nothing is exported for it (#828)
feat(instrumentation): instrument the httpx client, as requests already is — a call carries the trace context out, so the service it reaches continues the caller's trace, and spans a trace-returning service appends to its response are taken off it. Only hosts named in MONOCLE_TRACE_PROPAGATATION_URLS are traced, so an app's other httpx traffic is untouched (#826)
feat(instrumentation): collect_returned_spans() hands a caller the spans a service returned, for the common case where an SDK owns the response and the caller never sees it. Works for requests and httpx alike (#826)
fix(exporters): span obfuscation recognises Okahu API keys (okh_…). Monocle exports to Okahu, so its key is the one most likely to reach a span, and it was the only major vendor shape missing. It was redacted when passed by name (api_key=…, which credential_assignment matches) but exported in the clear when passed positionally, where no name is available and only the shape can give it away (#807)
feat(exporters): span obfuscation replaces inline media with its type and size, so a multimodal call exports "url": "<IMAGE:image/png,1.4MB>" instead of megabytes of base64. Previously the whole encoded image, audio clip or PDF landed in data.input verbatim, matched by no credential pattern. Covers OpenAI's data: URLs (data_url) and the bare base64 Anthropic and Gemini send (base64_blob, any unbroken run of 512+ characters). Turn it off with patterns=list(CREDENTIAL_PATTERNS) (#807)
fix(exporters): span obfuscation redacts a credential named by its payload key, not just one recognizable from its text — {"api_key": "hunter2"} now exports "api_key": "<REDACTED>", where previously a bare secret with no key=value syntax was exported in the clear. Only a custom output processor produces such a key. The new credential_key pattern runs last, so {"api_key": "sk-proj-…"} keeps its precise <API_KEY> marker (#807)
feat(test_tools): called_any_agent(*agent_names) and called_any_tool(*tool_names, agent_name=None) assert that any of the named agents/tools was invoked, for runs where more than one route is acceptable. Names may be passed separately or as a single list; the context narrows to the union of the matched entities' spans in trace order, so the chained checks see the whole set, and count/min_count/max_count constrain the total invocations across them. Their negatives, does_not_call_any_agent / does_not_call_any_tool, assert that none of the named entities ran — one call to any of them fails the assertion, and the failure names what ran rather than the whole candidate list (#812)
feat(test_tools): add the a2a runner, which tests an agent over the A2A protocol by driving it as an A2A client (installed with the a2a extra). The agent's spans join the test's trace: set A2A_TRACE_WORKFLOW and the runner pulls them from Okahu, leave it unset and the test reads the agent's trace file. a2a_transport.make_traced_httpx_client gives the same propagation to an agent that calls another agent over A2A (#825)
fix(instrumentation): the A2A client span records the call again — get_url read a field the current a2a SDK does not have, and the status and response accessors raised on a Message result, an error, or a task with no artifacts. The span handler drops accessor errors silently, so the attributes were simply missing (#825)
fix(instrumentation): a Hugging Face inference turn that calls tools records them — data.output lists each call's id, name and arguments, where it was empty because a tool-call turn has no text. Streamed tool calls also keep their arguments, for OpenAI and Hugging Face alike: a streamed call arrives split across chunks, the first carrying the id and name and the later ones slices of the argument JSON, and only the first chunk was recorded (#823)
fix(exporters): span obfuscation falls back to the built-in credentials obfuscator when none of the configured ones can be loaded, instead of exporting payloads unscrubbed. Previously MONOCLE_SPAN_OBFUSCATORS=presidio without the obfuscation extra installed silently disabled obfuscation altogether; listing it alongside another obfuscator was unaffected. Only an explicit opt-out (MONOCLE_DISABLE_SPAN_OBFUSCATION, an off-switch value, or span_obfuscators=[]) turns scrubbing off (#805)
fix(test_tools): an Okahu eval reports a failed trace upload as the failure, naming the HTTP status (or "no HTTP response"), instead of marking the trace exported and failing later with "Unexpected response format ... Expected 'result' key in response". An eval whose response carries an empty result list now says the service returned no results for the fact, with the job id and the likely causes, instead of calling a well-formed response malformed. OKAHU_EVAL_TIME_PAD_SECONDS set but empty falls back to its default (#819, fixes #816)
feat(instrumentation): MS Agent Framework multi-agent orchestrations are traced. The coordinator of a group chat (GroupChatOrchestrator, AgentBasedGroupChatOrchestrator) or Magentic (MagenticOrchestrator) orchestration from agent-framework-orchestrations emits an agentic.invocation span with subtype routing each time it routes a message, named after the orchestrator, with the pattern (group_chat or magentic) as its description and the routed message as data.input. Instrumented only when the package is installed (#808)
This discussion was created from the release Version 0.8.16.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
This is a patch release on the previous 0.8.15 release, fixing the issue(s) below.
--monocle-scenario-results(orMONOCLE_SCENARIO_RESULTS=true) also logs every result of a run to.monocle/test_results/scenarios_<time>.log. Off by default (#840)entity.3.nameanddata.outputboth kept onlytool_calls[0]— so a turn that booked a flight and a hotel together named one of them, even though both tools ran and both have their ownagentic.tool.invocationspan. One tool call still reads exactly as before; several are listed, so no existing trace changes shape.agent_inference_typelikewise reports a delegation wherever the handoff sits in the list (#261, #841)parent.is_remote = truewhen its parent came from another process, such as an A2A caller. An exported span keeps onlyparent_id, so a reader could not tell the root of this app's part of a shared trace from a span whose parent went missing. Spans with a local parent or no parent are unchanged (#841)http://::1:10333/. The Host header the client sent is used when present (http://localhost:10333/), and the socket fallback brackets an IPv6 host (#837)@MonocleValidator().monocle_scenario_testcase(seed)generates multi-turn test conversations from it: different kinds of users, users who hold details back until asked, requests the agent should refuse, and red-team attacks adapted from DeepTeam's templates.test_scenario_async(agent, agent_type, scenario)runs each one live against the agent as its own pytest case, with a simulated user playing the customer and an LLM judge deciding after each reply whether the scenario's success criteria are met — checked against the turn's actual tool calls, not the agent's claims. Generated scenarios are cached inscenarios.jsonbeside the test, so later runs make no model call, and every conversation is traced as its own session. See test_tools/README.md (#830, #842)monocle_test_toolsnow installslangchain,langchain-openaiandpyyamlby default, for the scenario tests' simulated user, judge and seed files. Thetest_agentextra is kept, empty, so existing installs keep working (#830)MONOCLE_SUPPRESS_SPANScontext value isTrue, instrumented calls still run but emit no spans — unlikeMONOCLE_SKIP_EXECUTIONS, which skips the call. Scenario tests use it to keep the simulated user's and judge's own LLM calls out of the agent's trace (#830)MONOCLE_RETRIEVAL_OUTPUT_MAX_CHARScharacters (default 65536). Retrieval spans are correspondingly larger; lower the cap to trim them (#829)entity.2.type = app_hosting.gcp_agent_engineand, asentity.2.name, the engine's full resource nameprojects/<project>/locations/<location>/reasoningEngines/<id>(the bare engine id when project or location is unset), built from theGOOGLE_CLOUD_AGENT_ENGINE_ID,GOOGLE_CLOUD_AGENT_ENGINE_LOCATIONandGOOGLE_CLOUD_PROJECTthe runtime sets. It is checked before any other host, since Agent Engine runs on Cloud Run and also setsK_SERVICE. Callsetup_monocle_telemetry()in the agent module:adk deploy agent_engine --otel_to_cloudimports it after installing Google's tracer provider, so Monocle and Cloud Trace share that provider and its per-request flush (#828)ParentBasedsampler an unsampled parent (for example an incomingtraceparentwith flags00, which Agent Engine can send) makes every span aNonRecordingSpan, and span processing read its missing status, attributes and parent and raised, failing the application's own request. The sync, async and streaming wrappers now just run the call, and nothing is exported for it (#828)requestsalready is — a call carries the trace context out, so the service it reaches continues the caller's trace, and spans a trace-returning service appends to its response are taken off it. Only hosts named inMONOCLE_TRACE_PROPAGATATION_URLSare traced, so an app's other httpx traffic is untouched (#826)collect_returned_spans()hands a caller the spans a service returned, for the common case where an SDK owns the response and the caller never sees it. Works forrequestsand httpx alike (#826)okh_…). Monocle exports to Okahu, so its key is the one most likely to reach a span, and it was the only major vendor shape missing. It was redacted when passed by name (api_key=…, whichcredential_assignmentmatches) but exported in the clear when passed positionally, where no name is available and only the shape can give it away (#807)"url": "<IMAGE:image/png,1.4MB>"instead of megabytes of base64. Previously the whole encoded image, audio clip or PDF landed indata.inputverbatim, matched by no credential pattern. Covers OpenAI'sdata:URLs (data_url) and the bare base64 Anthropic and Gemini send (base64_blob, any unbroken run of 512+ characters). Turn it off withpatterns=list(CREDENTIAL_PATTERNS)(#807){"api_key": "hunter2"}now exports"api_key": "<REDACTED>", where previously a bare secret with nokey=valuesyntax was exported in the clear. Only a custom output processor produces such a key. The newcredential_keypattern runs last, so{"api_key": "sk-proj-…"}keeps its precise<API_KEY>marker (#807)called_any_agent(*agent_names)andcalled_any_tool(*tool_names, agent_name=None)assert that any of the named agents/tools was invoked, for runs where more than one route is acceptable. Names may be passed separately or as a single list; the context narrows to the union of the matched entities' spans in trace order, so the chained checks see the whole set, andcount/min_count/max_countconstrain the total invocations across them. Their negatives,does_not_call_any_agent/does_not_call_any_tool, assert that none of the named entities ran — one call to any of them fails the assertion, and the failure names what ran rather than the whole candidate list (#812)a2arunner, which tests an agent over the A2A protocol by driving it as an A2A client (installed with thea2aextra). The agent's spans join the test's trace: setA2A_TRACE_WORKFLOWand the runner pulls them from Okahu, leave it unset and the test reads the agent's trace file.a2a_transport.make_traced_httpx_clientgives the same propagation to an agent that calls another agent over A2A (#825)get_urlread a field the current a2a SDK does not have, and thestatusandresponseaccessors raised on a Message result, an error, or a task with no artifacts. The span handler drops accessor errors silently, so the attributes were simply missing (#825)data.outputlists each call's id, name and arguments, where it was empty because a tool-call turn has no text. Streamed tool calls also keep their arguments, for OpenAI and Hugging Face alike: a streamed call arrives split across chunks, the first carrying the id and name and the later ones slices of the argument JSON, and only the first chunk was recorded (#823)credentialsobfuscator when none of the configured ones can be loaded, instead of exporting payloads unscrubbed. PreviouslyMONOCLE_SPAN_OBFUSCATORS=presidiowithout theobfuscationextra installed silently disabled obfuscation altogether; listing it alongside another obfuscator was unaffected. Only an explicit opt-out (MONOCLE_DISABLE_SPAN_OBFUSCATION, an off-switch value, orspan_obfuscators=[]) turns scrubbing off (#805)resultlist now says the service returned no results for the fact, with the job id and the likely causes, instead of calling a well-formed response malformed.OKAHU_EVAL_TIME_PAD_SECONDSset but empty falls back to its default (#819, fixes #816)GroupChatOrchestrator,AgentBasedGroupChatOrchestrator) or Magentic (MagenticOrchestrator) orchestration fromagent-framework-orchestrationsemits anagentic.invocationspan with subtyperoutingeach time it routes a message, named after the orchestrator, with the pattern (group_chatormagentic) as its description and the routed message asdata.input. Instrumented only when the package is installed (#808)This discussion was created from the release Version 0.8.16.
All reactions