E2E test improvements: observability, metrics, Opus 4.8 support by TomasTomecek · Pull Request #556 · packit/ai-workflows

TomasTomecek · 2026-06-02T14:33:56Z

Summary

Batch of E2E test infrastructure improvements:

Observability: tag OTEL spans with jira.issue via AgentSpanProcessor context so the trace-server can index spans by issue key; flush TracerProvider explicitly at teardown
Metrics: track tokens and tool calls per test run
Opus 4.8 support: patch litellm provider ID (vertexai → vertex_ai) so adaptive thinking detection works for Claude 4.6+
Trace server: add logging
Test selection: allow running specific test cases instead of all by default
Noise reduction: suppress beeai/litellm logs below WARN level
Container lifecycle: don't auto-restart or remove e2e test containers (useful for post-run investigation)
Compose: mount whole ymir/ and scripts/ directories
New test case: add triage E2E test for RHEL-174694 (not-affected resolution), restructure assertions so package/fix_version are only checked for BACKPORT/REBASE resolutions

Test plan

make run-triage-agent-e2e-tests passes
OTEL spans carry jira.issue attribute in trace-server

🤖 Generated with Claude Code

gemini-code-assist

Code Review

This pull request updates the end-to-end triage agent tests to support the NOT_AFFECTED resolution. It introduces a new test case using NotAffectedData and restructures the assertions in test_triage_agent to be resolution-specific. Feedback suggests strengthening the assertions for the NOT_AFFECTED resolution by verifying the data type is NotAffectedData and ensuring justification_category is present.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

gemini-code-assist · 2026-06-02T14:35:03Z

+    elif expected_output.resolution == Resolution.NOT_AFFECTED:
+        # TODO: validate justification_category against a known set of values
+        assert real_output.data.explanation


To ensure robust validation of the NOT_AFFECTED resolution, we should explicitly verify that real_output.data is an instance of NotAffectedData (preventing Pydantic union coercion issues) and assert that justification_category is populated and non-empty, as it is a required metadata field for Red Hat's triage process.

Suggested change

elif expected_output.resolution == Resolution.NOT_AFFECTED:

# TODO: validate justification_category against a known set of values

assert real_output.data.explanation

elif expected_output.resolution == Resolution.NOT_AFFECTED:

assert isinstance(real_output.data, NotAffectedData)

assert real_output.data.explanation

assert real_output.data.justification_category

TomasTomecek · 2026-06-09T05:29:47Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces several enhancements to the observability and logging of the agents and the trace server. Key changes include adding structured logging to the trace server, expanding the metrics middleware to track token usage and tool calls, wrapping E2E test workflows with JIRA issue tracing contexts, and adding a LiteLLM provider ID workaround for Vertex AI. The review feedback highlights opportunities to make token extraction in the metrics middleware more robust against dictionary payloads, remove a duplicate assertion in the triage tests, implement the documented flushing and shutdown behavior in the OpenTelemetry fixtures, and correct the framework logger name to properly suppress verbose logs.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Signed-off-by: Tomas Tomecek <ttomecek@redhat.com> Assisted-by: Claude

otherwise we get a flood of logs Signed-off-by: Tomas Tomecek <ttomecek@redhat.com> Assisted-by: Claude Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

instead of running all by default Signed-off-by: Tomas Tomecek <ttomecek@redhat.com> Assisted-by: Claude

Signed-off-by: Tomas Tomecek <ttomecek@redhat.com> Assisted-by: Cursor

tokens, tool calls Signed-off-by: Tomas Tomecek <ttomecek@redhat.com> Assisted-by: Claude

Wrap each test case's run_workflow() call with _span_processor.jira_issue_context(issue) so that emitted OTEL spans carry the jira.issue attribute. Without this, the trace-server cannot index spans by issue key. Both triage and backport e2e tests are updated: - observability_fixture now yields the AgentSpanProcessor and stores it in a module-level _span_processor for test cases to use - TracerProvider is explicitly flushed and shut down at teardown since the container may be killed before atexit handlers run Signed-off-by: Tomas Tomecek <ttomecek@redhat.com> Assisted-by: Claude

it's really useful to investigate later Signed-off-by: Tomas Tomecek <ttomecek@redhat.com> Assisted-by: Claude

Signed-off-by: Tomas Tomecek <ttomecek@redhat.com>

Beeai uses "vertexai" as the litellm provider ID, but litellm only recognizes "vertex_ai" for adaptive thinking detection (Claude 4.6+). Without this, litellm sends thinking.type=enabled which opus-4-8 rejects. Signed-off-by: Tomas Tomecek <ttomecek@redhat.com> Assisted-by: Claude

Signed-off-by: Tomas Tomecek <ttomecek@redhat.com>

lbarcziova

lgtm!

lbarcziova · 2026-06-09T05:50:23Z

+            [
+                test_case.jira_issue,
+                m.get("agent_name", ""),
+                f"{m.get('duration', 0):.0f}s",
+                m.get("tool_calls", 0),
+                m.get("prompt_tokens", 0),
+                m.get("completion_tokens", 0),


lbarcziova · 2026-06-09T05:51:21Z

 .PHONY: run-triage-agent-e2e-tests
 run-triage-agent-e2e-tests:
-	$(COMPOSE) -f $(COMPOSE_FILE) --profile=e2e-test run --rm \
+	$(COMPOSE) -f $(COMPOSE_FILE) --profile=e2e-test run \


is there some case where we need it cleaned up? Should we document it?

I'm assuming that @TomasKorbar just disliked these containers to linger after a test run and that he had to manually remove them.

For me it's exact opposite, I used them to review the run and access the logs - the --rm was a blocker for me.

nforro

LGTM. You use both Claude and Cursor?

TomasTomecek · 2026-06-09T06:25:11Z

LGTM. You use both Claude and Cursor?

I used Cursor for one of the commits because I had it opened in an IDE: the logging addition to trace-server. Wanted to give it a try, it did okay but for other work it was poor so I switched back to Claude Code.

gemini-code-assist Bot reviewed Jun 2, 2026

View reviewed changes

TomasTomecek force-pushed the e2e-frr branch 2 times, most recently from 904d033 to 36c7674 Compare June 9, 2026 05:27

TomasTomecek changed the title ~~Add E2E triage test case for not-affected resolution (RHEL-174694)~~ E2E test improvements: observability, metrics, Opus 4.8 support Jun 9, 2026

gemini-code-assist Bot reviewed Jun 9, 2026

View reviewed changes

Comment thread ymir/agents/metrics_middleware.py

Comment thread ymir/agents/tests/e2e/test_triage.py Outdated

Comment thread ymir/agents/tests/e2e/test_triage.py

Comment thread ymir/agents/tests/e2e/backport_agent/test_backport.py

Comment thread ymir/agents/tests/e2e/conftest.py

TomasTomecek and others added 10 commits June 9, 2026 07:37

add e2e triage test case for RHEL-174694

6aa9a6b

Signed-off-by: Tomas Tomecek <ttomecek@redhat.com> Assisted-by: Claude

e2e: only log warn for beeai/litellm

189500e

otherwise we get a flood of logs Signed-off-by: Tomas Tomecek <ttomecek@redhat.com> Assisted-by: Claude Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

e2e: enable selecting test cases to run

5b44e5b

instead of running all by default Signed-off-by: Tomas Tomecek <ttomecek@redhat.com> Assisted-by: Claude

trace-server: add logging

b7026b4

Signed-off-by: Tomas Tomecek <ttomecek@redhat.com> Assisted-by: Cursor

e2e: add more metrics

e6afe50

tokens, tool calls Signed-off-by: Tomas Tomecek <ttomecek@redhat.com> Assisted-by: Claude

don't remove the e2e triage container

84ac511

it's really useful to investigate later Signed-off-by: Tomas Tomecek <ttomecek@redhat.com> Assisted-by: Claude

compose: mount whole ymir/ and scripts/

7f5c88f

Signed-off-by: Tomas Tomecek <ttomecek@redhat.com>

dont autorestart e2e test containers

544e584

Signed-off-by: Tomas Tomecek <ttomecek@redhat.com>

TomasTomecek force-pushed the e2e-frr branch from d75a2b9 to 544e584 Compare June 9, 2026 05:38

lbarcziova approved these changes Jun 9, 2026

View reviewed changes

nforro approved these changes Jun 9, 2026

View reviewed changes

TomasTomecek merged commit c95eb07 into packit:main Jun 9, 2026
9 checks passed

TomasTomecek deleted the e2e-frr branch June 9, 2026 10:01

qodo-for-packit Bot mentioned this pull request Jul 8, 2026

A couple of deployment changes and trace server improvements #665

Merged

Navigation Menu

Uh oh!

E2E test improvements: observability, metrics, Opus 4.8 support - #556

E2E test improvements: observability, metrics, Opus 4.8 support#556
TomasTomecek merged 10 commits into
packit:mainfrom
TomasTomecek:e2e-frr

TomasTomecek commented Jun 2, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Jun 2, 2026

Uh oh!

TomasTomecek commented Jun 9, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lbarcziova left a comment

Uh oh!

lbarcziova Jun 9, 2026

Uh oh!

lbarcziova Jun 9, 2026

Uh oh!

TomasTomecek Jun 9, 2026

Uh oh!

nforro left a comment

Uh oh!

TomasTomecek commented Jun 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

TomasTomecek commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

TomasTomecek commented Jun 9, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lbarcziova left a comment

Choose a reason for hiding this comment

Uh oh!

lbarcziova Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

lbarcziova Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

TomasTomecek Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

nforro left a comment

Choose a reason for hiding this comment

Uh oh!

TomasTomecek commented Jun 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

TomasTomecek commented Jun 2, 2026 •

edited

Loading