Agentic CI Architecture

[kpj2006]

brainstorming the agentic CI architecture and have a few questions:

1. Repository and Podman context:
   How much context about the client repository and Podman configuration should we provide to the agent? Rather than relying solely on logs, I’m wondering if providing relevant codebase context would help the agent make better predictions and diagnoses.
   if yes, what ideally and optimally can we provide.

2. Log handling:
   CI logs can contain thousands of lines of output. My current plan is to truncate them to the last ~50 lines before sending them to the agent. Is there a smarter windowing or log-filtering strategy you would recommend for each workflow to improve efficiency and reduce unnecessary context?

(like this summary of logs )

3. Preferred workflow/output:
   What kind of workflow would you prefer for the agent? Something closer to a Rabbit-style review, where it analyzes and review on issues, or more of a Codecov/socket-style report/dashboard?

4. Client-code failures:
   Sometimes a workflow may fail because of issues in the client’s code rather than Podman itself. Should the agent also analyze and report these failures, even when they are unrelated to Podman?

5. Model and compute resources:
   To what extent could the Podman Foundation provide access to models and/or computational resources for this project? For example, would there be support for purchasing model/API access or providing compute for applying agentic for client?

6. Existing work and references:
   If anyone has already explored or implemented something similar, I’d really appreciate it if you could share any relevant references, repositories, papers, or prior work.

7. evaluating the agent's analysis

   Do you have a historical set of known flaky failures/root causes that could be used as a benchmark, or should the project establish its
   own evaluation dataset and metrics for measuring the agent's accuracy or we need to write it manually.

8. What level of actionability would you prefer from the agent's output?
   I’m currently thinking about the agent not being limited to simply analyzing the failure and explaining its root cause. Depending on the type of failure, it could potentially provide multiple levels of output:
   first, a plain-English analysis identifying the likely root cause

second, a concrete mitigation or code/configuration suggestion that can be presented as a committable change, similar to CodeRabbit's "Committable suggestion", where the contributor can review the generated diff and commit it directly from pr;(whenever we are completely sure about fixture, not on every review).

and third, a structured prompt/context for an AI coding agent that containing the analysis, relevant logs, and suggested approach.