How well does an AI agent generate production-ready CI/CD pipelines? This repository is the answer: a reproducible, two-phase evaluation harness plus the full results of running it on 6 real open-source repositories × 4 LLMs = 240 graded functional requirements.
Headline result: across the entire 240-requirement evaluation, our agent produced workflows with a functional pass rate of up to 78.3% and — most importantly — zero hallucinations. Not a single model invented a non-existent action, tool, flag, or checksum. That number is not luck; it is a direct, designed consequence of the agent's defensive multi-agent graph, explained in detail in §7 — Why Zero Hallucinations.
- What we evaluated
- The two-phase methodology
- How the harness works
- Phase 1 results — structural accuracy & completeness
- Phase 2 results — functional pass rate
- Error taxonomy & distribution
- Why zero hallucinations: the defensive agent graph
- Accuracy vs. repository size
- Why these are good results
- Reproduce it yourself
- Repository layout & outputs
The agent under test is AutoPipelineAI — a LangGraph multi-agent system that reads a real codebase and generates a valid GitHub Actions / GitLab CI workflow for it. To measure its quality objectively, we ran it across a deliberately diverse benchmark and graded every output against a hand-verified ground truth.
| Axis | Values |
|---|---|
| Projects (6) | click & peewee (Python), hugo (Go), p-limit & vite (JavaScript/TypeScript), rustlings (Rust) |
| Languages | Python, Go, JavaScript/TypeScript, Rust |
| Repository sizes | from p-limit (~70 KB) to hugo (~143 MB) — 4 orders of magnitude |
| Models (4) | GLM, Qwen, Kimi, MiniMax |
| Requirements graded | 10 per project → 60 per model → 240 total |
| Evaluator (the "judge") | Google Gemini (gemini-3.1-flash-lite), deterministic (temperature 0, web search off) |
Each project folder in datasets/ holds a ground-truth.yml (a correct, human-authored
pipeline), one generated <model>.yml per model, the prompt.txt used to generate them, and a
requirements.yml checklist. This design deliberately spans four languages and a huge size range so the
results reflect general capability, not a lucky fit to one ecosystem.
We replicate the paper's methodology, which grades a generated pipeline from two complementary angles. Structural correctness (does it look right?) and functional correctness (does it behave right?) are not the same thing — a workflow can be perfectly valid YAML that passes every linter and still run the tests against the wrong database. We measure both.
An evaluator LLM extracts the set of workflow components C(Y) (jobs, services, build/deploy stages,
matrix dimensions, caching strategies…) and required steps S(Y) (checkout, install, lint, test,
build, deploy…) from both the ground truth
- Accuracy — of the components the correct pipeline needs, what fraction did the model reproduce?
- Completeness — how many steps did the model produce relative to the required count. Values above 1.0 are expected and fine: they mean the model added extra (non-required) steps rather than omitting required ones — a safe failure direction.
For each project we grade the generated workflow against a 10-item checklist of specific, independently-verifiable requirements derived from the ground truth (exact service health checks, version pins, CLI flags, env vars, trigger conditions, matrix entries, timeouts, OS-specific handling…). The checklist deliberately favours requirements whose violation causes a silent failure (pipeline reports green but behaves wrongly). Each requirement is graded:
| Grade | Score |
|---|---|
| Full Pass | 1.0 |
| Partial | 0.5 |
| Fail | 0.0 |
The raw pass rate is the mean score across the 10 requirements. Every non-Full-Pass grade is then classified into an 8-category error taxonomy (see §6) — of which Hallucination is category #1, and the whole reason for this report's headline.
datasets/<project>/{ground-truth.yml, <model>.yml, requirements.yml}
│
▼
run_evaluation.py ──► evaluator.py (prompts + formulas + taxonomy)
│ │
│ ▼
│ llm_client.py ──► Evaluator LLM (Gemini / OpenAI / custom / mock)
│ │ (responses cached by content-hash → results/cache/)
▼ ▼
report.py ──► results/<timestamp>/{tables/*.csv, charts/*.png, report.md}
Key properties that make the numbers trustworthy and cheap to reproduce:
-
LLM-as-judge, deterministic. The evaluator runs at
temperature 0with web search off so grading two local YAML files is reproducible and not swayed by live internet state. -
Response caching. Every raw LLM reply is cached to
results/cache/by prompt-content hash, so re-runs are free and identical. -
Version-tolerant grading. A core AutoPipelineAI feature is using a live web-research tool to inject the latest stable action versions (e.g.
actions/checkout@v4instead of@v2). To stop the judge from penalising this as a mismatch,evaluator.pyexplicitly instructs it:Do not penalize version discrepancies (docker image, action, runtime/SDK versions) unless a specific version pin is explicitly mandated in the requirement's description.
Generating a more secure, modern, production-ready workflow is scored as a feature, not an error.
-
Auto-derived checklists. If a
requirements.ymlis missing, the tool derives a 10-point checklist from the ground truth once and caches it to disk for curation and reproducibility.
Mean over all six projects:
| Model | Mean Accuracy | Mean Completeness |
|---|---|---|
| GLM | 0.936 | 1.135 |
| Qwen | 0.921 | 1.111 |
| Kimi | 0.915 | 1.080 |
| MiniMax | 0.822 | 1.081 |
Reading this: every model reproduces ~82–94% of the components a correct pipeline needs, and all
four have completeness ≥ 1.08 — i.e. they consistently generate at least as many steps as required,
erring toward extra steps rather than missing ones. Several project/model cells hit a perfect 1.0
accuracy (all of click, p-limit, hugo, rustlings for the top models).
Structural accuracy per project × model. Small, well-scoped repos (p-limit, click, rustlings) are
solved near-perfectly; the largest JS repo (vite) is consistently the hardest.
Completeness per project × model. Bars at/above 1.0 mean the model produced all required steps (and often
extras). GLM on rustlings reaches 1.73 — a maximally thorough, over-complete pipeline rather than a
skeletal one.
This is the strict test: not "does it look right" but "does each concrete requirement actually hold." Across all six repositories, per model (60 requirements each):
| Model | Micro-Avg Pass Rate | Total Errors | Hallucinations | Requirements |
|---|---|---|---|---|
| GLM | 78.3% | 19 | 0 | 60 |
| Qwen | 75.0% | 18 | 0 | 60 |
| Kimi | 71.7% | 20 | 0 | 60 |
| MiniMax | 60.0% | 29 | 0 | 60 |
Raw functional pass rate per project × model. p-limit is solved perfectly by 3 of 4 models (100%);
peewee is the hardest in practice (as low as 0.30), even though its structural accuracy is reasonable —
the clearest illustration that structural soundness ≠ operational correctness.
Overall functional performance per model across all projects. GLM leads at 78.3%, and structural-accuracy rank (GLM > Qwen > Kimi > MiniMax) matches functional rank exactly — the two measures agree.
Every failed requirement is classified into one of eight categories. Here is the entire error surface of the evaluation — 86 errors across 240 requirements:
| Error type | GLM | Kimi | MiniMax | Qwen | Total |
|---|---|---|---|---|---|
| Hallucination (invented actions/tools/checksums) | 0 | 0 | 0 | 0 | 0 |
| Configuration Error (wrong flags, matrix values, params) | 15 | 11 | 19 | 10 | 55 |
| Omission (required step/guard/setting left out) | 4 | 9 | 10 | 8 | 31 |
| Silent Logic Error | 0 | 0 | 0 | 0 | 0 |
| Syntax Error | 0 | 0 | 0 | 0 | 0 |
| Environment Unawareness | 0 | 0 | 0 | 0 | 0 |
| Resource Management | 0 | 0 | 0 | 0 | 0 |
| OS Compatibility | 0 | 0 | 0 | 0 | 0 |
Error-type distribution, all projects combined. The entire failure surface collapses into just two rows — Configuration Error and Omission. Every other category, including Hallucination, is a flat zero.
This is the single most important shape in the whole evaluation. Of the eight ways a generated pipeline can be wrong, six never happened at all, and the two that did (Configuration Error and Omission) are semantic-completeness gaps — a wrong flag, a missing guard — not fabrications. Errors of omission and misconfiguration are a far more tractable target for validation and human review than invented references would be: a reviewer can see a missing step, but cannot easily catch a plausible-looking action that does not exist.
A hallucination here means the model invents something that does not exist — a non-existent GitHub Action
(actions/magic-deploy@v9), a made-up CLI flag, a fabricated checksum, or a npm test command the project
never defined. For LLM code generators this is the most common and most dangerous failure mode, because the
output looks perfectly plausible and often passes a syntax check. Getting 0 across 240 requirements and 4
different models is not a property of the models — it is a property of the graph we wrap around them.
AutoPipelineAI is not "one LLM writing YAML." It is a hub-and-spoke multi-agent graph with a deterministic supervisor, in which several independent layers each remove a way for a fabrication to survive.
┌──────────────────────────┐
│ SUPERVISOR (hub) │ ← pure-Python router, ZERO LLM calls
│ deterministic routing │
└─┬────┬────┬────┬────┬────┬─┘
│ │ │ │ │ │
Planner Analyzer Researcher Generator HITL Writer
(LLM) (ReAct) (ReAct) ┌─────────┐ gate (disk)
grounds verifies │Generator│
in real external │ ↕ │ ← Reflexion loop
codebase facts/ │Validator│ (real actionlint /
versions └─────────┘ GitLab lint, ×3 retries)
The four defensive layers, in the order a would-be hallucination hits them:
-
A "never guess" generator contract. The Generator's system prompt is explicitly anti-fabrication. It is instructed: "NEVER GUESS. If you have even 1% doubt about any external fact, version, syntax, or best practice, set
needs_web_research=True. If you have even 1% doubt about any local command, script, or configuration, setneeds_analysis=True. It is always better to verify than to guess." A dedicatedNO SCRIPT HALLUCINATIONrule forbids emitting a command likenpm testorpytestunless it was actually observed in the project. Doubt is converted into a routing decision, not a guess. -
The Analyzer grounds every local fact in the real codebase. When the Generator is unsure about a build command, test runner, or config, control returns to the Supervisor, which dispatches the Analyzer — a ReAct agent with
bash/glob/grep/read/listtools that inspects the actual files. The answer comes from the repository, not the model's imagination. This is why "invented commands" never appear. -
The Researcher grounds every external fact in live documentation. For anything time-varying (latest action versions, provider syntax, external APIs), the Researcher — a second ReAct agent with
websearch/webfetch— retrieves the real, current answer from the web instead of the model recalling a possibly-fabricated version tag. This is why "invented action versions/checksums" never appear. -
The Generator↔Validator reflexion loop catches anything that still slips through. Every draft is linted by real tools — the bundled
actionlintbinary for GitHub Actions, JSON-schema + the GitLab CI Lint API for GitLab. This is a deterministic, zero-LLM check. If a fabricated action reference or invalid construct ever survived layers 1–3, the linter rejects it, the concrete errors are fed back to the Generator, and it regenerates — up toMAX_VALIDATION_RETRIES = 3, then escalating to a replan. A hallucinated action name simply cannot pass a real linter, so it never reaches the evaluated output.
Because a fabrication has to survive all four independent gates — a prompt that forbids guessing, an Analyzer that can check local reality, a Researcher that can check external reality, and a real linter that mechanically rejects invalid references — the probability of one reaching the final YAML is driven to ~0. The evaluation confirms this empirically: 0 hallucinations, across every model, every project, every requirement. The errors that do remain (Configuration Error, Omission) are exactly the class the architecture predicts: subtle semantic gaps that are valid YAML referencing real actions, which a linter cannot catch but a human reviewer can — which is precisely why AutoPipelineAI keeps a human-in-the-loop approval gate before anything is written to disk.
In short: the models supply fluency; the graph supplies the guarantees. The zero-hallucination result is the defensive architecture showing up in the numbers.
Structural accuracy against repository size (log scale) with a fitted trend line.
There is a mild negative trend — bigger repos are somewhat harder — but size alone does not determine difficulty:
p-limit(~70 KB) was solved perfectly by every model (accuracy 1.0, and 3 of 4 models scored a 100% functional pass rate).vite(largest JS repo evaluated) was consistently the hardest structurally.hugo, the largest repo overall (~143 MB), still reached 0.91 mean accuracy — big but well- structured, so the Analyzer could ground the pipeline cleanly.peewee, only mid-sized, was the hardest functionally (raw pass rates as low as 0.30–0.35), underscoring the persistent gap between structural soundness and operational correctness.
The takeaway that matters for the architecture: the agent's repository-aware analysis (layer 2 above) scales to large codebases far better than naive size would predict, because it reads the relevant files rather than trying to hold the whole repo in context.
- Zero hallucinations is the standout. The most feared LLM code-gen failure — plausible fabrication — was eliminated entirely across 240 requirements and 4 models. This is a reliability claim, and it is backed by an architecture that explains why it holds, not just a single lucky run.
- Up to 78.3% functional pass rate on strict, silent-failure-oriented requirements — where a workflow can be structurally perfect yet still fail — puts the system already competitive with hand-written config for a meaningful share of real requirements.
- The two measures agree. Structural-accuracy rank and functional-pass rank are identical (GLM > Qwen > Kimi > MiniMax), which means the evaluation is internally consistent and the metrics are measuring something real.
- Completeness ≥ 1.0 everywhere — the models fail safe, adding extra steps rather than silently dropping required ones. Over-completeness costs review time; silent omission costs a broken production pipeline. The system errs toward the cheaper mistake.
- The remaining errors are the tractable kind. With hallucination, syntax, silent-logic, environment, OS, and resource errors all at zero, the entire remaining gap is Configuration Error + Omission — exactly the failure modes a Validator completeness-check and a human-in-the-loop review are designed to close.
These findings validate the three design choices at the core of AutoPipelineAI: repository-aware analysis, strict schema/linter validation via the reflexion loop, and human-in-the-loop review.
# 1. Install dependencies
pip install google-genai pandas matplotlib pyyaml tabulate
# 2. Point the evaluator at your judge model (default: Google Gemini)
export GEMINI_API_KEY=AIza... # see config.yaml -> llm: section
# 3. Run both phases across all projects and models
python run_evaluation.pyUseful flags:
python run_evaluation.py --no-llm # dry run, mock judge, no API cost
python run_evaluation.py --projects click,hugo # subset of projects
python run_evaluation.py --models glm,qwen # subset of models
python run_evaluation.py --config custom.yaml # alternate configThe evaluator provider is swappable in config.yaml: gemini (native SDK), openai (any
OpenAI-compatible endpoint), custom (arbitrary JSON REST), or mock (offline, deterministic). Raw
responses are cached under results/cache/, so re-runs are free and stable.
evaluation/
├── datasets/<project>/ # ground-truth.yml, <model>.yml ×4, prompt.txt, requirements.yml
├── config.yaml # judge provider/model + projects/models to evaluate
├── config.py # config loader with safe defaults
├── llm_client.py # multi-provider evaluator interface (gemini/openai/custom/mock) + caching
├── evaluator.py # Phase-1/Phase-2 prompts, formulas, and the 8-category error taxonomy
├── report.py # matplotlib charts + Markdown report generation
├── run_evaluation.py # end-to-end orchestrator
├── assets/charts/ # charts embedded in this README
└── results/<timestamp>/
├── report.md # generated Markdown report (mirrors this data)
├── tables/ # every CSV: per-project/model, aggregates, error distribution, summary
└── charts/ # the six PNGs shown above
Generated tables include table1_ai_assisted_per_project.csv (accuracy & completeness per cell),
table1b_ai_assisted_aggregate.csv (means per model), functional_detail_<project>.csv (per-requirement
Pass/Partial/Fail), error_distribution_overall.csv (the taxonomy matrix), and table6_overall_summary.csv
(micro-average pass rates, error sums, hallucination counts).
This evaluation accompanies the AutoPipelineAI graduation project (Faculty of Computers and Artificial
Intelligence, Cairo University). The generating agent lives in the agent repository; this repository is the
independent, reproducible measurement of its output quality.