The Self-Healing Pipeline

The book describes a pipeline that detects a production error, diagnoses its root cause, proposes a fix, and learns from what happens next, with a human in the loop for anything that matters. This repository is that pipeline as working code, plus every listing from the book, built so that you can experiment with it: change a threshold and watch the decision change, feed in your own incident, swap the AI model, then tear it all down.

• Nothing needs an AWS account until you decide you want one. The whole pipeline runs offline with a deterministic replay model.
• Nothing merges itself, at any confidence. The pipeline writes the pull request. A person clicks Merge.
• It has been run for real: real Sentry, real AWS, Bedrock, GitHub and Slack. Every hop was traced, and every log is saved in this repository (see Why you can trust it).

git clone https://github.com/dharmendradevops11/the-self-healing-pipeline.git && cd the-self-healing-pipeline
make venv && make demo

incident: AttributeError: 'NoneType' object has no attribute 'name'  (service demo-app, phase supervised, model replay)

=== result: pr_opened ===
root cause : get_display_name in src/auth.py dereferences session.user.profile, which is None for first-time OAuth users
confidence : model said 93, composite score 90 (high)
blast radius: low (1200 users)
decision   : auto_fix_pr, escalation tier informational -> platform-team
branch     : healer/2026-10-11-fix-src-auth-py-52ec9e
tests      : type-check: pass, unit: pass, lint: pass
pull request: .../prs/pr-1.md  labels: healer, healer: auto-fix-pr, healer: blast-radius-low, healer: human-review
audit steps: 6 written

It cloned a small app with a deliberate bug, asked a model for the cause, scored its own confidence, patched the bug on a healer/ branch, ran the app's tests, and wrote a pull request with a reviewer packet. Now change one input and watch the decision change:

For the "less sure" row, make a copy of the recorded model reply with a lower score and run that:

sed 's/"confidenceScore": 93/"confidenceScore": 20/' scenarios/replays/null-profile.json > /tmp/unsure.json
python -m healer.demo --replay /tmp/unsure.json

There are eight guided experiments in docs/experiments.md.

The architecture is Figure 2.1 of the book, with the plumbing the figure leaves out drawn in: an alert is verified, buffered and handed to a workflow (top), the workflow investigates and then either proposes a fix or escalates (middle), and a person decides (bottom). Every box inside the dashed frame is a Step Functions state you can click on in the AWS console.

The gate is the heart of it. The model's own confidence is never trusted alone (Chapter 9):

A model that says 95 about an error it has no evidence for gets a composite below 60, and the run goes to a human with a report, not to a fix generator. This was seen on real AWS: docs/aws-walkthrough.md.

It was run for real, and the evidence is in the repo.

The walkthrough follows one alert through every hop (webhook, SNS, SQS, the pipe, each Step Functions state, each Fargate task, the pull request, Slack, the ledger) with the request or log line at each step and a link to where to see it in the AWS console: docs/aws-walkthrough.md.

Safety is designed in, not bolted on:

You need Python 3.11 (through uv). Docker is needed only for LocalStack.

make venv        # Python 3.11 environment with the book's pinned versions
make test        # the whole test suite, no AWS and no Docker
make demo        # the pipeline end to end with a replay model and real git

make local-up    # LocalStack 3.4: S3, DynamoDB, SQS, SNS, Secrets Manager (Chapter 17, "Start Local")
python -m healer.demo --localstack
make local-test  # the integration tests

Store your secrets first, then deploy. Both are step by step in docs/deploy.md.

./scripts/deploy.sh -c healer:github_repo_url=https://github.com/you/your-repo \
                    -c healer:github_secret_arn=<arn> -c healer:phase=shadow \
                    -c healer:bedrock_model_id=<a model enabled in your account>

Start in shadow mode: the pipeline analyses every failure and records what it would have done, without acting. Connect Sentry with docs/sentry-setup.md. When you are done, ./scripts/teardown.sh removes everything (docs/teardown.md).

Cost. Local mode is free. On AWS a run is a few cents, almost all of it the model; an idle deployment costs about a dollar a month (two secrets). Set healer:billing_alarm_usd to be told if that changes.

scripts/test_flows.py drives each flow on a deployed copy: a fix, an escalation, a forged webhook, a duplicate alert, the learning loop. docs/aws-walkthrough.md has the table of commands and what you should see.

Every code listing from the book is in chapters/, one folder per chapter, in reading order. Most are excerpts that illustrate one idea; the ones that are complete, runnable Python are marked in each chapter's README. companion_code/ holds the reference implementations indexed in Appendix D, with tests. The book is the specification: docs/BOOK_SPEC.md lists what it promises and where this repository stands, and docs/book-to-code.md maps each chapter to its code.

This section is deliberately plain, because the book is.

• A reference implementation, not a product. It is built to teach and to experiment with, and to be read. Review and test everything in a staging account before it goes near production.
• Proven on real AWS: the webhook, the fix path, the escalate path, reviewer feedback into the ledger, shadow mode, the daily cap, Slack, and teardown. Not yet exercised there: the autonomous phase, multi-region Bedrock, and the model fallback.
• One repository per deployment, and the stack trace parser is tuned to the demo app's Python and Node frames.
• The confidence score is a heuristic, not a probability. The weights (30% evidence, 45% model certainty, 25% history) come from the book and have not been calibrated against measured outcomes, and a model's stated certainty is not a measured chance of being right. Treat 90 as "the evidence is strong", never as "90% likely correct". No precision or false-positive figures are published yet.
• Generated code runs only in a throwaway Fargate task with a path allowlist, a read-only view of the tests and no production credentials, but a path allowlist alone does not make arbitrary code safe to run. Keep review mandatory.
• The learning loop starts empty. The history input to the confidence score only activates once a failure category has five resolved reviewer verdicts.
• Slack message and notification text include the analysis and a link, and nothing from your secrets, but the incident text itself goes to the channel you choose.
• Where this repository and the book disagree, the book is right and the repository is the thing to fix. The deliberate differences, and the places where the book disagrees with itself, are in docs/DECISIONS.md and docs/BOOK_SPEC.md.

Corrections and issues go to this repository's issue tracker, and revisions are logged in .manuscript_changelog.md, as Appendix I describes. Tested versions follow Appendix E. See CONTRIBUTING.md before opening a pull request.

The code is licensed under the Apache License 2.0, see LICENSE. It is provided for educational and reference purposes, without warranty of any kind.