Repository navigation
Supply Chain Hardening Blueprint (English)
English version · Deutsche Fassung: Supply-Chain-Hardening-Blueprint
The supply-chain incident of March 2026 prompted me to harden the build and release chains of all my applications, from PaddleDoc to the internal tools. This post is the blueprint I followed. It is kept product-neutral and carries over to any application with an automated pipeline.
In March 2026, more than 2,500 companies and over 400,000 CI/CD pipelines were attacked through a single compromised open-source component. The pattern: a tool that build pipelines fetch unchecked in whatever version is "latest" was replaced with a malicious build. The malware then ran automatically inside the victims' release processes and stole credentials and signing keys. Not a single line had changed in the victims' own code.
This was possible because pipelines fetch their third-party components through mutable references ("take version 4", "take the latest") instead of immutable ones:
The decisive moment is step 3, the unverified fetch. Everything before it happens outside my sphere of influence. Everything after it is mere consequence. That transition is exactly where the blueprint intervenes.
No part of my build and release process may change without that change showing up on my side as a visible, review-required code change. Whoever compromises one of my suppliers must not automatically reach my deliverables.
The problem. CI pipelines are assembled from third-party building blocks:
checkout, build tooling, publish steps. They are usually referenced through
version labels like @v4. Technically, such a label is nothing but a movable
pointer. The vendor can repoint it at any time, an attacker holding the
vendor's credentials just as easily, and every pipeline will execute the new
code on its next run without asking. Repointing already-released labels was
the core of the incident. Real cases involving widely used CI components
existed before it.
The measure. Every third-party building block is pinned to the cryptographic fingerprint (commit hash) of an exact, reviewed state. A fingerprint cannot be repointed. It is the content. The human-readable version stays next to it as a comment. Completeness matters: the rule applies to all pipelines, especially the inconspicuous ones. Docs builds and helper workflows run with the same privileges as the prominent ones.
Effect and effort. A compromised vendor no longer reaches my pipeline without a visible change appearing on my side first. The migration costs less than an hour per application. Ongoing upkeep is handled by the automation in M5.
The problem. Pipelines download binary tools at runtime and execute them directly. HTTPS only protects the path, the source stays unverified. If the download server or the release artifact itself is compromised, the encrypted connection dutifully delivers the manipulated tool. Straight into a pipeline that then continues with publishing privileges.
The measure. For every tool fetched at runtime, I record the SHA-256 checksum the vendor publishes separately in the pipeline code. Before execution, the download is verified against it. Any mismatch fails the build hard.
Effect and effort. A manipulated artifact is caught before it runs. The effort is a few minutes per tool, once.
The problem. The shipped product consists largely of third-party packages. That includes the ones named directly and a tree of dozens to hundreds of transitively pulled-in packages. Every unpinned spot in that tree means: the next build may contain different code than the last one, without anyone having changed anything. The incident also demonstrated that package ecosystems can execute code at install time. A poisoned package never needs to be imported to do damage.
The measure. A lock file freezes the entire dependency tree, every package with its exact version and the checksums of its artifacts. Installation runs in strict mode: any artifact whose checksum is missing or different fails the build. That holds even at an unchanged version number and closes the second, subtler attack class, the swap of an already-published artifact.
Two details decide how much this is worth. First, I generate and validate the lock file in exactly the environment that builds the real releases: same OS, same architecture, same interpreter. Dependency resolution is platform-dependent, and a lock produced on a laptop can resolve differently in the release build. Second, strict resolution surfaces real inconsistencies. In my case, the newest release of a central package declared requirements that contradicted those of its own sub-dependency. The previous unpinned build had quietly resolved that conflict "somehow" for years. The lock makes such states visible and decidable.
Effect and effort. Builds become reproducible. Neither a malicious new version nor a swapped old artifact reaches the product unnoticed. The one-off cost is a few hours including validation in the build context. From then on, M5 maintains the file.
The problem. In many pipelines a single signal is enough to publish: pushing a release tag. The incident showed that precisely this signal can be forged. And even without an attacker: an untested release is an operational risk.
The measure. Before publishing, the release pipeline runs the complete automated verification suite, on exactly the state that is about to be released. Only a fully green run unlocks the publish steps.
Effect and effort. A manipulated or simply broken state can no longer be published silently. That costs a few extra minutes per release. This verification is what the release process exists for.
The problem. Pinned states age. Pinning without an update process trades supply-chain risk for the risk of outdated components. And taking updates automatically reopens the door just closed. The incident spread precisely through unchecked adoption of "the latest".
The measure. An automated service checks all pinned references weekly (pipeline building blocks, product dependencies, base images) and files a review-required change request for every available update: a visible diff, links to the release notes, an automatic test run. Nothing is adopted without my review. Auto-merge is switched off. The service also maintains the fingerprints from M1 and the lock files from M3, which keeps the hardening operable for good.
Effect and effort. Security and currency stop being a trade-off. The ongoing cost is the weekly review of the proposals. Minutes, not hours.
Every measure has a clear enforcement point, an enforcing party and a moment in time. That is what separates a catalog of controls from a statement of intent:
| Reference point | Before | After |
|---|---|---|
| CI building blocks (actions) | version label, movable after the fact | commit fingerprint, immutable (M1) |
| Runtime tools | downloaded and executed unchecked | checksum verification before execution (M2) |
| Product dependencies | partly unpinned, "latest wins" | version + checksum, incl. transitive packages (M3) |
| Release trigger | release signal suffices | release signal and green verification run (M4) |
| Updates | implicit with the next build | explicit, review-required change requests (M5) |
| A change at a supplier | flows in invisibly | shows up as a visible code change on my side |
I verified every pinned fingerprint independently against its original source, every checksum against the vendor's official publication. I tested the installability of the locked dependencies in the real build context before adopting the changes. The first real release after the migration went through the hardened chain end to end, including the quality gate and hash-verified installation of every locked package.
Honesty is part of any security concept worth relying on. What this blueprint does not do:
- Compromise before pinning. Pinning a state that was already malicious pins the damage with it. These measures freeze trust. They cannot create it. It has to be earned in the review of new states (M5).
- Quality of the human review. M5 replaces automatic adoption with a human decision. If I approve every proposal unread, the benefit evaporates.
- Compromise of my own accounts. Whoever gains write access to the repository or registry bypasses the supply-chain controls. Other layers help there: short-lived minimal-privilege tokens, tag and branch protection, two-factor authentication.
- Runtime behaviour. The measures secure the way into the artifact. What a legitimately included package does at runtime is outside their scope.
- Protect release tags and the main branch against retroactive overwriting (a platform setting, not code).
- Pin base container images by fingerprint.
- Remove "latest" defaults from deployment configurations.
- Cryptographically sign your own published artifacts.
- Inventory: record every place the pipeline fetches third-party content (CI building blocks, tool downloads, package installs, base images).
- M1: switch CI building blocks to commit fingerprints. The version number stays readable as a comment.
- M2: record the official checksum for every tool download.
- M3: raise unpinned installs to exact versions and move them into a hash-locked dependency file. Validate in the real build context.
- M4: make the publish pipeline depend on the full verification run.
- M5: enable automated update proposals, without auto-merge. They also maintain the fingerprints from M1 and the locks from M3.
- Verification: independently re-check fingerprints and checksums against the original sources before adopting the changes.
Initial effort is in the range of a few hours per application. What remains is the weekly review of update proposals.
Five snippets from the real pipelines, trimmed to the essentials.
# before: a movable label
- uses: actions/checkout@v5
# after: a commit fingerprint, the version stays readable as a comment
- uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0- name: Install Helm
run: |
curl -fsSL https://get.helm.sh/helm-v3.16.2-linux-amd64.tar.gz -o helm.tgz
echo "9318379b847e333460d33d291d4c088156299a26cd93d570a7f5d0c36e50b5bb helm.tgz" | sha256sum -c -
tar -xzf helm.tgz
sudo mv linux-amd64/helm /usr/local/bin/helmIf the checksum does not match, the build fails. A tool update is now a two-line diff: new version, new checksum.
COPY requirements-worker.txt /app/requirements-worker.txt
RUN pip install --no-cache-dir --require-hashes -r /app/requirements-worker.txtThe lock file is generated with pip-compile --generate-hashes in the same
container image that builds the releases. Any artifact without a matching
hash stops the installation.
jobs:
ci:
uses: ./.github/workflows/pr-ci.yml
publish:
needs: ciThe release tag alone no longer publishes anything. Only a green verification run unlocks the publish job.
version: 2
updates:
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"The same blocks exist for npm, pip and docker. Auto-merge stays off.
Dependabot is diligent, but it decides nothing. Every proposal takes the same path:
| Signal | Comes from | What needs to happen | When |
|---|---|---|---|
| Patch or minor update (npm, pip, actions) | Dependabot PR | read the diff and release notes, merge on green CI | in the weekly batch review |
| Major update or new base image | Dependabot PR | check breaking changes and test locally, merge only then | once there is time for a real test |
| Verification run red on main | CI notification | look immediately, fix or roll back | same day |
| Release tag pushed | myself | nothing. The gate decides: only a green run publishes | automatic |
| New dependency in code | development | pin and hash belong in the same commit as the dependency | before merge |
And when a proposal is rejected? The PR gets closed with a one-sentence reason. Dependabot only comes back once the next version is released.
Source on the incident: CloudSEK, "AI Supply Chain Breach — 2,500 Companies, 434,000 CI/CD Pipelines" (March 2026).