Skip to content

Added copa to the images build#247

Merged
bluvulture merged 78 commits into
5.xfrom
image_copa
Jul 21, 2026
Merged

Added copa to the images build#247
bluvulture merged 78 commits into
5.xfrom
image_copa

Conversation

@bluvulture

@bluvulture bluvulture commented May 26, 2026

Copy link
Copy Markdown
Contributor

Adds an opt-in, security-hardened image flavour to the release pipeline. Plain
images are always published as-is (even if they contain CVEs); Copacetic (Copa) then
produces a patched -hardened variant with fixable OS-level vulnerabilities
resolved. Every published image gets a Trivy-generated SPDX SBOM.

Users pick per image: plain (e.g. php8.5-debug-v5) or hardened
(php8.5-debug-v5-hardened).

Vulnerability scanning & patching

  • Trivy scans the built plain image for fixable OS vulnerabilities (--ignore-unfixed).
  • Copa patches those vulnerabilities into a new -hardened image.
  • A severity gate (fail_on_severity, default CRITICAL,HIGH, threshold semantics —
    naming a severity also gates everything above it; NONE disables) decides whether the
    hardened image passes. If the gate fails, the plain image still ships and only the
    hardened tag is skipped.

Local patching via the containerd image store

On the hardened (stable) legs the workflow enables Docker's containerd image store
(containerd-snapshotter) so Copa patches the locally built image through dockerd's
embedded BuildKit — no standalone BuildKit daemon and no registry round-trip. This replaces
the earlier standalone-buildkitd approach and means a publish: false dispatch runs the
full hardened path (build → patch → gate → SBOM) entirely on the runner, pushing nothing.

SBOM (compliance)

Trivy generates an SPDX SBOM for each published image (legal requirement). SBOMs are
attached as OCI referrers via oras (best-effort) and uploaded as workflow artifacts.

Publishing controls

  • publishdefault false: publishing is opt-in for manual workflow_dispatch runs.
  • publish_hardeneddefault false: the hardened image is always built, scanned and
    gated, but only pushed when explicitly enabled (requires publish: true). Scheduled/tag
    runs and dry-runs never push hardened tags.
  • Multi-arch: per-arch tags are merged into a logical multi-arch manifest only when both
    arches were pushed in the same run.

Housekeeping

  • Local images are removed after each matrix leg to conserve runner disk.
  • New CI scripts (scan-patch-gate.sh, attach-sbom.sh, normalize-severity.sh) ship with
    bash unit tests, and actionlint lints the workflows in CI.

Testing the hardened path without publishing

Trigger the workflow via workflow_dispatch with publish: false — the stable images are
built, Copa-patched, scanned and gated on the runner with nothing pushed. Use
publish: true + publish_hardened: false to publish plain tags while still building and
gating hardened locally.

Changed publishing process to publish only patched images
Copilot AI review requested due to automatic review settings May 26, 2026 09:52

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the Docker image release workflow to introduce an automated “scan + patch” stage (Trivy + Copa) before pushing images to registries, aiming to publish images after OS-level vulnerability remediation.

Changes:

  • Defaults manual (workflow_dispatch) runs to not publish images unless explicitly enabled.
  • Installs Trivy and Copa in the workflow, and starts a BuildKit daemon to support Copa patching.
  • Changes the build/publish flow to build locally, scan + patch the image, then tag and push all tags manually.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread .github/workflows/release.yml Outdated
Comment thread .github/workflows/release.yml Outdated
Comment thread .github/workflows/release.yml Outdated
Comment thread .github/workflows/release.yml Outdated
Comment thread .github/workflows/release.yml Outdated
Copilot AI review requested due to automatic review settings May 26, 2026 13:45

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 3 comments.

Comment thread .github/workflows/release.yml Outdated
Comment thread .github/workflows/release.yml Outdated
Comment thread .github/workflows/release.yml Outdated
Copilot AI review requested due to automatic review settings May 26, 2026 14:06

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 5 comments.

Comment thread .github/workflows/release.yml Outdated
Comment thread .github/workflows/release.yml Outdated
Comment thread .github/workflows/release.yml Outdated
Comment thread .github/workflows/release.yml Outdated
Comment thread .github/workflows/release.yml
Copilot AI review requested due to automatic review settings May 26, 2026 14:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 4 comments.

Comment thread .github/workflows/release.yml Outdated
Comment thread .github/workflows/release.yml Outdated
Comment thread .github/workflows/release.yml Outdated
Comment thread .github/workflows/release.yml Outdated
Copilot AI review requested due to automatic review settings May 26, 2026 15:15

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 4 comments.

Comment thread .github/workflows/release.yml Outdated
Comment thread .github/workflows/release.yml Outdated
Comment thread .github/workflows/release.yml
Comment thread .github/workflows/release.yml Outdated
Copilot AI review requested due to automatic review settings May 26, 2026 15:46

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 6 comments.

Comment thread .github/workflows/release.yml
Comment thread .github/workflows/release.yml Outdated
Comment thread .github/workflows/release.yml Outdated
Comment thread .github/workflows/release.yml Outdated
Comment thread .github/workflows/release.yml
Comment thread .github/workflows/release.yml Outdated
Copilot AI review requested due to automatic review settings July 20, 2026 14:42
bluvulture and others added 3 commits July 20, 2026 16:43
…efined')

The 5.x merge (c3586e2) reintroduced the full production matrix without the
per-entry 'hardened' field, but 5 steps still gate on matrix.build.hardened
-> actionlint failed (property not defined). Re-add hardened: true on the
stable v* legs (v2.3, v3.8 x8.2/8.3, v4.2, v5.2) and hardened: false on the
*.x dev legs, matching the README contract (-hardened is for stable release
tags; -dev tags are plain-only).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

.github/workflows/release.yml:406

  • always() is also true after an operator cancels the workflow, so process-tags can still publish logical manifests from whichever per-architecture artifacts completed before cancellation. Use !cancelled() here; it still allows this job to run after failed matrix legs (preserving plain-tag aggregation after gate failures) while honoring cancellation.
                uses: docker/setup-buildx-action@v4

Comment thread .github/workflows/release.yml Outdated
if compgen -G '.docker-state/*/gate_failed.txt' > /dev/null; then
echo "The following variants failed the severity gate; their -hardened tags were NOT published:"
grep -H . .docker-state/*/gate_failed.txt
echo "::error::One or more variants failed the severity gate (plain images were published as-is)"
Copilot AI review requested due to automatic review settings July 20, 2026 14:51

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

.github/workflows/release.yml:129

  • These Oras downloads run under set -e before any image is built or pushed, so a transient Oras release/checksum outage aborts the entire matrix leg. That makes OCI referrer attachment mandatory in practice even though the PR defines it as best-effort. Guard the Oras installation and continue with an explicit warning when it is unavailable; attach-sbom.sh can then skip attachment while SBOM artifact generation and image publishing continue.
                    curl -fsSL -o oras.tar.gz "https://github.com/oras-project/oras/releases/download/v${ORAS_VERSION}/oras_${ORAS_VERSION}_linux_${ORAS_ARCH}.tar.gz"
                    curl -fsSL -o oras_checksums.txt "https://github.com/oras-project/oras/releases/download/v${ORAS_VERSION}/oras_${ORAS_VERSION}_checksums.txt"

sparse-checkout-cone-mode: false

- name: Enable containerd image store
if: ${{ matrix.build.hardened }}
Comment thread .github/workflows/release.yml
Copilot AI review requested due to automatic review settings July 20, 2026 15:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

.github/workflows/release.yml:12

  • The new publishing controls leave the existing publish input defaulted to true on line 9. This contradicts the PR's stated opt-in/default-false contract and means a routine manual dispatch pushes all plain tags unless the operator explicitly disables publishing. Change that default to false so workflow_dispatch is a dry run by default.
            publish_hardened:
                description: 'Also publish the Copa-hardened (-hardened) tags. Requires publish=true. Off by default so hardened stays unpublished (built + scanned + gated but not pushed) on scheduled/tag runs and during testing, until explicitly enabled on a manual dispatch.'

…-run)

The severity-gate failure error claimed '(plain images were published as-is)',
which is false on a publish:false dry-run where nothing is pushed. Reword to
state only that the -hardened tags were skipped and plain handling is unaffected
-- accurate in both publish and dry-run modes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 20, 2026 15:19

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

.github/workflows/release.yml:12

  • The new publishing contract says manual publishing is opt-in, but the publish input above still defaults to true. A manual dispatch that accepts the defaults will therefore push every plain tag, contrary to the PR description and the documented publish: false default. Change that default to false.
                description: 'Also publish the Copa-hardened (-hardened) tags. Requires publish=true. Off by default so hardened stays unpublished (built + scanned + gated but not pushed) on scheduled/tag runs and during testing, until explicitly enabled on a manual dispatch.'

Comment thread .github/workflows/release.yml Outdated
Comment on lines +445 to +448
if ! docker buildx imagetools create \
--tag "$lt" \
"${lt}-amd64" \
"${lt}-arm64"; then
bluvulture and others added 6 commits July 20, 2026 21:13
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…empty)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ertion gap)

Final-review (Fable) found the S assertion was vacuous: 'pimcore/pimcore:php8.5-v5.2'
is a substring of the imagetools-create line already in the log, so it passed even if
attach targeted the child (…-amd64) ref. Replace with exact subject-binding assertions
per arch, and fix the garbled 'got 0\n0' count message. Mutation-verified: attaching to
${lt}-amd64 instead of $lt now fails both assertions.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 20, 2026 20:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

.github/workflows/release.yml:12

  • The PR description makes manual publishing opt-in (publish defaults to false), but the existing publish input still defaults to true at line 9. A manual dispatch that accepts the defaults will therefore push all plain images, contrary to the advertised safe dry-run default. Change that default to false.
                description: 'Also publish the Copa-hardened (-hardened) tags. Requires publish=true. Off by default so hardened stays unpublished (built + scanned + gated but not pushed) on scheduled/tag runs and during testing, until explicitly enabled on a manual dispatch.'

@bluvulture
bluvulture merged commit bb5ba49 into 5.x Jul 21, 2026
5 checks passed
@bluvulture
bluvulture deleted the image_copa branch July 21, 2026 06:11
@bluvulture
bluvulture restored the image_copa branch July 21, 2026 06:11
@github-actions github-actions Bot locked and limited conversation to collaborators Jul 21, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants