ci: harden pipeline for secure, validated releases

[scotwells]

Why

Today a change can reach main — and an image can be published to oci://ghcr.io/datum-cloud/backstage-kustomize and ghcr.io/datum-cloud/backstage — without ever running our typecheck, lint, or unit tests, and the published artifacts carry no signatures, SBOMs, or provenance. This PR makes releases validated before merge, gated before publish, and verifiable after publish. It is a draft for discussion.

What changes

1. Merge gate: no PR merges without lint + typecheck + test + E2E green

• New CI workflow runs on every pull request: typecheck (yarn tsc:full), lint (yarn lint:all), and unit tests (yarn test:all, CI=true) as three parallel jobs.
• Node/Yarn setup is factored into a reusable composite action (.github/actions/setup-node-yarn) so CI and the publish gate share one source of truth.
• The existing E2E workflow (kind + CloudNativePG + smoke tests) continues to run on PRs. Together with CI these are the checks that should be marked required in branch protection.

2. Publish gate: broken images can't ship

• Publish Docker Image now depends on the CI checks and a full E2E run before it builds and pushes anything. Wired with needs: against the CI and E2E workflows (both are now reusable via workflow_call).
• E2E's standalone push: main trigger is removed because publish now invokes it on main/tags/releases, avoiding a duplicate kind run per push while keeping the same coverage.
• The existing reusable-workflow calls to datum-cloud/actions are untouched in behavior — only fronted by the gate.

3. Supply chain: sign, SBOM, attest

The reusable datum-cloud/actions@v1.16.0 publish-docker.yaml builds and pushes but does not sign, generate an SBOM, or attest (the attestations: write permission was granted but unused). A new sign-and-attest job fills the gap after the image is pushed:

• Cosign keyless signing (Sigstore/OIDC) of the image by digest — no long-lived keys or secrets required.
• SPDX SBOM generation with Syft.
• Build-provenance and SBOM attestations via GitHub's native attestation actions, pushed to the registry alongside the image.

4. Vulnerability scanning

• New Trivy filesystem scan on PRs, main, and weekly. It is non-blocking (exit-code: 0) and uploads SARIF to the Security tab, matching how we're introducing a new scanner without gating merges on pre-existing findings. Recommend flipping it to blocking once the backlog is triaged.
• Snyk already runs (and already blocks on high via the reusable default) — left as-is.
• Pinned the datum-cloud/actions reusable workflow calls (publish, snyk) to a commit SHA instead of the mutable v1.16.0 tag. Directly-referenced third-party actions in this repo were already SHA-pinned; new ones added here are pinned too.

Blocking vs. non-blocking

Check	Blocks merge	Blocks publish
CI (typecheck/lint/test)	yes (via required checks)	yes (needs:)
E2E (kind)	yes (via required checks)	yes (needs:)
Snyk (high)	yes (existing)	no
Trivy fs	no (report only)	no
Sign / SBOM / attest	n/a	runs after push

Preconditions for reviewers

• Branch protection: mark CI (typecheck, lint, test) and E2E as required status checks on main — the needs: gate covers the publish path, branch protection covers the merge path.
• Secrets: cosign keyless and GitHub attestations need only the built-in id-token/attestations/packages permissions (already declared) — no new secrets. SNYK_TOKEN remains a precondition for the existing Snyk scan.
• Org setting: the upstream snyk/actions/setup@master floating ref lives inside datum-cloud/actions and can't be pinned from this repo — worth a follow-up there.

Related

Cross-links datum-cloud/infra#2967 to operationalize the service catalog once the pipeline is hardened.

[github-advanced-security]

You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool.

What Enabling Code Scanning Means:

• The 'Security' tab will display more code scanning analysis results (e.g., for the default branch).
• Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results.
• You will be able to see the analysis results for the pull request's branch on this overview once the scans have completed and the checks have passed.

For more information about GitHub Code Scanning, check out the documentation.