build: rewrite CI to use matrices and cut down on workflow calls

[matt-codecov]

rewrites all our CI to:

• deduplicate _build-app.yml and the image-building parts of _self-hosted.yml into a single workflow: _build-images.yml
• deduplicate _push-env.yml and the image-pushing parts of _self-hosted.yml into a single workflow: _push-images.yml
• make any push to staging deploy both worker and api regardless of what the commit actually touched
• instead of having worker-ci.yml, api-ci.yml, and shared-ci.yml all call _build-images.yml (for instance), make _build-images.yml use a matrix to run on worker/api/shared

the main motivation is that we are pretty close to GHA's 20-workflow-call limit on PRs that need to run worker-ci.yml, api-ci.yml, and shared-ci.yml. in fact, when i tried to make staging pushes work better (#216), i hit the limit and the job wouldn't work. so, instead of calling like 6 workflows for worker, 6 for api, and 6 for shared for a total of 18, this PR makes us call 6 in total and have each workflow handle worker/api/shared internally.

to be honest, i think this is a worse setup than how it currently works. matrix shenanigans are complicated and very copypasty, it forces synchronization at the end of each "step" instead of allowing worker/api/shared to run start-to-finish in parallel, and it doesn't allow for any uniqueness for different projects. but, it takes us far below the workflow-call limit and it makes staging deploys work better.

the other option i tried was #234. it uses the gh cli to sidestep the workflow-call limit by using the REST API to do workflow dispatches instead. however, workflows started that way don't show up in the PR checks section, so you have to use the REST API to create your own checks. that's not great.

Legal Boilerplate

Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. In 2022 this entity acquired Codecov and as result Sentry is going to need some rights from me in order to utilize my contributions in this PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms.

[seer-by-sentry]

Sentry detected 2 potential issues in your recent changes

A potential issue exists where GitHub Actions workflows call non-existent Makefile targets, such as `worker.build.app`, because required Makefile include files are missing from the repository. This could lead to workflow termination.

• Description: The build and push workflows depend on Makefile targets (e.g., worker.build.app, worker.load.app) that are not defined. The root Makefile includes docker/Makefile.docker and docker/Makefile.ci-tests, but these files are missing from the repository. Consequently, when a workflow attempts to execute a command like make worker.build.app, the make utility will fail with a "No rule to make target" error. For example, if docker/Makefile.docker were present, it might define _load.app, but the workflow calls worker.load.app, which is not resolved due to the missing included Makefiles. This will cause the GitHub Actions workflow to terminate unexpectedly.
• Code location: .github/workflows/_build-images.yml:126
• Suggested fix: Ensure all Makefiles included by the root Makefile (e.g., docker/Makefile.docker, docker/Makefile.ci-tests) are present in the repository and define the necessary targets (e.g., worker.build.app, api.load.app) that the workflows depend on.

The fromJSON() function, used in conditional logic, may receive an empty string if the upstream jq command fails, leading to a JSON parsing error and workflow crash.

• Description: The jq command at app-ci.yml:75-78 generates a JSON string. If jq fails (e.g., due to malformed input from steps.filter.outputs.changes), the $matrix variable becomes an empty string. This empty string is then used as steps.strip.outputs.matrix. When this empty string is passed to fromJSON() in expressions like contains(fromJSON(needs.change-detection.outputs.changes), 'worker') (e.g., at app-ci.yml:147), fromJSON('') will cause a JSON parsing error. This error is not handled, leading to workflow failure. For example, if steps.filter.outputs.changes was {"invalid_json", jq would fail, $matrix would be '', and fromJSON('') would crash. This lack of error handling around JSON generation and parsing creates a single point of failure that could crash multiple workflows simultaneously.
• Code location: .github/workflows/app-ci.yml:147
• Suggested fix: Implement error handling for the jq command to ensure valid JSON output, or provide a default valid JSON array if jq fails. Add a check for empty string before calling fromJSON().

_{Did you find this useful? React with a 👍 or 👎}

To fix the issue, we need to avoid dynamically accessing secrets using secrets[matrix.codecov-instance.url]. Instead, explicitly define the secrets required for each possible value of matrix.codecov-instance.url and pass only those secrets to the workflow. This can be achieved by using conditional logic or matrix configurations to map specific secrets to specific scenarios.

For example, if matrix.codecov-instance.url can have values like url1 and url2, we should define separate secrets for each (secrets.CODECOV_URL1 and secrets.CODECOV_URL2) and explicitly reference them in the workflow.

To fix the problem, we need to replace the dynamic secret access (secrets[matrix.codecov-instance.token]) with explicit references to the required secrets. This involves defining specific secrets for each matrix configuration and referencing them directly. For example, if the matrix includes configurations for different environments or instances, secrets should be explicitly defined for each case (e.g., secrets.CODECOV_TOKEN_PROD, secrets.CODECOV_TOKEN_DEV, etc.).

The changes will involve:

1. Defining explicit secrets for each matrix configuration in the workflow file.
2. Replacing the dynamic secret access with direct references to these secrets.

To fix the issue, we need to avoid dynamically accessing secrets using secrets[matrix.codecov-instance.url]. Instead, explicitly define the secrets required for each possible value of matrix.codecov-instance.url and pass only those secrets to the workflow. This can be achieved by using conditional logic or matrix configurations to map specific secrets to specific scenarios.

For example, if matrix.codecov-instance.url can have values like url1 and url2, we should explicitly define secrets.CODECOV_URL1 and secrets.CODECOV_URL2 and use them directly in the workflow. This ensures that only the necessary secrets are passed to the runner.

To fix the issue, we need to avoid dynamically accessing secrets using secrets[matrix.codecov-instance.token]. Instead, explicitly define the secrets required for each matrix configuration. This can be achieved by adding specific secrets as inputs to the workflow or by using conditional logic to assign the correct secret based on the matrix configuration. This ensures that only the necessary secrets are passed to the workflow runner.

For example, replace the dynamic secret access with explicit references to specific secrets like secrets.CODECOV_TOKEN_PROD or secrets.CODECOV_TOKEN_DEV, depending on the matrix configuration.

---

To fix the problem, we need to replace the dynamic secret access (secrets[matrix.codecov-instance.url]) with explicit references to specific secrets. This ensures that only the required secrets are passed to the workflow runner, adhering to the principle of least privilege.

The best approach is to define separate secrets for each possible matrix.codecov-instance.url value and reference them explicitly in the workflow. For example, if the matrix can have values like instance1 and instance2, we should define secrets like CODECOV_INSTANCE1_URL and CODECOV_INSTANCE2_URL and use conditional logic to select the appropriate secret based on the matrix value.

To fix the issue, we need to replace the dynamic secret access (secrets[matrix.codecov-instance.token]) with explicit references to the required secrets. This involves defining specific secrets for each matrix configuration (e.g., GH_PAT_PROD, GH_PAT_DEV) and using conditional logic to assign the correct secret based on the matrix values. This ensures that only the necessary secrets are passed to the workflow runner.

---

To fix the issue, we need to replace the dynamic access to secrets (secrets[matrix.codecov-instance.url]) with explicit references to the required secrets. This involves defining specific secrets for each possible value of matrix.codecov-instance.url and using conditional logic to assign the correct secret based on the matrix configuration. This ensures that only the necessary secrets are passed to the workflow.

[codspeed-hq]

CodSpeed Performance Report

Merging #256 will not alter performance

_{Comparing matt/ci-rewrite (fdfadc2) with main (93a3ac0)}

Summary

✅ 9 untouched benchmarks

To fix the issue, we will add a permissions block at the root of the workflow file. This block will define the minimal permissions required for the workflow to function correctly. Based on the provided workflow, the contents: read permission is sufficient unless specific steps require additional permissions. If any job or step requires write permissions (e.g., for pull requests or issues), we will explicitly define those permissions at the job level.

---

To fix the issue, we will add a permissions block at the root of the workflow file. This block will define the minimal permissions required for the workflow to function correctly. Based on the workflow's operations (e.g., checking out code, caching, and interacting with Docker), the contents: read permission is sufficient for most steps. If specific steps require additional permissions (e.g., pull-requests: write), they can be added explicitly.

The permissions block will be added after the name field at the top of the file to apply to all jobs in the workflow.

---

[codecov-notifications]

Codecov Report

All modified and coverable lines are covered by tests ✅

✅ All tests successful. No failed tests found.

📢 Thoughts on this report? Let us know!

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.25%. Comparing base (9806daa) to head (fdfadc2).

✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #256   +/-   ##
=======================================
  Coverage   94.25%   94.25%           
=======================================
  Files        1215     1215           
  Lines       45114    45114           
  Branches     1437     1437           
=======================================
  Hits        42522    42522           
  Misses       2291     2291           
  Partials      301      301           

Flag	Coverage Δ
apiunit	96.52% <ø> (ø)
sharedintegration	39.78% <ø> (ø)
sharedunit	88.23% <ø> (ø)
workerintegration	61.64% <ø> (ø)
workerunit	90.60% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

To fix the issue, add a permissions block to the workflow. Since the workflow involves actions like checking out code, caching, authenticating to Google Cloud, and pushing Docker images, the permissions should be restricted to the minimum required. For example:

• contents: read for accessing repository contents.
• packages: write for pushing Docker images.
• Additional permissions can be added as needed for specific steps.

The permissions block can be added at the workflow level (applies to all jobs) or at the job level (specific to the push-image job). In this case, adding it at the workflow level is more concise and ensures all jobs inherit the same minimal permissions.

---

[Swatinem]

I think these changes make sense. The matrix definitions might be a bit annoying but I think its better that spreading this across different files.