Skip to content

workflows: run external-package purge once per workflow, not per chunk#286

Merged
igorpecovnik merged 2 commits intomainfrom
purge-chunk-gate
Apr 19, 2026
Merged

workflows: run external-package purge once per workflow, not per chunk#286
igorpecovnik merged 2 commits intomainfrom
purge-chunk-gate

Conversation

@igorpecovnik
Copy link
Copy Markdown
Member

@igorpecovnik igorpecovnik commented Apr 19, 2026

Summary

  • infrastructure-download-external.yml's preclean / postclean didn't use CHUNK_INDEX to filter. Both built the same (release × package) matrix on every chunk invocation and ran repo.sh -c delete against the shared /publishing/repository-debs repo.
  • With the 4-chunk fanout in infrastructure-repository-update.yml, that was 4 parallel copies of aptly's delete op contending on the same repository state. aptly's lockfile usually saves it from corruption, but it's redundant work at best and a corruption hazard the moment a held lock times out or gets bypassed.
  • Gate both jobs on inputs.CHUNK_INDEX == 0 so the purge runs exactly once per workflow. Chunks 1..3 now skip preclean / postclean entirely and go straight to their slice of the download matrix.
  • Download parallelism is preserved — the download step already slices via CHUNK_INDEX and is the reason for the 4-way fanout. Purge goes back to being the single-threaded operation aptly's data model requires.

Test plan

  • Manual dispatch of Infrastructure: APT repositories update with purge_external=true — confirm Purge jobs show completed on chunk 0 only and skipped on chunks 1..3.
  • Confirm chunk 1..3's Mirror / download matrix jobs run as before (unaffected by the gate).
  • Confirm the Purge old <pkg> from <release> sub-matrix runs once per (release, package) tuple total, not 4× per tuple.
  • Post-run: aptly repo still lists the expected surviving versions of code, google-chrome-stable, etc. (one version each, older ones deleted).

The repository-update workflow fans out infrastructure-download-external
across 4 chunks so download-time aptly operations stay under
strategy.matrix's 256-entry cap, and the actual per-package download
step does use CHUNK_INDEX to slice its work.

preclean / postclean don't. Both build the same (release × package)
matrix on every invocation and run repo.sh -c delete against the shared
/publishing/repository-debs repository. At 4 chunks that meant four
parallel copies of aptly's delete op touching the same database at the
same time. aptly serialises via its own lockfile so it usually doesn't
corrupt, but it's wasted work at best and a corruption hazard the
moment a lock-wait times out or is bypassed.

Gate both jobs on CHUNK_INDEX == 0 so the purge runs exactly once per
workflow, independent of the download fanout. Chunks 1..3 now skip
preclean/postclean entirely and jump straight to their slice of the
download matrix. Download parallelism is preserved; purge goes back to
being the single-threaded operation aptly's data model requires.
@github-actions github-actions bot added size/small PR with less then 50 lines 05 Milestone: Second quarter release GitHub Actions GitHub Actions code Needs review Seeking for review labels Apr 19, 2026
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Apr 19, 2026

Walkthrough

A GitHub Actions workflow file was modified to change the conditional execution of preclean and postclean jobs. These jobs now require both inputs.PURGE == true and inputs.CHUNK_INDEX == 0 to execute, replacing the previous condition that only checked inputs.PURGE == true. This change restricts the purge operations to execute only during the first chunk of a chunked workflow run, preventing concurrent execution of purge operations across parallel chunks.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately reflects the main change: gating the purge operation to run once per workflow on chunk 0 instead of once per chunk, which is the primary purpose of this changeset.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description check ✅ Passed The pull request description clearly explains the problem (parallel purge operations on shared repository state), the solution (gating on CHUNK_INDEX == 0), and includes a test plan validating the changes.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch purge-chunk-gate

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In @.github/workflows/infrastructure-download-external.yml:
- Around line 138-143: Add a job-level concurrency block to the job that runs
when inputs.PURGE == true and inputs.CHUNK_INDEX == 0 so purge operations across
workflow runs are serialized; specifically, add a concurrency: group expression
that is enabled only when inputs.PURGE is true (e.g. derive a constant group
name like "infrastructure-purge-repo-debs" when inputs.PURGE is true) and set
cancel-in-progress: false to avoid cancels; update the job that contains the
CHUNK_INDEX gate (the job that performs the repo.sh -c delete / postclean steps)
to include this concurrency block so concurrent workflow runs do not race
mutations to /publishing/repository-debs.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 211f2441-05f2-4a25-bd86-787df5597081

📥 Commits

Reviewing files that changed from the base of the PR and between 719a9f7 and 6b20c76.

📒 Files selected for processing (1)
  • .github/workflows/infrastructure-download-external.yml

Comment thread .github/workflows/infrastructure-download-external.yml
The self-hosted 'repository' runner pool is shared across workflows
(and across runs of the same workflow). Several jobs already pre-chown
$GITHUB_WORKSPACE back to the runner user before actions/checkout@v6
fires — update-repository, Sync, Index — precisely because prior sudo
operations on tools/repository/repo.sh output can leave root-owned
files in the workspace. Checkout's default clean: true then fails
with EACCES unlink when it hits one:

  Error: File was unable to be removed Error: EACCES: permission denied,
  unlink '/home/actions-runner-23/_work/armbian.github.io/armbian.github.io/build/.coderabbit.yaml'

Copying and prepare-repos in the caller workflow, and postclean in
the reusable download-external workflow, were the three remaining
repo-runner jobs missing this guard. Add the same 'Fix workspace
ownership' step to each. Makes workspace state robust to whatever
the previous job left behind, without having to reason about which
specific sudo op produced the root-owned file.
@github-actions github-actions bot added size/medium PR with more then 50 and less then 250 lines and removed size/small PR with less then 50 lines labels Apr 19, 2026
@igorpecovnik igorpecovnik merged commit 264b125 into main Apr 19, 2026
5 checks passed
@igorpecovnik igorpecovnik deleted the purge-chunk-gate branch April 19, 2026 16:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

05 Milestone: Second quarter release GitHub Actions GitHub Actions code Needs review Seeking for review size/medium PR with more then 50 and less then 250 lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant