Skip to content

fix(ci): clean up orphaned apt/dpkg processes between Playwright install retries - #1265

Merged
Wikid82 merged 1 commit into
mainfrom
fix/apt-orphaned-lock-cleanup
Aug 19, 2026
Merged

fix(ci): clean up orphaned apt/dpkg processes between Playwright install retries#1265
Wikid82 merged 1 commit into
mainfrom
fix/apt-orphaned-lock-cleanup

Conversation

@Wikid82

@Wikid82 Wikid82 commented Aug 19, 2026

Copy link
Copy Markdown
Owner

Summary

PR #1262's re-run surfaced a new bug in the apt-reliability hardening from #1263, on both WebKit shards (shard 1, shard 2):

Attempt 1: hits the full 10-minute timeout (mirror slowness — not a full stall, #1263's actual target)
Attempt 2: fails in ~17s — E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 3951 (apt-get)
Attempt 3: fails in ~17s — same lock error

The outer timeout 10m wrapping npx playwright install --with-deps kills the top-level node process when it fires, but the underlying apt-get runs as root via sudo in a separate process tree that timeout doesn't reach. When attempt 1 genuinely times out, the orphaned apt-get process keeps running in the background holding the dpkg lock — so attempt 2 fails immediately on the lock, and so does attempt 3, burning all remaining retries in seconds with zero chance of success.

This is the same underlying class of bug as the nick-fields/retry EPERM issue fixed in #1259: killing a process that has escalated privilege via sudo doesn't reliably reach its children.

Fix

Before each retry (all 10 install sites), clean up any lingering apt/apt-get/dpkg processes and clear stale lock files so the next attempt starts clean instead of immediately failing on a lock held by the previous attempt's orphan:

sudo pkill -9 -f '(^|/)(apt|apt-get|dpkg)( |$)' 2>/dev/null || true
sudo rm -f /var/lib/dpkg/lock-frontend /var/lib/dpkg/lock /var/cache/apt/archives/lock 2>/dev/null || true
sudo dpkg --configure -a > /dev/null 2>&1 || true

Test plan

  • actionlint clean (including shellcheck on the embedded script)
  • lefthook run pre-commit clean
  • Confirmed all 10 install-retry sites got the cleanup step

…all retries

The outer `timeout 10m` wrapping npx playwright install --with-deps
kills the top-level node process when it fires, but the underlying
apt-get runs as root via sudo in a separate process tree that timeout
doesn't reach. When an attempt genuinely times out (a slow mirror,
not a full stall), the orphaned apt-get process keeps running in the
background holding the dpkg lock -- so the very next retry attempt
fails immediately with "Could not get lock /var/lib/dpkg/lock-frontend.
It is held by process <pid> (apt-get)", burning all remaining retries
in seconds with no chance of success.

Same underlying class of bug as the nick-fields/retry EPERM issue
fixed previously: killing a process that has escalated privilege via
sudo doesn't reliably reach its children. Before each retry, kill any
lingering apt/apt-get/dpkg processes and clear stale lock files so the
next attempt starts clean instead of immediately failing on a lock
held by the previous attempt's orphan.

Observed on PR #1262: both WebKit shards' first install attempt hit
the full 10-minute timeout (mirror slowness, not a hang), then both
retries failed in ~17s each with the dpkg lock error above.
@codecov

codecov Bot commented Aug 19, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Supply Chain Verification Results

⚠️ WARNING

📦 SBOM Summary

  • Components: 1753

🔍 Vulnerability Scan

Severity Count
🔴 Critical 0
🟠 High 2
🟡 Medium 5
🟢 Low 2
Total 13

📎 Artifacts

  • SBOM (CycloneDX JSON) and Grype results available in workflow artifacts

Generated by Supply Chain Verification workflow • View Details

@github-advanced-security

Copy link
Copy Markdown
Contributor

You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool.

What Enabling Code Scanning Means:

  • The 'Security' tab will display more code scanning analysis results (e.g., for the default branch).
  • Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results.
  • You will be able to see the analysis results for the pull request's branch on this overview once the scans have completed and the checks have passed.

For more information about GitHub Code Scanning, check out the documentation.

@Wikid82
Wikid82 merged commit 21fce8a into main Aug 19, 2026
47 of 48 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants