Own DPkg::Lock::Timeout in Puppet and bound Exec[gha-boot-security-upgrade]

[akuzminsky]

Rewritten again 2026-07-31. My previous rewrite claimed the mechanism in the original report was disproven. It was not — I had tested with instances launched without user-data, which skips cloud-init's bootcmd, which is where the masking actually comes from. The original diagnosis was substantially right. Both earlier versions are in this issue's edit history.

The full chain

1. cloud-init masks the units. terraform-aws-cloud-init's bootcmd runs systemctl stop + systemctl mask on apt-daily{,-upgrade}.{service,timer} and unattended-upgrades.service on every boot, before runcmd starts ih-puppet (their Add jumphost profile and install pdsh in it #87 — those timers race Puppet for the dpkg lock). The five units are exactly $unmask_units in profile::unattended_upgrades.
2. Puppet unmasks them and starts the timers mid-catalog. So the unmask execs on lines 43–50 are live, and service { $enabled_units: ensure => running } does real work. The original report was right about this.
3. Starting a timer is when systemd evaluates its backlog. The AMI shipped /var/lib/systemd/timers/ stamps dated AMI build time, so Persistent=true scheduled a catch-up apt-daily-upgrade — jittered across RandomizedDelayUSec (1h), counted from the moment Puppet started the timer, i.e. mid-run.
4. The catch-up unattended-upgrade takes /var/lib/dpkg/lock-frontend, and every subsequent Package resource fails instantly, because apt-get does not wait for the lock.
5. A failed resource terminates the instance. ih-puppet apply passes --detailed-exitcodes and exits 4 or 6 on resource failures; ih-bootstrap.sh runs under set -euo pipefail with trap _ih_signal_abandon ERR, which calls ih-aws autoscaling complete <hook> --result ABANDON. Not a 1200s timeout — an immediate ABANDON.

Fixed upstream

infrahouse/infrahouse-ubuntu-pro#20, verified on ami-0ad90a31b549bce10:

• /var/lib/systemd/timers/ is cleaned before snapshotting. 0 of 10 stamps predate boot, LastTriggerUSec empty, no catch-up in the journal. Step 3 above no longer happens.
• The AMI also ships /etc/apt/apt.conf.d/99-lock-timeout with an unscoped DPkg::Lock::Timeout "300", so step 4 no longer fails instantly.

Policy decision

unattended-upgrades is wanted on these hosts and Puppet is authoritative for it. cloud-init's masking predates vulnerability management and is obsolete; it will be removed in infrahouse/terraform-aws-cloud-init#91. Until that lands the unmask execs are load-bearing and must not be removed; once it lands they become no-ops and can go.

Work in this repo

Implemented in environments/development, pending promotion to sandbox and modules/:

• profile::apt_lock_timeout, declared by profile::repos with stage => init so the drop-in exists before any Package resource in stage main. Manages the same path the AMI writes (two drop-ins setting the key would resolve by lexical filename order — a silent trap). Value from the apt_lock_timeout custom fact, defaulting to 300. This stops the guarantee depending on which AMI vintage a host happens to be running.
• Exec[gha-boot-security-upgrade] is now bounded. The retry logic moved into gha-boot-security-upgrade.sh with a hard cumulative deadline (480s, exec timeout 540) rather than tries/try_sleep, because exec's timeout is per-attempt and tries multiplies the worst case with no cap — and per step 5, an overrun terminates the instance rather than merely running long. Verified: gives up at its budget, retries through transient failure, and a hung command cannot overshoot the deadline.
• Promote to sandbox and modules/.
• After terraform-aws-cloud-init#91: drop the unmask execs.

Measurements

Against a held /var/lib/dpkg/lock-frontend on noble / apt 2.8.3:

	result
apt-get install, no options	exit=100, elapsed 0s
apt-get -o DPkg::Lock::Timeout=90 install	exit=0, elapsed 43s — waited
unscoped drop-in, no -o flag	exit=0, elapsed 45s — waited

Ubuntu's own default is binary::apt::DPkg::Lock::Timeout "120", scoped to the apt command only. Puppet's package provider, cloud-init and the AWS agents all use apt-get, which inherits nothing — which is why a host can appear to have a lock timeout while every Puppet Package resource still fails fast.

Three locks, not two

1. dpkg frontend lock — covered by DPkg::Lock::Timeout
2. apt lists lock (/var/lib/apt/lists/lock) — not covered; apt-get update fails in ~1s with the option set. This is why the boot upgrade needs retries rather than a longer timeout.
3. unattended-upgrades' own self-exclusion lock — not covered, and what made the boot exec return 1 in the original report.

Other boot-time lock contenders

Independent of unattended-upgrades, on every boot: inspectorssmplugin installs at ~T+56s and amazon-guardduty-agent at ~T+67s. These are unaffected by any of the above and are why the lock timeout matters even after the AMI fix.