Skip to content

CI: the e2e appliance leg has no apt retry, and its abort message blames the network for a mirror mid-sync #442

Description

@VijitSingh97

What

The e2e appliance leg installs its prerequisites with a bare apt-get install and no retry, so an
Ubuntu archive mid-sync reds the job for a reason that has nothing to do with the tree — and the
message it aborts with names the wrong cause
, which is the part that costs a reader time.

Measured

Run 33938797652, attempt 1, job End-to-end (Docker), 2026-09-05T02:34:43Z, on a PR whose diff
touches only tests/run.sh. The in-container suite passed 88/0 in the same job — the failure is
in the E2E (appliance) leg that runs after it, against ubuntu:24.04:

E: Failed to fetch .../dists/noble-updates/restricted/binary-amd64/Packages.gz  Hash Sum mismatch
   Hashes of expected file:  - Filesize:1902505 [weak]  - SHA256:758ef89a...
   Hashes of received file:  - SHA256:d5cc29ce...      - Filesize:1902505 [weak]
   Last modification reported: Fri, 04 Sep 2026 19:06:24 +0000
   Release file created at:    Sat, 05 Sep 2026 01:12:57 +0000
E: Failed to fetch .../dists/noble-updates/main/binary-amd64/Packages.gz
FAIL: apt-get install jq gettext-base python3 failed (no network / archive down) — aborting the e2e run.

The index served is six hours older than the Release file that describes it. That is a mirror
mid-sync, and it clears on a re-run — this one did.

Two separate things to fix

  1. No retry. One apt-get update/install attempt against a public mirror is a coin flip on a
    bad day. A bounded retry with a backoff is the whole fix.
  2. The abort message misattributes it. "no network / archive down" is what the reader is handed,
    and neither was true: the archive answered every request and served a complete file. A reader who
    trusts it goes looking at runner connectivity or at an outage page, and finds nothing wrong. The
    message should say the install failed and leave the cause to the apt output above it, or name the
    mirror-sync case explicitly when apt reports a hash or size mismatch.

Note for whoever takes it

The sibling in the other repo (pithead #1802, "the image builds have no apt retry") is scoped to
apt-get update in image builds and its measured signature is File has unexpected size (N != N+1).
This one is a Hash Sum mismatch at an IDENTICAL filesize, so a retry keyed on the size symptom, or
a log matcher grepping for unexpected size, would not catch this half. Retry on the command failing,
not on a recognised message.

I have not measured a base rate for this repo — a single occurrence is what I have, and the pithead
issue's five-in-eighty figure was measured there, not here.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    infraDeployment, packaging, releasestestingTests, CI, and test infrastructure

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions