sudo apt-get update is flaky in Ubuntu 20.04

[magurotuna]

Describe the issue

We're using ubuntu-20.04-xl in GitHub Actions. Running sudo apt-get update within the runner, it occasionally fails (i.e. flaky) although the probability is not high.

When did the issue occur?

Not really sure, but we started encountering this issue ~5 days ago

Steps to Reproduce

Here's a GitHub Actions workflow file:

name: repro
on: [push]
jobs:
  repro:
    runs-on: ubuntu-20.04-xl
    steps:
      - name: step1
        run: |
          sudo apt-get update
          sudo apt-get install postgresql-client-14

Actual Result

Running this gave us the error as follows, sometimes, saying Hash Sum mismatch on armhf packages:

Run sudo apt-get update
Hit:1 http://azure.archive.ubuntu.com/ubuntu focal InRelease
Get:2 http://azure.archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:3 http://azure.archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Get:4 http://azure.archive.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Get:5 https://packages.microsoft.com/ubuntu/20.04/prod focal InRelease [3065 B]
Get:6 http://azure.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2345 kB]
Hit:7 http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu focal InRelease
Get:8 http://azure.archive.ubuntu.com/ubuntu focal-updates/main Translation-en [405 kB]
Get:9 http://azure.archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [16.3 kB]
Get:10 http://azure.archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1021 kB]
Get:11 http://azure.archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [237 kB]
Get:12 http://azure.archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [23.5 kB]
Get:13 http://azure.archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [24.9 kB]
Get:14 http://azure.archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B]
Get:15 http://azure.archive.ubuntu.com/ubuntu focal-security/main amd64 Packages [1968 kB]
Get:16 http://azure.archive.ubuntu.com/ubuntu focal-security/main Translation-en [322 kB]
Get:17 http://azure.archive.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [12.0 kB]
Get:18 http://azure.archive.ubuntu.com/ubuntu focal-security/universe amd64 Packages [792 kB]
Get:19 http://azure.archive.ubuntu.com/ubuntu focal-security/universe Translation-en [154 kB]
Get:20 http://azure.archive.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [17.0 kB]
Get:21 https://packages.microsoft.com/ubuntu/20.04/prod focal/main arm64 Packages [31.5 kB]
Get:22 https://packages.microsoft.com/ubuntu/20.04/prod focal/main armhf Packages [12.3 kB]
Err:22 https://packages.microsoft.com/ubuntu/20.04/prod focal/main armhf Packages
  Hash Sum mismatch
  Hashes of expected file:
   - Filesize:12258 [weak]
   - SHA512:74d0015f1291ffe0cad4626869cbcebf1d1549bf099c4ec25c8ab013c1eefdb587425ffdf3cdf69fab22ae8917eafb73cec2301368ff06a8b543ae0284ef9c0e
   - SHA256:92bd048a636060b0aa8421130bd9e6772990b4d74faa45cb25771fd5c85e1bd9
  Hashes of received file:
   - SHA512:57367683229dfc9887ad38622ac0733922775048e960c58008837a612b0acec86d06aa7c40030cd94a89f3050f83f16a6963f9245226ca0897eae9b66eedb9ba
   - SHA256:7a0258cee1ca6235c1ae8a56a3d030b09f43fdb814f904dfa7179fc8199ae235
   - Filesize:12258 [weak]
  Last modification reported: Fri, 03 Feb 2023 01:58:39 +0000
  Release file created at: Fri, 03 Feb 2023 09:04:19 +0000
Get:23 https://packages.microsoft.com/ubuntu/20.04/prod focal/main amd64 Packages [171 kB]
Err:23 https://packages.microsoft.com/ubuntu/20.04/prod focal/main amd64 Packages
  
Get:24 https://packages.microsoft.com/ubuntu/20.04/prod focal/main all Packages [2176 B]
Fetched 7895 kB in 2s (4658 kB/s)
Reading package lists...
E: Failed to fetch https://packages.microsoft.com/ubuntu/20.04/prod/dists/focal/main/binary-armhf/Packages.gz  Hash Sum mismatch
   Hashes of expected file:
    - Filesize:12258 [weak]
    - SHA512:74d0015f1291ffe0cad4626869cbcebf1d1549bf099c4ec25c8ab013c1eefdb587425ffdf3cdf69fab22ae8917eafb73cec2301368ff06a8b543ae0284ef9c0e
    - SHA256:92bd048a636060b0aa8421130bd9e6772990b4d74faa45cb25771fd5c85e1bd9
   Hashes of received file:
    - SHA512:57367683229dfc9887ad38622ac0733922775048e960c58008837a612b0acec86d06aa7c40030cd94a89f3050f83f16a6963f9245226ca0897eae9b66eedb9ba
    - SHA256:7a0258cee1ca6235c1ae8a56a3d030b09f43fdb814f904dfa7179fc8199ae235
    - Filesize:12258 [weak]
   Last modification reported: Fri, 03 Feb 2023 01:58:39 +0000
   Release file created at: Fri, 03 Feb 2023 09:04:19 +0000
E: Failed to fetch https://packages.microsoft.com/ubuntu/20.04/prod/dists/focal/main/binary-amd64/Packages.gz  
E: Some index files failed to download. They have been ignored, or old ones used instead.
Error: Process completed with exit code 100.

Expected Result

It always succeeds without any errors

Additional context

I reported this issue to actions/runner-images#7067, and was suggested opening an issue here.

[mbearup]

Thanks for reporting this issue. We recently moved to a pull-through cache to improve some elements of performance and storage allocation. We configured a low Time-To-Live (TTL) for the metadata, but we see that is insufficient, as artifacts can fall out of sync (i.e. client gets a fresh InRelease file but a cached Packages file). We're expediting a fix to disable caching of these metadata files, to ensure requesters always get the latest metadata.

[rowanseymour]

We're still seeing this today with Ubuntu 22.04, e.g. https://github.com/nyaruka/mailroom/actions/runs/4147292294/jobs/7174029289

[mbearup]

Thanks for reporting this, Rowan. We've developed the fix described above and are in the process of rolling it out to the mirrors. We must do this gradually (in accordance with Safe Deployment Procedures), but it is in progress.

[rowanseymour]

Still broken this morning.. hope the fix isn't too gradual...

Err:27 https://packages.microsoft.com/ubuntu/22.04/prod jammy/main arm64 Packages
  File has unexpected size (9590 != 9661). Mirror sync in progress? [IP: 13.90.21.104 443]

[mbearup]

We've actually rolled our delivery mechanism back due to a separate issue as of 1915UTC, so this should now be mitigated. Content should now be served via files on disk instead of pull-through cache.

[magurotuna]

Thanks!

[wbreza]

Broken this afternoon..

Fetched 8828 kB in 1s (7764 kB/s)
Reading package lists...
E: Failed to fetch https://packages.microsoft.com/ubuntu/20.04/prod/dists/focal/main/binary-armhf/Packages.gz  File has unexpected size (13238 != 13167). Mirror sync in progress? [IP: 40.118.250.56 443]
   Hashes of expected file:
    - Filesize:13167 [weak]
    - SHA512:fe7019ffc407e6690d9f12a69574e57913d201ce6032497576a861f21b506127814845d5f48bdaff27af30e7dde4d594234f25e7656f8168c5534a2b0835a437
    - SHA256:24f59d547e7178d668df67746905d3c74e982642bb3c902d31e6a215080191bf
   Release file created at: Wed, 08 Mar 2023 22:51:16 +0000
E: Failed to fetch https://packages.microsoft.com/ubuntu/20.04/prod/dists/focal/main/binary-arm64/Packages.gz  
E: Failed to fetch https://packages.microsoft.com/ubuntu/20.04/prod/dists/focal/main/binary-all/Packages.gz  
E: Failed to fetch https://packages.microsoft.com/ubuntu/20.04/prod/dists/focal/main/binary-amd64/Packages.gz