cache errors recently?

[vsoch]

Hi! We are getting 1-2 errors nightly (seems to be picking up) for our automation, and it's usually some kind of checksum or connectivity thing (resolved by a rebuild)

Is something up with the cache? Thanks!

[vsoch]

This exact issue reproduced again: https://github.com/flux-framework/spack/actions/runs/8445698258/job/23133308663

With libsodium:

2024-03-27T02:22:41.1353021Z ==> Fetching https://download.libsodium.org/libsodium/releases/libsodium-1.0.19.tar.gz
2024-03-27T02:22:52.2296929Z ==> Error: FetchError: All fetchers failed for spack-stage-libsodium-1.0.19-hlfc52gsy2m7wobsae3cq53hf6zk3j5o
2024-03-27T02:22:52.2310003Z ==> Warning: Skipping build of libzmq-4.3.5-znuhqu2gjyu4f2r3vzoih5otkivy3a32 since libsodium-1.0.19-hlfc52gsy2m7wobsae3cq53hf6zk3j5o failed
2024-03-27T02:22:52.2319925Z ==> Warning: Skipping build of flux-core-0.60.0-ppkrdk4j2l3oqyjqodvhsblswvq4yzo5 since libzmq-4.3.5-znuhqu2gjyu4f2r3vzoih5otkivy3a32 failed
2024-03-27T02:22:52.2374607Z ==> Warning: Skipping build of flux-sched-0.33.0-qeqhkflaa5gvmvobujrfpknwobylecy2 since flux-core-0.60.0-ppkrdk4j2l3oqyjqodvhsblswvq4yzo5 failed

[vsoch]

Is there (or could there be) a parameter for N retries with exponential backoff?

[vsoch]

Happened again tonight, same libsodium

[vsoch]

Again tonight.

[vsoch]

This is still failing almost all our builds and updates, almost every night, reliably. 😢

2024-04-18T02:50:48.8777773Z ==> pkgconf: Executing phase: 'configure'
2024-04-18T02:50:52.5041740Z ==> pkgconf: Executing phase: 'build'
2024-04-18T02:50:54.0386857Z ==> pkgconf: Executing phase: 'install'
2024-04-18T02:50:54.2167672Z ==> pkgconf: Successfully installed pkgconf-1.9.5-ksvm2uifwg22nsuv2ed2nzcynvblcywx
2024-04-18T02:50:54.2169225Z   Stage: 0.17s.  Autoreconf: 0.00s.  Configure: 3.62s.  Build: 1.53s.  Install: 0.15s.  Post-install: 0.01s.  Total: 5.52s
2024-04-18T02:50:54.2170695Z [+] /opt/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.5.0/pkgconf-1.9.5-ksvm2uifwg22nsuv2ed2nzcynvblcywx
2024-04-18T02:50:54.2301267Z ==> Installing libsodium-1.0.19-hlfc52gsy2m7wobsae3cq53hf6zk3j5o [16/29]
2024-04-18T02:50:54.2302846Z ==> No binary for libsodium-1.0.19-hlfc52gsy2m7wobsae3cq53hf6zk3j5o found: installing from source
2024-04-18T02:51:22.9817435Z ==> Fetching https://download.libsodium.org/libsodium/releases/libsodium-1.0.19.tar.gz
2024-04-18T02:51:43.9374186Z ==> Error: FetchError: All fetchers failed for spack-stage-libsodium-1.0.19-hlfc52gsy2m7wobsae3cq53hf6zk3j5o
2024-04-18T02:51:43.9388585Z ==> Warning: Skipping build of flux-security-0.11.0-ikuaxnthcxmznytio7eismoijfsdawrt since libsodium-1.0.19-hlfc52gsy2m7wobsae3cq53hf6zk3j5o failed
2024-04-18T02:51:43.9477812Z ==> Installing zlib-ng-2.1.6-afecxcw2lltluaauxc6uzh7wmqrlsvaq [17/29]
2024-04-18T02:51:43.9479385Z ==> No binary for zlib-ng-2.1.6-afecxcw2lltluaauxc6uzh7wmqrlsvaq found: installing from source
2024-04-18T02:51:44.9989913Z ==> Fetching https://github.com/zlib-ng/zlib-ng/archive/2.1.6.tar.gz
2024-04-18T02:51:44.9990835Z ==> No patches needed for zlib-ng

Could this be added to the binary cache? Can we change the source of the install for libsodium to somewhere more reliable? Please?