Snap: Caddy has no armhf archive, and a store hiccup is not a bad snap.

Two failures in the v10.78 snap jobs, neither of them a problem with
WeKan.

ARMHF ASKED FOR A CADDY ARCHITECTURE THAT DOES NOT EXIST. With the
libcurl4t64 fix in, the armhf build got past the stage-packages and died
further along:

    :: Downloading Caddy 2.11.4 (linux/armhf) from GitHub releases...
    :: + curl -fsSL -o /tmp/caddy.tar.gz https://github.com/caddyserver/
       caddy/releases/download/v2.11.4/caddy_2.11.4_linux_armhf.tar.gz
    :: curl: (22) The requested URL returned error: 404
    'override-build' in part 'caddy' failed with code 1.

Caddy is built by Go and its release assets carry GO's architecture
names, not Debian's. There is no linux_armhf archive and there never has
been - the 32-bit ARM ones are armv5, armv6 and armv7 - and the case
statement had no armhf branch, so it fell through to a default that
passed the Debian name straight into the URL. The "fall back to the
pinned version" path then retried the SAME wrong name, so the failure
read as "Caddy stopped publishing this architecture" when it was this
file's mapping all along.

armhf maps to armv7, not armv6: Debian armhf's baseline is ARMv7-A with
VFPv3-D16 hard-float, and Go's armv7 build is GOARM=7, which is exactly
that. This is NOT the armhf/armv7 distinction that matters for Node.js
in wekan/node-patches - that one is about NEON, and GOARM=7 does not use
NEON. Checked against the actual release: all six mapped URLs answer,
and linux_armhf 404s.

The default branch now names the problem and stops, instead of guessing
a name and letting a 404 blame the wrong project.

THREE GOOD SNAPS WERE LOST TO A STORE HICCUP. riscv64, ppc64el and s390x
each built on Launchpad, downloaded, and were then refused:

    Status: error while processing
    Issues while processing snap:
    - binary_sha3_384: Error checking upload uniqueness.

That is the store failing its OWN duplicate check on a digest it had
just computed - a server-side error, not a bad snap - and the message
the job printed was about 'is not a valid file', credentials and ACLs,
none of which applied. The upload is retried three times with a backoff
now, and the give-up message says the snap is fine and nothing here
needs changing.

The retry stays narrow on purpose: a rejected file, unparseable
credentials or a missing ACL will be rejected identically three times,
and retrying those only buries the one message that says what to fix.
The classifier is tested against all four, not just the one that
happened.

New checks in tests/snapStagePackages.test.cjs (every snap architecture
maps to a Caddy asset that exists, every build-for has a mapping, and -
negatively - an unmapped one fails loudly instead of 404ing) and in
tests/provenanceTable.test.cjs (the processing failure is retried, the
four unretryable messages are not, and the give-up message does not send
somebody debugging a build that succeeded). Verified the Caddy guard
fails when the armhf branch is removed.

Thanks to xet7 !

Author: xet7

.github/workflows/release-all.yml

@@ -3670,11 +3670,42 @@ jobs:
           fi
           # base: core24 allows publishing to stable and candidate (core26 does not).
           for f in "${snaps[@]}"; do
-            echo "Uploading $f to the Snap Store (stable,candidate,beta,edge)..."
-            if ! snapcraft upload --release=stable,candidate,beta,edge "$f"; then
-              echo "::error::The Snap Store refused the upload of $f. The snapcraft line above says which: 'is not a valid file' means the build produced nothing usable; 'Credentials could not be parsed' means SNAP_AUTH is not a readable exported login; 401/403 means it lacks the ACL for this snap name. The last two are fixed by re-exporting for all three names: snapcraft export-login --snaps wekan,wekan-ondra,wekan-gantt-gpl --acls package_access,package_push,package_release,package_update snap-auth.txt"
+            # RETRIED, because the upload can fail for a reason that has nothing
+            # to do with the snap. The store accepts the file, scans it, and can
+            # then answer:
+            #     Status: error while processing
+            #     Issues while processing snap:
+            #     - binary_sha3_384: Error checking upload uniqueness.
+            # which is the store failing its OWN duplicate check on the digest
+            # it just computed - a server-side hiccup, not a bad snap. It hit
+            # riscv64, ppc64el and s390x in the same v10.78 run, each after a
+            # Launchpad build that had succeeded, so three good snaps were lost
+            # to it at once. Re-uploading the identical file is what fixes it.
+            up_attempts=3
+            uploaded=0
+            for u in $(seq 1 "$up_attempts"); do
+              echo "Uploading $f to the Snap Store (stable,candidate,beta,edge), attempt $u/$up_attempts..."
+              if snapcraft upload --release=stable,candidate,beta,edge "$f" 2>&1 | tee /tmp/snap-upload.log; then
+                uploaded=1
+                break
+              fi
+              # Only the store's own processing errors are worth retrying. A
+              # rejected file, or credentials the store will not accept, will be
+              # rejected exactly the same way three times - and retrying those
+              # only buries the message that says what to fix.
+              if grep -qiE 'Error checking upload uniqueness|error while processing|502 Bad Gateway|503 Service|504 Gateway|Timeout|Connection (reset|aborted)' /tmp/snap-upload.log; then
+                if [ "$u" -lt "$up_attempts" ]; then
+                  echo "::warning::The Snap Store failed while PROCESSING $f (not a problem with the snap). Retrying in $((u * 30))s."
+                  sleep $((u * 30))
+                  continue
+                fi
+                echo "::error::The Snap Store failed while processing $f on every one of $up_attempts attempts. This is the store's own upload-uniqueness/processing check failing on a snap it accepted - the file itself built and downloaded fine, so re-running this job later usually publishes it unchanged. Nothing in this repository needs to change." >&2
+                exit 1
+              fi
+              echo "::error::The Snap Store refused the upload of $f. The snapcraft line above says which: 'is not a valid file' means the build produced nothing usable; 'Credentials could not be parsed' means SNAP_AUTH is not a readable exported login; 401/403 means it lacks the ACL for this snap name. The last two are fixed by re-exporting for all three names: snapcraft export-login --snaps wekan,wekan-ondra,wekan-gantt-gpl --acls package_access,package_push,package_release,package_update snap-auth.txt" >&2
               exit 1
-            fi
+            done
+            [ "$uploaded" -eq 1 ] || exit 1
           done
 
       - name: Attach the ${{ matrix.arch }} snap to the GitHub Release

snapcraft.yaml

@@ -502,7 +502,32 @@ parts:
                 s390x)   CADDY_ARCH=s390x ;;
                 ppc64el) CADDY_ARCH=ppc64le ;;   # Caddy uses ppc64le, not Debian's ppc64el
                 riscv64) CADDY_ARCH=riscv64 ;;
-                *) CADDY_ARCH="${CRAFT_ARCH_BUILD_FOR}" ;;
+                # Caddy is built by Go and its release assets carry GO's names,
+                # not Debian's. There is no linux_armhf archive and there never
+                # has been; the 32-bit ARM ones are armv5, armv6 and armv7. With
+                # no branch here armhf fell through to the default below, asked
+                # for caddy_<v>_linux_armhf.tar.gz, and got
+                #     curl: (22) The requested URL returned error: 404
+                # which failed the whole armhf snap in v10.78.
+                #
+                # armv7, not armv6: Debian armhf's baseline is ARMv7-A with
+                # VFPv3-D16 hard-float, and Go's armv7 build is GOARM=7, which
+                # is exactly that. Note this is NOT the armhf/armv7 distinction
+                # that matters for Node.js in wekan/node-patches - that one is
+                # about NEON, and Go's GOARM=7 does not use NEON.
+                armhf)   CADDY_ARCH=armv7 ;;
+                *)
+                    # An unmapped architecture used to become a URL that 404s,
+                    # and the "fall back to the pinned version" path below then
+                    # retried the same wrong name and failed too - so the error
+                    # blamed the Caddy RELEASE for something this case statement
+                    # got wrong. Say which it is.
+                    echo "caddy: no Caddy architecture is mapped for '${CRAFT_ARCH_BUILD_FOR}'." >&2
+                    echo "  Caddy publishes linux_amd64, linux_arm64, linux_armv5, linux_armv6," >&2
+                    echo "  linux_armv7, linux_ppc64le, linux_riscv64 and linux_s390x - Go's names," >&2
+                    echo "  not Debian's. Add a branch above rather than guessing here." >&2
+                    exit 1
+                    ;;
             esac
             echo "Downloading Caddy ${CADDY_VERSION} (linux/${CADDY_ARCH}) from GitHub releases..."
             # The newest release does not have to publish every architecture -

tests/provenanceTable.test.cjs

@@ -374,4 +374,56 @@ test('a release-note lookup can never fail the build it annotates', () => {
   });
 });
 
+// ───────────────────── the Snap Store can fail for reasons that are not ours
+
+test('a store PROCESSING failure is retried; a rejected snap is not', () => {
+  // The store accepts the file, scans it, and can then answer:
+  //     Status: error while processing
+  //     Issues while processing snap:
+  //     - binary_sha3_384: Error checking upload uniqueness.
+  // which is the store failing its OWN duplicate check on a digest it just
+  // computed. It hit riscv64, ppc64el and s390x in one v10.78 run, each after a
+  // Launchpad build that had SUCCEEDED, so three good snaps were lost at once
+  // and the printed advice was about credentials and ACLs - none of which
+  // applied.
+  //
+  // Retrying must stay narrow. A rejected file, bad credentials or a missing
+  // ACL will be rejected identically three times, and retrying those only
+  // buries the one message that says what to fix.
+  const src = read(WORKFLOW);
+  const step = src.slice(src.indexOf('Uploading $f to the Snap Store'));
+  const upTo = step.slice(0, step.indexOf('- name:'));
+
+  assert.ok(/Error checking upload uniqueness/.test(upTo),
+    'the store-side processing failure is not recognised');
+  assert.ok(/up_attempts/.test(upTo) && /sleep/.test(upTo),
+    'there is no retry with a backoff');
+
+  // The classifier itself, applied to the messages that must NOT be retried.
+  const m = upTo.match(/grep -qiE '([^']+)'/);
+  assert.ok(m, 'the retryable-error pattern is gone');
+  const re = new RegExp(m[1], 'i');
+  assert.ok(re.test('- binary_sha3_384: Error checking upload uniqueness.'),
+    'the real v10.78 failure must be retried');
+  assert.ok(re.test('Status: error while processing'));
+  [
+    'wekan_10.78_s390x.snap is not a valid file',
+    'Credentials could not be parsed',
+    'Error 403: forbidden - no permission for snap name',
+    'Error 401: unauthorized',
+  ].forEach(line => assert.ok(!re.test(line),
+    `"${line}" must NOT be retried - three identical rejections hide the fix`));
+});
+
+test('and it says the snap is fine when the store is not', () => {
+  // The failure that remains after three attempts is still not a problem with
+  // the snap, and the message has to say so - otherwise the next person goes
+  // looking through a build that succeeded.
+  const src = read(WORKFLOW);
+  const step = src.slice(src.indexOf('Uploading $f to the Snap Store'));
+  const upTo = step.slice(0, step.indexOf('- name:'));
+  assert.ok(/Nothing in this repository needs to change/.test(upTo),
+    'the give-up message must not send somebody debugging a good build');
+});
+
 console.log(`\n${passed} tests passed`);

tests/snapStagePackages.test.cjs

@@ -75,6 +75,18 @@ const T64_RENAMED = {
   'libboost-filesystem1.83.0': null,      // NOT renamed - real on armhf
 };
 
+// The Caddy architecture `case` block, and not the WEKAN_ARCH one above it:
+// snapcraft.yaml has more than one `case "${CRAFT_ARCH_BUILD_FOR}" in`, and
+// taking the first match tested the wrong block while looking like it worked.
+function caddyCase() {
+  const at = yaml.indexOf('CADDY_ARCH=');
+  assert.ok(at > 0, 'the Caddy architecture mapping is gone');
+  const start = yaml.lastIndexOf('case "${CRAFT_ARCH_BUILD_FOR}" in', at);
+  assert.ok(start > 0, 'the Caddy mapping is no longer a case statement');
+  const end = yaml.indexOf('esac', start);
+  return yaml.slice(start, end);
+}
+
 test('snapcraft.yaml declares stage-packages at all', () => {
   const pkgs = stagePackages();
   assert.ok(pkgs.length >= 10,
@@ -132,4 +144,58 @@ test('the reason is written down where the next person will read it', () => {
     'and the error it produces, so a search for it lands here');
 });
 
+// ─────────────────────────────────── Caddy is built by Go, and named like it
+
+test('THE BUG: every snap architecture maps to a Caddy asset that exists', () => {
+  // Caddy's release assets carry GO's architecture names, not Debian's. There
+  // is no linux_armhf archive and there never has been - the 32-bit ARM ones
+  // are armv5, armv6 and armv7 - so armhf fell through to the default branch,
+  // asked for caddy_<v>_linux_armhf.tar.gz and got
+  //     curl: (22) The requested URL returned error: 404
+  // which failed the whole armhf snap. The "fall back to the pinned version"
+  // path then retried the SAME wrong name, so the error blamed the Caddy
+  // release for something the case statement got wrong.
+  const caseBlock = caddyCase();
+
+  const EXPECTED = {
+    amd64: 'amd64',
+    arm64: 'arm64',
+    s390x: 's390x',
+    ppc64el: 'ppc64le',   // Caddy uses Go's name, not Debian's
+    riscv64: 'riscv64',
+    armhf: 'armv7',       // Debian armhf's baseline IS Go's GOARM=7
+  };
+  Object.entries(EXPECTED).forEach(([deb, go]) => {
+    const re = new RegExp(`${deb.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')}\\)\\s*CADDY_ARCH=${go}\\b`);
+    assert.ok(re.test(caseBlock),
+      `${deb} must map to Caddy's ${go}; Caddy publishes no linux_${deb} archive`);
+  });
+});
+
+test('every architecture snapcraft builds has a Caddy mapping', () => {
+  // The list that decides which snaps exist is build-for in this same file, so
+  // a new architecture added there without a Caddy branch is caught here rather
+  // than by a 404 halfway through a release.
+  const buildFor = [...new Set(
+    [...yaml.matchAll(/^\s*build-for:\s*(\S+)\s*$/gm)].map(m => m[1]))];
+  assert.ok(buildFor.length >= 5, `expected the build-for list, found ${buildFor.length}`);
+  const caseBlock = caddyCase();
+  buildFor.forEach(a => {
+    assert.ok(new RegExp(`(^|\\s|\\|)${a}\\)`, 'm').test(caseBlock),
+      `snapcraft builds ${a} but no Caddy architecture is mapped for it`);
+  });
+});
+
+test('NEGATIVE: an unmapped architecture fails loudly instead of 404ing', () => {
+  // The old default silently produced a URL that cannot exist. A guess that
+  // looks like a Caddy release name is worse than no guess, because the failure
+  // then reads as "Caddy stopped publishing this architecture".
+  const caseBlock = caddyCase();
+  assert.ok(!/\*\)\s*CADDY_ARCH="\$\{CRAFT_ARCH_BUILD_FOR\}"/.test(caseBlock),
+    'the default branch still passes the Debian name straight through');
+  assert.ok(/no Caddy architecture is mapped for/.test(caseBlock),
+    'the default branch must say what went wrong');
+  assert.ok(/exit 1/.test(caseBlock), 'and must stop rather than 404 later');
+});
+
 console.log(`\n${passed} tests passed`);