build.py: podman imagebuilder out of sync

[efahl]

cc: @lorand-horvath

ASU build has been generating these errors on x86/64 snapshot for a day or so, several similar errors being reported on the forum.

Received incorrect version r26608-b2a84c86e3 (requested r26620-6b03050192)

Since the "janitor: drop" commit the metadata updates rapidly, and both
https://sysupgrade.openwrt.org/json/v1/snapshots/targets/x86/64/generic.json
and
https://downloads.openwrt.org/snapshots/targets/x86/64/profiles.json
contain the correct 26620 version_code, so it seems that the previous source of this issue has been resolved.

This time it appears to me to be the podman image is out of sync, behind the current downloads imagebuilder. I couldn't find anything in update.py that would invalidate or update the podman.images collection.

But in build.py we know something is wrong. Just a wild, untested idea, when this specific error is detected, maybe just delete the podman image, so the next try does a fresh podman.images.pull? (At least I'm assuming that's how pull works.) In build.py:

    log.info(f"Pulling {image}...")
    pm_image = podman.images.pull(image)        <<< Save the returned image...
    log.info(f"Pulling {image}... done")

...

    if "version_code" in req:
        if version_code != req.get("version_code"):
            pm_image.remove(force=True)          <<< Remove it before reporting error, or maybe 'reload'???
            report_error(

Refs:
https://podman-py.readthedocs.io/en/stable/podman.domain.images_manager.html#podman.domain.images_manager.ImagesManager.pull
https://podman-py.readthedocs.io/en/stable/podman.domain.images.html#podman.domain.images.Image.remove

[efahl]

@aparcar Does my analysis above make any sense? I just tried again, and the metadata is now updated to 26629, but ASU's imagebuilder remains back on 26608 as above...

Error: Received incorrect version r26608-b2a84c86e3 (requested r26629-7a9b1f7f25)

[lorand-horvath]

The strangest thing just occurred: I've requested a custom build with ASU (luci-app-attendedsysupgrade), it said it had a newer build available, namely 26620. I requested it, but it came back with the old version stuck at 26593 (as I expected due to this bug). I proceeded to upgrade to this so-called 26593 version, at least that's what ASU said it was.

After it flashed and rebooted, I check the version in LuCI and low and behold, it has actually flashed the latest version 26620. What the heck is going on?

Note: ath79/generic unit WR1043ND v3

[efahl]

luci-app-attendedsysupgrade

There's your culprit in this case. Everything else (auc, owut, firmware selector) specifies the version_code in the build blob, which the ASU builder verifies request and imagebuilder as being the same (https://github.com/openwrt/asu/blob/main/asu/build.py#L72). If you leave it out, then you can sometimes get a build to go through, even when the metadata is different than the actual imagebuilder.

I experimented with an option in owut to turn off the version code spec, but decided it was more robust to keep it and to try to fix up the builds, because half (or more) of the time it didn't work and just wasted build cycles (got a different error later, like "package version mismatch" or some such thing).

[efahl]

Right now (2024-06-15 18:40Z) the ASU server is ahead of the downloads site:

snapshots x86/64 generic
  r26634-7d9b9762c9  ASU Server
  r26633-22f92cce22  Build Server

But the ASU builder is still stuck at 26608

Error: Received incorrect version r26608-b2a84c86e3 (requested r26633-22f92cce22)

My interpretation is that the web hooks are not flushing the old imagebuilders (and are firing off at the start of the build, not its completion???).

[lorand-horvath]

WNDRMAC v2 (ath79/generic) - I tried to build a new custom firmware with the firmware-selector, it said 26620 is available, but after build it still came back with the old 26593 from 5 days ago.

Then I checked with ASU (luci-app-attendedsysupgrade) and it said 26620 was available, I requested it, came back with 26593, I even downloaded it from the ASU interface to my PC and I confirm it's 26593 indeed. Then I proceeded to ask ASU to actually flash it and, as you might have guessed, it's now on 26620. Very strange.

This is a mess.

Also, since ath79 was bumped from kernel 6.1 to 6.6 yesterday, it doesn't even build at all anymore.

[pko76]

As far as I understand it is a similar problem and is "under construction"
https://forum.openwrt.org/t/auc-sysupgrade-does-not-work-for-me/196782/39

[dannil]

There's your culprit in this case. Everything else (auc, owut, firmware selector) specifies the version_code in the build blob, which the ASU builder verifies request and imagebuilder as being the same (https://github.com/openwrt/asu/blob/main/asu/build.py#L72). If you leave it out, then you can sometimes get a build to go through, even when the metadata is different than the actual imagebuilder.

In the meantime before this problem get fixed, I've created openwrt/luci#7163 to add the check to luci-app-attendedsysupgrade so it behaves in the same way as auc.

[aparcar]

My guess is that this issue got multiple layers. First the Docker containers would download from downloads.openwrt.org which is behind a CDN. Even though the cache are cleared on every fresh target upload, there may be a timing issue where Docker containers start downloading ImageBuilders while not yet fully cleared (a matter of a few seconds I think).

Second is that ASU caches successful builds for 7 days. The identify the "same" build request and allow this kind of caching, a bund of parameters are put into account, including version_code. Since the LuCI lacked that parameter, build requests from up to 7 days ago would be returned. Thanks to @dannil this issue should be fixed for LuCI.

Right now this isn't happening for the firmware selector, so please feel free to implement it there, too.

[aparcar]

        pm_image.remove(force=True)          <<< Remove it before reporting error, or maybe 'reload'???

Using podman.images.pull() update the container whenever something new is available, no need to remove it locally

[aparcar]

As you say in German, all good things are three: The GitHub token ran out over the weekend causing the webhook to never run any container updates.

[efahl]

The GitHub token ran out over the weekend causing the webhook to never run any container updates.

GAH! That explains a lot.

[efahl]

I'm still seeing the same issue. Ran two builds on different x86 machines, about a half hour apart.

Received incorrect version r26657-531b3f667c (requested r26670-5f9fb964c3)

ASU and downloads profiles match with the 26670 version, but the image is updated from previous, but now stuck back at 26657

Builds at
https://sysupgrade.openwrt.org/api/v1/build/eaf666a98b390bda3f8896a8b4000ee0
https://sysupgrade.openwrt.org/api/v1/build/eecd821426ce6f223f079b925029be2c
if that helps...