Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"no supported platform found in manifest list" / "no matching manifest for XXX in the manifest list entries" #3835

Open
tianon opened this Issue Dec 21, 2017 · 15 comments

Comments

Projects
None yet
9 participants
@tianon
Copy link
Member

tianon commented Dec 21, 2017

TLDR: Not all architectures are created equal, but perhaps even more importantly, not all build servers we have access to are equal in performance, power, or ability to process builds reliably.

Important: Please do not post here with reports of individual image issues -- we're aware of the overall problem, and this issue is a discussion of solving it generally. Off-topic comments will be deleted.


When we merge an update PR to https://github.com/docker-library/official-images, it triggers Jenkins build jobs over in https://doi-janky.infosiftr.net/job/multiarch/ (see #2289 for more details on our multiarch approach).

Sometimes, we'll have non-amd64 image build jobs finish before their amd64 counterparts, and due to the way we push the manifest list objects to the library namespace on the Docker Hub, that results in amd64-using folks (our primary target users) getting errors of the form "no supported platform found in manifest list" or "no matching manifest for XXX in the manifest list entries" (see linked issues below for several reports from users of this variety).

Thus, manifest lists under the library are "eventually consistent" -- once all arches complete successfully, the manifest lists get updated to include all the relevant sub-architectures.

Our current method for combating the main facet of this problem (missing amd64 images while other arches are successfully built and available) is to trigger amd64 build jobs within an hour after the update PR is merged, and all other arches only within 24 hours. This helps to some degree in ensuring that amd64 builds first, but not always. For example, our arm32vN servers are significantly faster than our AWS-based amd64 server, so if those jobs happen to get queued at the same time as existing amd64 jobs are, they'll usually finish a lot more quickly. Additionally, given the slow IO speed of our AWS-based amd64 build server, the queue for amd64 build jobs piles up really quickly (which also doesn't help with keeping our build window low).

As for triggering jobs more directly, the GitHub webhooks support in Jenkins makes certain assumptions about how jobs and pipelines are structured/triggered, and thus we can't use GitHub's webhooks to effectively trigger these jobs (without doing additional custom development to sit between the two systems), and thus rely on the built-in Jenkins polling mechanism. This has been fine (we haven't noticed any scalability issues with how often we're polling), and even if we were triggering builds more aggressively, that's only half the problem (since then our build queues would just pile up faster).

One solution that has been proposed is to wait until all architectures successfully build before publishing the relevant manifest list. If a naïve version of this suggestion were implemented right now, we would have no image updates published because our s390x worker is currently down (as an example -- we do frequently lose builder nodes given that all non-amd64 arches are using donated resources). Additionally, as noted above, some architectures build significantly slower than others (before we got our hyper-fast ARM hardware, arm32vN used to take days to build images like python), so it isn't exactly fair to force all architectures to wait for the one slowpoke before providing updated images to our userbase. As a final thought on this solution, some architectures outright fail, and the maintainers don't necessarily notice or even care (for example, mongo:3.6 on windows-amd64 has been failing consistently with a mysterious Windows+Docker graph driver error that we haven't had a chance to look into or escalate, and wouldn't be fair to block updated image availability on).

One compromise would be to use the Jenkins Node API (https://doi-janky.infosiftr.net/computer/multiarch-s390x/api/json) to determine whether a particular builder is down in order to determine whether to block on builds of that architecture. Additionally, we could try to get creative with checking pending builds / queue length for a particular architecture's builds to determine whether or not a given architecture is significantly backlogged and thus a good candidate for not waiting.

We could also attempt to determine when a particular tag was added/merged, and set a time limit for some number of hours before we just assume it must be backlogged, failing, or down and move along without that tag, but this is slightly more complicated (since we don't have a modification time for a particular tag directly, and really can only determine that information on an image level without complex Git walking / image manifest file parsing). Perhaps even just a time limit on the image level would be enough, but in the case of our mongo:3.6 example, that would mean all tag updates to mongo (whether they're related to the 3.6 series or not) would wait the maximum amount of time before being updated due to one version+architecture combination failing.


Related issues: (non-comprehensive)

@m5p3nc3r

This comment has been minimized.

Copy link

m5p3nc3r commented Jan 11, 2018

Hey @tianon Thanks for introducing me to this issue yesterday. Initial thoughts:

The core of the problem seems to be the availability of amd64 images (breaks my heart being an Arm guy - but its a fair statement of the situation today!). We therefore need to make sure that the fat manifest is only published once the amd64 build has completed successfully?

How about implementing a system whereby the manifest is only published when a list of 'gold' architectures have build successfully? This way, you could ensure that the amd64 issue never rears its ugly head again? It would also mean that for specific images - say popular base images like Alpine, that the manifest is only published when for example amd64, arm64v8, ppc64le and s360x are successful?

Its not an ideal solution, but it could act as a way of stabilising things whilst a better solution could be implemented.

One last thought - if this were to be implemented, it would be useful to have a global 'gold architecture' list and a per project delta from that list.

@MattF-NSIDC

This comment has been minimized.

Copy link

MattF-NSIDC commented Jan 25, 2018

What advice do you have for developers affected by this bug? I'm using an image which depends on docker-library/tomcat and I've been unable to build for about a half hour. I read your post pretty carefully, I think, but didn't see any mention of a workaround. Based on what I've read, this is not a problem that can be solved on my side, I would just have to wait.

If that is the case, is there any way for me to do maybe an API query to estimate a wait time until dockerhub reaches a consistent state?

@tianon

This comment has been minimized.

Copy link
Member Author

tianon commented Jan 25, 2018

@MattF-NSIDC fair point -- this was intended as a tracking issue for the problem and discussion around how to solve the crux of it properly; I think a short blurb here about how to work around it in the meantime is definitely appropriate. Here's my current recommendation:

If you rely on a specific image, use https://github.com/docker-library/repo-info (linked from every image description) to find the exact sha256 digest (also available from the docker pull output, but the repo-info repository has retroactive digests in the Git history), or even simply use a more specific tag.

If you're looking for a specific architecture, use the architecture-specific namespace to find it (as linked from both https://github.com/docker-library/official-images#architectures-other-than-amd64 and every image description under "Supported architectures").

As for ETA, even if we find a reasonable solution to "wait for things to be available", we'll still have a limit on how long we wait before pushing whatever we've got, which will likely be on the order of hours but still less than 24 (because IMO, 24h ought to be the absolute maximum we wait before we storm ahead, and should roughly match our current builds-scheduling timing).

@tianon

This comment has been minimized.

Copy link
Member Author

tianon commented Jan 25, 2018

@m5p3nc3r yeah, the "gold architectures" solution is basically exactly the solution I wrote on a quick note on my desk the first time we had this problem 😄

I don't love it, but it does seem like the closest we can get, and definitely is going to be better than what we're doing now. It would also allow us to simply trigger all builds as soon as possible after merge and let everything simply trickle through. The main challenge I see is how to implement the "timeout" functionality, but perhaps we simply punt and put the timeout on the full image instead of individual tags and call it a day.

@tianon tianon pinned this issue Dec 24, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.