Docker Version badge producing "invalid response data" error

[jauderho]

Are you experiencing an issue with...

shields.io

🐞 Description

Recently, I've been experimenting with signing container images using cosign and SBOM generation.

Here's the workflow to doing so: https://github.com/jauderho/dockerfiles/blob/099655baac390f47854f151eab3c1da3f0396aee/.github/workflows/age.yml

This results in additional artifacts being created and uploaded to Docker Hub. See https://hub.docker.com/r/jauderho/age/tags

🔗 Link to the badge

Here's the link to the badge in question: https://github.com/jauderho/dockerfiles/blob/main/age/README.md

💡 Possible Solution

I believe that these artifacts are causing parsing issues resulting in the invalid response data seen above instead of correctly displaying the version.

Therefore, it would be great if shields.io could ignore artifacts of the form sha256-SHA256SUM.[att|sig] . Much thanks.

More background information here:

• [Feature Request]: support signing container images with cosign docker/roadmap#269 (comment)
• Discuss ideal implementation of container signing and SBOM generation jauderho/dockerfiles#149

[calebcartwright]

Thanks for letting us know and including some background and hypothesis.

However, this error is indicative of the raw response we get from Dockerhub not matching the expected hub in terms of present/missing fields, not anything related to parsing/processing of tag values. I've updated the issue title to that effect to clarify the underlying problem vs. a proposed solution.

I haven't had a chance to actually go through the API response which can start to get fairly gnarly in repos with high quantities of tags, but the issue is going to be somewhere around here:

shields/services/docker/docker-version.service.js

Lines 12 to 25 in e385409

	const buildSchema = Joi.object({
	count: nonNegativeInteger.required(),
	results: Joi.array().items(
	Joi.object({
	name: Joi.string().required(),
	images: Joi.array().items(
	Joi.object({
	digest: Joi.string(),
	architecture: Joi.string().required(),
	})
	),
	})
	),
	}).required()

where one of those required fields aren't present for your repo target. we need to try to figure out why/if that's an intentional and expected pattern from the API side. if not, then that's an issue that should be taken upstream to the dockerhub folks, but if so, then we'll need to relax that schema a bit and then if necessary handle the error scenario internally

[jauderho]

@calebcartwright , thanks for the quick feedback.

I do not know much about Javascript but I'll take a stab at things. Looks like the URL is constructed here:

shields/services/docker/docker-version.service.js

Line 84 in e385409

url: `https://registry.hub.docker.com/v2/repositories/${getDockerHubUser(

Based on that construction, I can now pull json data from two images:

• https://registry.hub.docker.com/v2/repositories/jauderho/age/tags (with sig and SBOM artifacts)
• https://registry.hub.docker.com/v2/repositories/jauderho/rustybgp/tags (works fine)

After diffing the json, I think I know what the problem is. architecture is a required field and the .sig/.att artifacts unsurprisingly have no defined architecture nor os.

See https://gist.github.com/jauderho/9c5451ef361c8265bbb3f1db0914f058#file-sig-json-L13 vs. https://gist.github.com/jauderho/0258a4bab184c77bc694e297b56754f2#file-normal-json-L13

I think you would want to keep the architecture requirement for normal use cases but maybe you can use a .without() to exclude name fields that match an example pattern of sha256-a2b7f0e3a43e7daa1e13204cdf093cf03f5042be26a451817cca561b345725ec.sig

I have not used Joi so will need some help constructing an exclusion pattern.

[calebcartwright]

I'm not entirely sure I follow how you're going about signing images, but more broadly I'm surprised that the API call is returning an "extra" object that ostensibly represents an actual tag. I'd be more curious to find out whether this behavior is intentional, or potential a bug on the Dockerhub API side.

[jauderho]

At a high level, until the Docker GitHub action actually gains the ability to sign images, right now we have to sign the image as a separate step.

So it's roughly:

• Build and push container image to container registry
• Capture the image digest as part of the output
• Separate step to sign the image digest
• Push resulting .sig to container registry
• Do something similar for SBOM generation

Per @dlorenc, this is going to be the behavior until changes to the OCI spec and registries can be made (ETA unknown). To be clear, this affects more than Docker Hub.

See the following examples from GHCR and GitLab:

• GitHub - https://github.com/jauderho/sandbox/pkgs/container/age-sandbox
• GitLab - https://gitlab.com/jauderho/sandbox/container_registry/2714238

Therefore, if we want to store signatures and attestations in the registries in the near future, the shields badges will not work as things currently stand and hence the invalid response data until those are filtered/ignored.

[calebcartwright]

I wasn't aware of these efforts before but I get the gist. Seems there's a big push these days to get signature and bills for just about everything, so I'm not surprised in the least that this is occuring in the container image space too.

I also appreciate you sharing the extra context around the breadth, though I suspect the nature of the other registries will be largely orthogonal in our specific context. For reasons too complicated to delve into here, we don't integrate with those others and are ultimately focused on the request/response relationship between us and the Dockerhub API.

Given the observed behavior, and mentions of things like needing spec updates, I'll go out on a relatively safe limb and assume that the Docker tooling (ostensibly both client and server side) doesn't yet natively support the association of these artifacts to their target image tag (at least not well). This will change at some point, but for now, folks are strictly using auxiliary tooling which is working around those native gaps by distributing the associated artifacts in a way that from a registry & API POV looks like separate tags.

Our Docker badges are among our most popular, and we deal with a fairly high number of requests that in turn process responses from Dockerhub that are on the larger side compared to many other servicse/badges. As such, before we take the plunge of incorporating some type of regex based processing on every element in the collections in those responses, there's a few things I'd love to get some clarity on, or at least pointers to relevant upstream issues so that we can better track things/make sure we drop any extra processing as soon as it becomes unnecessary:

• Is my summary above a reasonably, if oversimplified, recap?
• Assuming there are different auxiliary tools being used for generating+validating these extra artifacts, are they all following the same paradigm in terms of how the artifacts ultimately end up represented in the Dockerhub API response structure? (would like to avoid having multiple bespoke solutions to account for tool variance)
• Presumably Docker are aware of the current state of affairs and the tactical solution that's being utilized today, have they given any kind of 👍 anywhere to indicate their approval/acceptance of this tactical solution?
  • Similarly, any tracking issues/public pronouncement from the Docker folks relative to their plans with the API (e.g. are they planning on doing any filtering themselves serverside, any additional query parameters to control said hypothetical filtering, etc.)
• What, if any, issues exist in relevant upstream Docker+friends repositories would you say are worth tracking?

My hope is that there's enough of a Docker+community collaboration on this topic that Docker are thinking about ways to sort this, even tactically, on the server side such that each individual client doesn't have to separately reinvent the same wheel.

[chris48s]

Thinking about this as more of a "black box" (rather than getting stuck into the implementation details), it seems like what we've discovered here is that in practice, sometimes this API returns objects that have an empty architecture property (and the reason for that appears to be valid, if a bit unexpected). Is that a fair summary?

If so, I think we can somewhat simplify the issue of how we fix the badge by saying: that means we need to allow architecture to be empty (i.e: Joi.string().allow('')) without getting too bogged down in the reasons for it (although I don't want to derail an interesting conversation on that topic). This would be in line with our stated processes for input validation: https://github.com/badges/shields/blob/master/doc/input-validation.md and how we fix validation problems when they occur.

Forgive me if there's some detail I am missing here.

[calebcartwright]

If so, I think we can somewhat simplify the issue of how we fix the badge by saying: that means we need to allow architecture to be empty (i.e: Joi.string().allow(''))

Realize this is the fastest path to market, so to speak, but the reason we're seeing this is because the platform's being used in unexpected ways (you'd always expect an arch for image). Would personally like to get more info to make sure we don't end up chasing a moving target, or that we don't go down our own quick fix route only to have it be obviated by upstream changes we're similarly not aware of.

If i were to attempt to analogize with some hyperbole, it would be like requesting the versions of an npm package on npmjs and amongst the collection alongside typical version objects seeing something very unexpected like an xml or binary reference. Yes we can, and likely will, need to work around it, but it's unexpected enough that it warrants spending a little more time on the "why" IMO

[jauderho]

As I was suggesting above, I would not necessarily recommend a blanket allow for architecture to be empty since there's a valid reason for making sure there is generally an architecture available, rather I'm suggesting that if name matches a specific pattern of sha256-*.* to then permit the architecture to be empty.

This preserves the prior behavior and checks when allowing for the fact that signatures and attestations are going to more of a thing going forward.

I'd be happy to track this for upstream changes of direction and inform here if that does happen. But based on what I'm seeing upstream, changes to the registries might take a while so this might be the best imperfect solution for now.

[calebcartwright]

I'd be happy to track this for upstream changes of direction and inform here if that does happen. But based on what I'm seeing upstream, changes to the registries might take a while so this might be the best imperfect solution for now.

I appreciate the offer, but that's not the ask and I don't think it would be very efficient anyway. It'll be a lot easier to have the relevant touch points directly navigable from here, so simply sharing some links to upstream issues would be super helpful if you're up for it!

[jauderho]

@calebcartwright Just checking in. Is this something that can/will be fixed or is the plan to see what the registries do?

I'd like to start signing all of the images that I build but am not keen on having the badges showing "invalid response" once they are signed. Just trying to figure out if I should start planning to do this now or wait 6 months. Thanks.