Manifest list resolution breaks CRI-O based clusters

[markusthoemmes]

HEAD

Expected Behavior

I expect all images to work on Knative.

Actual Behavior

Images from a registry that supports multiple architectures (like Dockerhub.com) don't work.

Steps to Reproduce the Problem

1. Get a CRI-O based cluster (like Openshift)
2. Run TestCmdArgsService (uses the python images)
3. It doesn't work.

Relevant information

This is related to cri-o/cri-o#2157. According to @jonjohnsonjr we only recently switched to resolving the manifest list vs. only resolving amd64/linux.

[markusthoemmes]

@jonjohnsonjr would it be a workable solution to provide a know to set the architecture to resolve (the default being to resolve the manifest list and letting the node handle the final resolution)?

I think that'd unblock CRI-O based clusters for now (we can almost certainly safely assume amd64/linux there) until that bug in CRI-O is fixed.

[jonjohnsonjr]

We can use runtime.GOOS and runtime.GOARCH to pull the image that matches the controller's system. This won't work for clusters that have heterogeneous nodes, e.g. a master on amd64/linux scheduling jobs on a bunch of raspberry pis. I don't see a better option until CRI-O is fixed, though.

[markusthoemmes]

@jonjohnsonjr what if we put a knob in front of that as proposed above?

[jonjohnsonjr]

What kind of knob are we thinking? Configmap? Environment variables?

[jonjohnsonjr]

I think #3998 is going to work in almost every case, so I'm reluctant to add a knob if we'll never need to use it.

[jonjohnsonjr]

This would also break schema 1 images, but I'm okay with that.