[Installer]: report on container registry problems

[mrsimonemms]

This is to be used as a notepad for an issue. Please ignore

• this worked with all clouds/container registries before [image-builder] Fix image-push and logs #6681 was merged - why is GCP external not working in Self-Hosted when working for SaaS?
• FQDN workspaces use https://github.com/csweichel/test-repo/tree/1c287d4c5ef15f97e12b8316191c22cf47b8c1d3
• non-FQDN workspaces use any GP example
• also check https://github.com/csweichel/test-repo - this uses an FDQN image with image in GCP container registry

Cloud	Registry	Working?	Notes
GCP	InCluster FQDN	Y
GCP	InCluster non-FQDN	Y
GCP	GCP FQDN	N	As non-FQDN.
GCP	GCP non-FQDN	N	Lots of 502 Bad Gateway errors, ending in #5 37.17 error: unexpected status: 502 Bad Gateway ------ error: failed to solve: unexpected status: 502 Bad Gateway {"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","command":"build","error":"exit status 1","level":"error","message":"build failed","serviceContext":{"service":"bob","version":""},"severity":"ERROR","time":"2021-11-19T15:42:35Z"}
Azure	InCluster FQDN	N	Seems to not get the auth token to validate the HEAD https://registry-1.docker.io/v2/gitpod/workspace-mongodb/manifests/latest call. Works fine if using GCR based image
Azure	InCluster non-FQDN	N	As FQDN
Azure	Azure FQDN	N
Azure	Azure non-FQDN	N

Observations:

• GCP registry errors appear to be unable to authorize to the gcr.io registry, so not relevant to the FQDN status. The .dockerconfigjson value appears valid when running locally as can push an image with the value set to ~/.docker/config.json

Questions:

• is the issue caused by a different workspace.runtime.containerdRuntimeDir values? In GCP it's /var/lib/containerd/io.containerd.runtime.v2.task/k8s.io, in Azure it's /run/containerd/io.containerd.runtime.v1.linux/k8s.io
• or containerd versions?
• Azure uses shiftfs, GCP uses fuse
• when comparing GCP incluster to Azure incluster, there are various differences:
  • Azure has the following lines before the proxy auth request:

{"alias":"d428f509-7d88-4274-8454-ed1cc99ca470","cmd":"/bin/bash","level":"info","message":"started new terminal","serviceContext":{"service":"supervisor","version":"commit-0be2ad26006e15a151aba93ae5c7b4ea3a3311ef"},"severity":"INFO","time":"2021-11-21T16:45:50Z"}
{"command":"{\nsudo -E /app/bob build\n}; exit","level":"info","message":"task terminal has been started","pid":34,"serviceContext":{"service":"supervisor","version":"commit-0be2ad26006e15a151aba93ae5c7b4ea3a3311ef"},"severity":"INFO","terminal":"d428f509-7d88-4274-8454-ed1cc99ca470","time":"2021-11-21T16:45:50Z"}
{"level":"info","message":"Writing build output to /workspace/.gitpod/prebuild-log-0","pid":34,"serviceContext":{"service":"supervisor","version":"commit-0be2ad26006e15a151aba93ae5c7b4ea3a3311ef"},"severity":"INFO","time":"2021-11-21T16:45:50Z"}

• Azure has http: proxy error: HEAD https://registry-1.docker.io/v2/gitpod/workspace-mongodb/manifests/latest which GCP doesn't (is this causal or symptomatic?)

[csweichel]

Generalising: the issue seems to arise when we want to pull a base image from docker hub. The question then becomes: why does it work when testing in a workspace, or more broadly, why does it work at all?

[mrsimonemms]

@csweichel I wonder if this is the key. I've added loads of additional (insecure) logs in a branch (not to be merged) and found something interesting....

I've checked the response from repo.Auth.Authorize.

In GCP, it ALWAYS returns back without an error (even on the very first time).
In Azure, it ALWAYS returns back this error:

{
  "@type": "type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent",
  "error": "Get \"https://auth.ipv6.docker.com/token?scope=repository%3Agitpod%2Fworkspace-mongodb%3Apull\u0026service=registry.docker.io\": dial tcp [2600:1f18:2148:bc00:cec:e441:76aa:4346]:443: connect: network is unreachable\nfailed to fetch anonymous token\ngithub.com/containerd/containerd/remotes/docker.(*authHandler).doBearerAuth\n\t/workspace/go/pkg/mod/github.com/containerd/containerd@v1.5.5/remotes/docker/authorizer.go:306\ngithub.com/containerd/containerd/remotes/docker.(*authHandler).authorize\n\t/workspace/go/pkg/mod/github.com/containerd/containerd@v1.5.5/remotes/docker/authorizer.go:228\ngithub.com/containerd/containerd/remotes/docker.(*dockerAuthorizer).Authorize\n\t/workspace/go/pkg/mod/github.com/containerd/containerd@v1.5.5/remotes/docker/authorizer.go:112\ngithub.com/gitpod-io/gitpod/image-builder/bob/pkg/proxy.(*Proxy).reverse.func2\n\t/tmp/build/components-image-builder-bob--app.65626b7c97210e6bd97e4ad3c440d299218b427f/pkg/proxy/proxy.go:173\ngithub.com/hashicorp/go-retryablehttp.(*Client).Do\n\t/workspace/go/pkg/mod/github.com/hashicorp/go-retryablehttp@v0.7.0/client.go:584\ngithub.com/hashicorp/go-retryablehttp.(*RoundTripper).RoundTrip\n\t/workspace/go/pkg/mod/github.com/hashicorp/go-retryablehttp@v0.7.0/roundtripper.go:44\nnet/http/httputil.(*ReverseProxy).ServeHTTP\n\t/home/gitpod/go/src/net/http/httputil/reverseproxy.go:298\ngithub.com/gitpod-io/gitpod/image-builder/bob/pkg/proxy.(*Proxy).ServeHTTP\n\t/tmp/build/components-image-builder-bob--app.65626b7c97210e6bd97e4ad3c440d299218b427f/pkg/proxy/proxy.go:127\nnet/http.(*ServeMux).ServeHTTP\n\t/home/gitpod/go/src/net/http/server.go:2424\nnet/http.serverHandler.ServeHTTP\n\t/home/gitpod/go/src/net/http/server.go:2878\nnet/http.(*conn).serve\n\t/home/gitpod/go/src/net/http/server.go:1929\nruntime.goexit\n\t/home/gitpod/go/src/runtime/asm_amd64.s:1581",
  "level": "error",
  "message": "sje auth err",
  "serviceContext": {
    "service": "bob",
    "version": ""
  },
  "severity": "ERROR",
  "time": "2021-11-22T13:28:08Z"
}

So, why is the network unreachable?

[mrsimonemms]

It appears that in Azure, the www-authenticate value that comes back from the initial call to https://registry-1.docker.io/v2/gitpod/workspace-mongodb/manifests/latest is to auth.ipv6.docker.com rather than auth.docker.com.

And I've no idea why

Interestingly, the X-Forwarded-For value in Azure is ::1 and in GCP it's 127.0.0.1 - this appears to set the IPV6 URL

---

Edit: unbelievably, if I set X-Forwarded-For to 127.0.0.1, it works