"ctr images pull" debugging should show the underlying hosts where layers are being pulled from

[vielmetti]

What is the problem you're trying to solve

We are seeing slow image pulls from "ctr images pull" from certain global locations to ghcr.io (Github Container Registry) hosted images. But IPv4 and IPv6 ping times to ghcr.io look fine from all locations.

It turns out that some ghcr.io assets, specifically layers, are served through a CDN at pkg-containers.githubusercontent.com . Inspection of IPv4 routes to that host look fine globally, but a number of locations have "scenic routes" to IPv6 addresses for that content. The resulting bad routing leads to very slow or stopped downloads.

None of the debug options I could figure out for "ctr images pull" exposed the "pkg-containers.githubusercontent.com" destination for image pulls, which made debugging much harder.

The feature request is to add a flag that would print out this detail to assist in debugging.

Describe the solution you'd like

Add a debug flag that prints out the domain name and IPv4 or IPv6 address of the host from which image layers are being retrieved.

Additional context

Discussion here:

https://hachyderm.io/@w8emv/111942173630607597
https://mastodon.online/@karppinen/111958791252380328

[estesp]

We added both trace and dump options from the HTTP Go API that you can use with ctr i pull:

   --http-dump                       dump all HTTP request/responses when interacting with container registry
   --http-trace                      enable HTTP tracing for registry interactions

I've used them before and gained a lot of insight into the HTTP transactions; can you let me know if either of those are valuable for what you are trying to accomplish?

[vielmetti]

Thanks @estesp .

I see this header in my trace

INFO[0003] Host: pkg-containers.githubusercontent.com

which starts to look helpful, but there's nothing in the trace that I can recognize that would tell me whether that host is being reached by IPv4 (where routing is fine and everything is peachy) or IPv6 (where routing is fubar and nothing is right).

[estesp]

Might have to get the Go DNS debug resolver involved then:

export GODEBUG=netdns=2

That should dump the name->IP resolver decisions from Golang and help you find out where it is doing IPv4 vs. IPv6 resolution?