Registry:3.0.0 failed with private s3 compatible bucket

[erfanw]

Description

Hi,
I am testing the latest main branch since I would need the basic auth feature which is not supported by registry:2.8.3.
The registry was configured to use the pull-through cache mode.
I cloned the repo and built it. I modified the config to use s3 for storage, but it failed.
I will provide more details below, thanks!

Reproduce

1. clone the latest repo, main branch (As of today May 20th, 2024)
2. build docker image. I set the name as "distribution:3.0.0"
3. Run docker run -it --rm --entrypoint cat distribution:3.0.0 /etc/docker/registry/config.yml > config.yml to get the config.yml file
4. Modify the config.yml to use my s3-compatible bucket

storage:
delete:
enabled: true
cache:
blobdescriptor: inmemory
s3:
accesskey: ACCESS_KEY
secretkey: SECRET_KEY
region: eu-central-1
regionendpoint: https://some-s3-endpoint.net
bucket: docker-registry-test
maintenance:
uploadpurging:
enabled: false

5. run the registry docker run -p 5005:5000 --restart=always -v $(pwd)/config.yml:/etc/docker/registry/config.yml distribution:3.0.0. Then I got:

time="2024-05-20T13:52:35.498586674Z" level=debug msg="using "text" logging formatter"
time="2024-05-20T13:52:35.499167841Z" level=info msg="redis not configured" environment=development go.version=go1.21.8 instance.id=f752128d-d6a1-417f-bc80-04f062fc038c service=registry version=3.0.0-alpha.1-155-ge0a54de7
time="2024-05-20T13:52:35.499192591Z" level=info msg="using inmemory blob descriptor cache" environment=development go.version=go1.21.8 instance.id=f752128d-d6a1-417f-bc80-04f062fc038c service=registry version=3.0.0-alpha.1-155-ge0a54de7
panic: s3aws: RequestError: send request failed
caused by: Get "https://docker-registry-test.some-s3-endpoint.net/?list-type=2&max-keys=1&prefix=scheduler-state.json": dial tcp: lookup docker-registry-test.some-s3-endpoint.net on 192.168.65.7:53: no such host
goroutine 1 [running]:
github.com/distribution/distribution/v3/registry/handlers.NewApp({0x15a0bc0?, 0x4000916b10?}, 0x4000616e00)
github.com/distribution/distribution/v3/registry/handlers/app.go:336 +0x1734
github.com/distribution/distribution/v3/registry.NewRegistry({0x15a0bc0?, 0x4000615530?}, 0x4000616e00)
github.com/distribution/distribution/v3/registry/registry.go:142 +0x38
github.com/distribution/distribution/v3/registry.glob..func1(0x40005a0900?, {0x400061d1e0, 0x1, 0x1})
github.com/distribution/distribution/v3/registry/registry.go:111 +0xe0
github.com/spf13/cobra.(*Command).execute(0x204b6c0, {0x400061c7d0, 0x1, 0x1})
github.com/spf13/cobra@v1.8.0/command.go:987 +0x828
github.com/spf13/cobra.(*Command).ExecuteC(0x204b9a0)
github.com/spf13/cobra@v1.8.0/command.go:1115 +0x344
github.com/spf13/cobra.(*Command).Execute(0x1fc2f58?)
github.com/spf13/cobra@v1.8.0/command.go:1039 +0x1c
main.main()
github.com/distribution/distribution/v3/cmd/registry/main.go:25 +0x24

Expected behavior

The registry should be running. When I did the same using registry:2.8.3 using the same s3-compatible bucket, the registry was actually running

% docker run -p 5005:5000 --restart=always -v $(pwd)/config.yml:/etc/docker/registry/config.yml registry:2
time="2024-05-20T14:04:48.047459055Z" level=info msg="debug server listening :5001"
time="2024-05-20T14:04:48.047999013Z" level=warning msg="No HTTP secret provided - generated random secret. This may cause problems with uploads if multiple registries are behind a load-balancer. To provide a shared secret, fill in http.secret in the configuration file or set the REGISTRY_HTTP_SECRET environment variable." environment=development go.version=go1.20.8 instance.id=5d656110-25cc-45bf-9843-bc5a2b97eb6e service=registry version=2.8.3
time="2024-05-20T14:04:48.048025972Z" level=info msg="redis not configured" environment=development go.version=go1.20.8 instance.id=5d656110-25cc-45bf-9843-bc5a2b97eb6e service=registry version=2.8.3
time="2024-05-20T14:04:48.048167513Z" level=info msg="using inmemory blob descriptor cache" environment=development go.version=go1.20.8 instance.id=5d656110-25cc-45bf-9843-bc5a2b97eb6e service=registry version=2.8.3
time="2024-05-20T14:04:48.181423597Z" level=debug msg="s3aws.Stat("/scheduler-state.json")" environment=development go.version=go1.20.8 instance.id=5d656110-25cc-45bf-9843-bc5a2b97eb6e service=registry trace.duration=133.229958ms trace.file="github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat" trace.id=6f82a016-e1ee-46c6-b3fd-048efd07e7eb trace.line=155 version=2.8.3
time="2024-05-20T14:04:48.214957847Z" level=debug msg="s3aws.GetContent("/scheduler-state.json")" environment=development go.version=go1.20.8 instance.id=5d656110-25cc-45bf-9843-bc5a2b97eb6e service=registry trace.duration=33.502625ms trace.file="github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent" trace.id=7deb7a14-90c4-4a2c-8464-888f6cb02008 trace.line=95 version=2.8.3
time="2024-05-20T14:04:48.215078263Z" level=info msg="Starting cached object TTL expiration scheduler..." environment=development go.version=go1.20.8 instance.id=5d656110-25cc-45bf-9843-bc5a2b97eb6e service=registry version=2.8.3
time="2024-05-20T14:04:48.340977263Z" level=info msg="Registry configured as a proxy cache to https://some-upstream.com" environment=development go.version=go1.20.8 instance.id=5d656110-25cc-45bf-9843-bc5a2b97eb6e service=registry version=2.8.3
time="2024-05-20T14:04:48.341032097Z" level=warning msg="Registry does not implement RempositoryRemover. Will not be able to delete repos and tags" environment=development go.version=go1.20.8 instance.id=5d656110-25cc-45bf-9843-bc5a2b97eb6e service=registry version=2.8.3
time="2024-05-20T14:04:48.341106722Z" level=info msg="providing prometheus metrics on /metrics"
time="2024-05-20T14:04:48.341263972Z" level=info msg="listening on [::]:5000" environment=development go.version=go1.20.8 instance.id=5d656110-25cc-45bf-9843-bc5a2b97eb6e service=registry version=2.8.3

registry version

registry version=3.0.0-alpha.1-155-ge0a54de7

Additional Info

No response

[milosgajdos]

panic: s3aws: RequestError: send request failed
caused by: Get "https://docker-registry-test.some-s3-endpoint.net/?list-type=2&max-keys=1&prefix=scheduler-state.json": dial tcp: lookup docker-registry-test.some-s3-endpoint.net on 192.168.65.7:53: no such host
goroutine 1 [running]:

This is failing DNS lookup which has nothing to do with registry I'm afraid. You're gonna have to sort that out first.

As an aside it's strange that registry is panicking when the DNS lookup fails, though 🤔

[erfanw]

@milosgajdos Thanks for your super fast reply!
My colleagues suspect this is DNS issue also, but it somehow worked well with registry:2.8.3, with exactly the same config for s3 in config.yml. Then it became a bit strange.
Do we have any official document for the configuration for registry:3.0.0? I am not sure if the format for the configuration file has changed and thus caused this.
I will also try to check the code if anything changed

[milosgajdos]

The latest docs are here https://distribution.github.io/distribution/about/configuration/

[erfanw]

I seem to find the issue:

In registry 3.0.0-alpha, you have this commit which allows independent setting of "ForcePathStyle" even when the RegionEndpoint is set:
8654a0e

In version 2 this was set to True by default when the RegionEndpoint Parameter was present:
https://github.com/distribution/distribution/blob/v2.8.3/registry/storage/driver/s3-aws/s3.go#L428

I have limited knowledge with s3 configurations, but since in my configuration the "forcepathstyle" option was not present, it went False and caused the errors.

Thanks for your quick replies again and I think this issue can be closed.

[milosgajdos]

Awesome! I remember that PR and I completely forgot to think of that. Thanks, closing.