Docker Registry v3.7.0: Deployment fails using S3 in eu-west-2 region

[juanwolf]

We recently upgraded our cluster to the new version of openshift (v3.7.0), everything was fine until I tried to upgrade manually the docker registry. I incremented the deployment configuration to the new version, but the new pods were continuously crashing.

Version

oc v3.7.0+7ed6862
kubernetes v1.7.6+a08f5eeb62
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://my.master.internal:8443
openshift v3.7.0+7ed6862
kubernetes v1.7.6+a08f5eeb62

Steps To Reproduce

1. Create a S3 bucket in eu-west-2 (London) region
2. Install a registry with v3.7.0 configured to access to the previous bucket

Current Result

The new pods crash, log example:

time="2017-12-01T16:34:13.467831981Z" level=info msg="start registry" distribution_version="v2.6.2+unknown" kubernetes_version=v1.7.6+a08f5eeb62 openshift_version=v3.7.0+7ed6862 
time="2017-12-01T16:34:13.468960939Z" level=info msg="quota enforcement disabled" go.version=go1.8.3 instance.id=02691c95-874c-45a4-b7c3-0a5a858cb69b openshift.logger=registry 
panic: Invalid region provided: eu-west-2
goroutine 1 [running]:
github.com/openshift/origin/vendor/github.com/docker/distribution/registry/handlers.NewApp(0x7f014ee2cd38, 0xc420227c50, 0xc420948a80, 0x7f014ee2cd38)
	/tmp/openshift/build-rpm-release/tito/rpmbuild-origintkMEca/BUILD/origin-3.7.0/_output/local/go/src/github.com/openshift/origin/vendor/github.com/docker/distribution/registry/handlers/app.go:123 +0x4cb
github.com/openshift/origin/pkg/dockerregistry/server.NewApp(0x7f014ee2cd78, 0xc420227c50, 0x30a85c0, 0xc42000e520, 0xc420948a80, 0xc4202baf60, 0x0, 0x0, 0x0, 0x0)
	/tmp/openshift/build-rpm-release/tito/rpmbuild-origintkMEca/BUILD/origin-3.7.0/_output/local/go/src/github.com/openshift/origin/pkg/dockerregistry/server/app.go:89 +0x394
github.com/openshift/origin/pkg/cmd/dockerregistry.Start(0xc420948a80, 0xc4202baf60, 0xc420948a80, 0xc4202baf60)
	/tmp/openshift/build-rpm-release/tito/rpmbuild-origintkMEca/BUILD/origin-3.7.0/_output/local/go/src/github.com/openshift/origin/pkg/cmd/dockerregistry/dockerregistry.go:180 +0x474
github.com/openshift/origin/pkg/cmd/dockerregistry.Execute(0x30a3880, 0xc42000e3d0)
	/tmp/openshift/build-rpm-release/tito/rpmbuild-origintkMEca/BUILD/origin-3.7.0/_output/local/go/src/github.com/openshift/origin/pkg/cmd/dockerregistry/dockerregistry.go:153 +0x214
main.main()
	/tmp/openshift/build-rpm-release/tito/rpmbuild-origintkMEca/BUILD/origin-3.7.0/_output/local/go/src/github.com/openshift/origin/cmd/dockerregistry/main.go:53 +0x264

Expected Result

Registry working smoothly

Let me know if you need any more tests or information.
Thanks!

[mfojtik]

The new regions were added here in January:

distribution/distribution@9e510d6#diff-a9205dcfd4a6f7c2cbe8be01566ff84a

Not sure if/how we dropped this in 3.7 or if the aws library registry uses was bumped and does not include these regions now.

//cc @miminar @legionus

[juanwolf]

Hi @mfojtik,

I digged a bit more on this issue and found that it's related to distribution/distribution-library-image#63.

I saw that a new version of the registry:2 image is available. I'll give a try to build the openshift-registry image and I let you know if it fixed my issue.

[dmage]

We've lost #12451

[bparees]

any reason distribution/distribution#2140 wasn't also contributed to the upstream release-2.6 branch? Do they not accept that sort of change?

[mfojtik]

I can open a PR against that branch in docker-registry, we might need to backport #12451 to release-3.7 branch in origin.

[bparees]

@mfojtik

origin3.7: #17585 (though I don't think we need it).
ose3.7: https://github.com/openshift/ose/pull/959

i'll leave you to push the change through upstream docker-distribution, at which point we'll have to update our openshift fork of docker-distribution and then bump the image-registry deps...