registry connect ceph storage with s3 compatible

[qwjhq]

Use S3 Browser can operate ceph storage with s3 compatible but registry cann't connect ceph storage.

registry config:
storage: cache: layerinfo: inmemory maintenance: uploadpurging: enabled: false delete: enabled: true s3: accesskey: PMPXYYEK53AFMZIM46NZ secretkey: VGFtJgMiT2R2ayjl4o3q7K1hAqlm2EktCtPWEJaK region: us-west-1 regionendpoint: 10.199.128.215:6780 bucket: registry encrypt: false secure: false v4auth: true
images can't be pushed because of the error unknown error completing upload: response completed with error err.message="unknown error"
registry log:

10.100.84.66 - - [09/Feb/2017:17:35:21 +0800] "POST /v2/paas/sshdserver/blobs/uploads/ HTTP/1.1" 500 117 "" "docker/1.10.3-el7.centos go/go1.4.2 git-commit/6aaaac7-unsupported kernel/3.10.0-327.13.1.el7.x86_64 os/linux arch/amd64"
time="2017-02-09T17:35:21.266874429+08:00" level=debug msg="s3aws.PutContent("/docker/registry/v2/repositories/paas/sshdserver/_uploads/5be184b8-bb0f-4382-8cf8-ba18749e8d9d/startedat")" auth.user.name=paasci go.version=go1.7.3 http.request.host=vipdocker-f9nub.vclound.com http.request.id=0b2c51cb-d04b-4cd6-b06f-a76041a7a348 http.request.method=POST http.request.remoteaddr=10.100.84.66 http.request.uri="/v2/paas/sshdserver/blobs/uploads/" http.request.useragent="docker/1.10.3-el7.centos go/go1.4.2 git-commit/6aaaac7-unsupported kernel/3.10.0-327.13.1.el7.x86_64 os/linux arch/amd64" instance.id=8bbbb0f6-7543-45a1-be29-919f5150c1ee service=registry trace.duration=1.841023ms trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).PutContent" trace.id=faa57ca8-c59e-4bfd-82b9-4b45490a8561 trace.line=95 vars.name="paas/sshdserver" version=v2.6.0
time="2017-02-09T17:35:21.266989141+08:00" level=error msg="response completed with error" auth.user.name=paasci err.code=unknown err.detail="s3aws: : \n\tstatus code: 400, request id: " err.message="unknown error" go.version=go1.7.3 http.request.host=vipdocker-f9nub.vclound.com http.request.id=0b2c51cb-d04b-4cd6-b06f-a76041a7a348 http.request.method=POST http.request.remoteaddr=10.100.84.66 http.request.uri="/v2/paas/sshdserver/blobs/uploads/" http.request.useragent="docker/1.10.3-el7.centos go/go1.4.2 git-commit/6aaaac7-unsupported kernel/3.10.0-327.13.1.el7.x86_64 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=4.51584ms http.response.status=500 http.response.written=117 instance.id=8bbbb0f6-7543-45a1-be29-919f5150c1ee service=registry vars.name="paas/sshdserver" version=v2.6.0
10.100.84.66 - - [09/Feb/2017:17:35:21 +0800] "POST /v2/paas/sshdserver/blobs/uploads/ HTTP/1.1" 500 117 "" "docker/1.10.3-el7.centos go/go1.4.2 git-commit/6aaaac7-unsupported kernel/3.10.0-327.13.1.el7.x86_64 os/linux arch/amd64"
time="2017-02-09T17:35:21.305578378+08:00" level=debug msg="s3aws.PutContent("/docker/registry/v2/repositories/paas/sshdserver/_uploads/90cd6d8c-9a9d-4808-b88d-ab869b283752/startedat")" auth.user.name=paasci go.version=go1.7.3 http.request.host=vipdocker-f9nub.vclound.com http.request.id=d8085c78-9334-416f-86f9-a32e0f27aadc http.request.method=POST http.request.remoteaddr=10.100.84.66 http.request.uri="/v2/paas/sshdserver/blobs/uploads/" http.request.useragent="docker/1.10.3-el7.centos go/go1.4.2 git-commit/6aaaac7-unsupported kernel/3.10.0-327.13.1.el7.x86_64 os/linux arch/amd64" instance.id=8bbbb0f6-7543-45a1-be29-919f5150c1ee service=registry trace.duration=41.177395ms trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).PutContent" trace.id=db508c69-7cc7-4bf6-812b-d4a5c872d510 trace.line=95 vars.name="paas/sshdserver" version=v2.6.0
time="2017-02-09T17:35:21.305687803+08:00" level=error msg="response completed with error" auth.user.name=paasci err.code=unknown err.detail="s3aws: : \n\tstatus code: 400, request id: " err.message="unknown error" go.version=go1.7.3 http.request.host=vipdocker-f9nub.vclound.com http.request.id=d8085c78-9334-416f-86f9-a32e0f27aadc http.request.method=POST http.request.remoteaddr=10.100.84.66 http.request.uri="/v2/paas/sshdserver/blobs/uploads/" http.request.useragent="docker/1.10.3-el7.centos go/go1.4.2 git-commit/6aaaac7-unsupported kernel/3.10.0-327.13.1.el7.x86_64 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=43.925479ms http.response.status=500 http.response.written=117 instance.id=8bbbb0f6-7543-45a1-be29-919f5150c1ee service=registry vars.name="paas/sshdserver" version=v2.6.0
10.100.84.66 - - [09/Feb/2017:17:35:21 +0800] "POST /v2/paas/sshdserver/blobs/uploads/ HTTP/1.1" 500 117 "" "docker/1.10.3-el7.centos go/go1.4.2 git-commit/6aaaac7-unsupported kernel/3.10.0-327.13.1.el7.x86_64 os/linux arch/amd64"

docker push message:

docker push vipdocker-f9nub.vclound.com/paas/sshdserver:v1
The push refers to a repository [vipdocker-f9nub.vclound.com/paas/sshdserver]
5f70bf18a086: Retrying in 1 seconds
e11a59a920c9: Retrying in 1 seconds
b9e962564cac: Retrying in 1 seconds
020c6d1376d5: Retrying in 1 seconds
076e9485593f: Retrying in 1 seconds
700274572fe0: Retrying in 5 seconds
10efca284ad1: Retrying in 5 seconds
60b17c837546: Waiting
4ba5ab91c14a: Waiting
164ffaed5f84: Waiting
58b8a3295350: Waiting
614a715e2fc2: Waiting
b48d48e0b455: Waiting
Error: Status 404 trying to push repository paas/sshdserver: "\r\n<title>404 Not Found</title>\r\n<body bgcolor="white">\r\n

404 Not Found

\r\n

---

nginx/1.9.15\r\n\r\n\r\n"

use S3 Browser can see the req message in ceph storage log:

{"log":"2017-02-09 17:32:54.728202 7eff367fc700 1 ====== req done req=0x7eff367f6710 op status=0 http_status=200 ======\r\n","stream":"stdout","time":"2017-02-09T09:32:54.728373154Z"}
{"log":"2017-02-09 17:32:54.728271 7eff367fc700 1 civetweb: 0x7eff200008c0: 10.100.73.151 - - [09/Feb/2017:17:32:54 +0800] "GET / HTTP/1.1" 200 0 - S3 Browser 5-8-9 http://s3browser.com\r\n","stream":"stdout","time":"2017-02-09T09:32:54.728379897Z"}
{"log":"2017-02-09 17:32:54.933684 7eff367fc700 20 RGWEnv::set(): HTTP_USER_AGENT: S3 Browser 5-8-9 http://s3browser.com\r\n","stream":"stdout","time":"2017-02-09T09:32:54.93403509Z"}
{"log":"2017-02-09 17:32:54.933700 7eff367fc700 20 RGWEnv::set(): HTTP_AUTHORIZATION: AWS PMPXYYEK53AFMZIM46NZ:DnkclVLbAMDYiUuppEYhEJLitgA=\r\n","stream":"stdout","time":"2017-02-09T09:32:54.934064756Z"}

use registry connect ceph, ceph no message, and registry no error log yet!

thx all

[qwjhq]

use registry 2.6.0 can auth in ceph and ceph can see logs and can put file to bucket but docker push always retry

ceph logs :

{"log":"2017-02-10 17:13:35.972215 7f01677fe700 10 string to sign = AWS4-HMAC-SHA256\r\n","stream":"stdout","time":"2017-02-10T09:13:35.974002511Z"}
{"log":"2d8cef0e58e142b48290da00a8a3076fe4c79419937f1cb36ab54cf05ebe2e38\r\n","stream":"stdout","time":"2017-02-10T09:13:35.974017521Z"}
{"log":"2017-02-10 17:13:35.972369 7f01677fe700 10 signature_k = 0ebb11f41b0aa2df39f4e7b9d1b73722c70fe97bd0e8faccb3ea3df292330663\r\n","stream":"stdout","time":"2017-02-10T09:13:35.974212679Z"}
{"log":"2017-02-10 17:13:35.972370 7f01677fe700 10 new signature = 0ebb11f41b0aa2df39f4e7b9d1b73722c70fe97bd0e8faccb3ea3df292330663\r\n","stream":"stdout","time":"2017-02-10T09:13:35.974218059Z"}
{"log":"2017-02-10 17:13:35.972372 7f01677fe700 10 ----------------------------- Verifying signatures\r\n","stream":"stdout","time":"2017-02-10T09:13:35.974223442Z"}
{"log":"2017-02-10 17:13:35.972373 7f01677fe700 10 Signature = 0ebb11f41b0aa2df39f4e7b9d1b73722c70fe97bd0e8faccb3ea3df292330663\r\n","stream":"stdout","time":"2017-02-10T09:13:35.974228602Z"}
{"log":"2017-02-10 17:13:35.972374 7f01677fe700 10 New Signature = 0ebb11f41b0aa2df39f4e7b9d1b73722c70fe97bd0e8faccb3ea3df292330663\r\n","stream":"stdout","time":"2017-02-10T09:13:35.974234036Z"}
{"log":"2017-02-10 17:13:35.972375 7f01677fe700 10 -----------------------------\r\n","stream":"stdout","time":"2017-02-10T09:13:35.974239399Z"}
{"log":"2017-02-10 17:13:35.972377 7f01677fe700 10 v4 auth ok\r\n","stream":"stdout","time":"2017-02-10T09:13:35.974244529Z"}
{"log":"2017-02-10 17:13:35.972403 7f01677fe700 10 x\u003e\u003e x-amz-acl:private\r\n","stream":"stdout","time":"2017-02-10T09:13:35.974249563Z"}
{"log":"2017-02-10 17:13:35.972409 7f01677fe700 10 x\u003e\u003e x-amz-content-sha256:0c007fc0c814e2ef56b84ea778a5db6c55638afae839bb4f20823d98d14fd19b\r\n","stream":"stdout","time":"2017-02-10T09:13:35.974255743Z"}
{"log":"2017-02-10 17:13:35.972413 7f01677fe700 10 x\u003e\u003e x-amz-date:20170210T091335Z\r\n","stream":"stdout","time":"2017-02-10T09:13:35.974261626Z"}
{"log":"2017-02-10 17:13:35.972417 7f01677fe700 10 x\u003e\u003e x-amz-server-side-encryption:AES256\r\n","stream":"stdout","time":"2017-02-10T09:13:35.974271759Z"}
{"log":"2017-02-10 17:13:35.972420 7f01677fe700 10 x\u003e\u003e x-amz-storage-class:STANDARD\r\n","stream":"stdout","time":"2017-02-10T09:13:35.974451014Z"}
{"log":"2017-02-10 17:13:35.972433 7f01677fe700 20 get_obj_state: rctx=0x7f01677f7e50 obj=registry:docker/registry/v2/repositories/mysql/_uploads/52ec0be3-3e7b-4864-8d20-b2813a85e427/hashstates/sha256/0 state=0x7f00cc056bd8 s-\u003eprefetch_data=0\r\n","stream":"stdout","time":"2017-02-10T09:13:35.974457624Z"}
{"log":"2017-02-10 17:13:35.972474 7f01677fe700 1 -- 240.30.128.215:0/2429516846 --\u003e 240.30.128.34:6806/135911 -- osd_op(client.6194459.0:393039 47.7bfa45ee 4325fe60-129e-4f42-9e10-7a573e843b7d.6194459.4_docker/registry/v2/repositories/mysql/_uploads/52ec0be3-3e7b-4864-8d20-b2813a85e427/hashstates/sha256/0 [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e2159) v7 -- ?+0 0x7f00cc05e200 con 0x7f0020014e90\r\n","stream":"stdout","time":"2017-02-10T09:13:35.974463947Z"}
{"log":"2017-02-10 17:13:35.973529 7f007c4f5700 1 -- 240.30.128.215:0/2429516846 \u003c== osd.138 240.30.128.34:6806/135911 490 ==== osd_op_reply(393039 4325fe60-129e-4f42-9e10-7a573e843b7d.6194459.4_docker/registry/v2/repositories/mysql/_uploads/52ec0be3-3e7b-4864-8d20-b2813a85e427/hashstates/sha256/0 [getxattrs,stat] v0'0 uv232 ondisk = 0) v7 ==== 312+0+1383 (3643038862 0 3525307965) 0x7efefc05fed0 con 0x7f0020014e90\r\n","stream":"stdout","time":"2017-02-10T09:13:35.974471054Z"}
{"log":"2017-02-10 17:13:35.973660 7f01677fe700 10 manifest: total_size = 157\r\n","stream":"stdout","time":"2017-02-10T09:13:35.974477907Z"}
{"log":"2017-02-10 17:13:35.973669 7f01677fe700 20 get_obj_state: setting s-\u003eobj_tag to 4325fe60-129e-4f42-9e10-7a573e843b7d.6194459.2512\r\n","stream":"stdout","time":"2017-02-10T09:13:35.974483041Z"}
{"log":"2017-02-10 17:13:35.973683 7f01677fe700 20 get_obj_state: rctx=0x7f01677f7e50 obj=registry:docker/registry/v2/repositories/mysql/_uploads/52ec0be3-3e7b-4864-8d20-b2813a85e427/hashstates/sha256/0 state=0x7f00cc056bd8 s-\u003eprefetch_data=0\r\n","stream":"stdout","time":"2017-02-10T09:13:35.974488631Z"}
{"log":"2017-02-10 17:13:35.973705 7f01677fe700 10 setting object write_tag=4325fe60-129e-4f42-9e10-7a573e843b7d.6194459.2603\r\n","stream":"stdout","time":"2017-02-10T09:13:35.974494731Z"}
{"log":"2017-02-10 17:13:35.973758 7f01677fe700 20 reading from default.rgw.data.root:.bucket.meta.registry:4325fe60-129e-4f42-9e10-7a573e843b7d.6194459.4\r\n","stream":"stdout","time":"2017-02-10T09:13:35.974501027Z"}
{"log":"2017-02-10 17:13:35.973766 7f01677fe700 20 get_system_obj_state: rctx=0x7f01677f6a40 obj=default.rgw.data.root:.bucket.meta.registry:4325fe60-129e-4f42-9e10-7a573e843b7d.6194459.4 state=0x7f00cc061598 s-\u003eprefetch_data=0\r\n","stream":"stdout","time":"2017-02-10T09:13:35.974506511Z"}
{"log":"2017-02-10 17:13:35.973773 7f01677fe700 10 cache get: name=default.rgw.data.root+.bucket.meta.registry:4325fe60-129e-4f42-9e10-7a573e843b7d.6194459.4 : hit (requested=22, cached=23)\r\n","stream":"stdout","time":"2017-02-10T09:13:35.974512617Z"}
{"log":"2017-02-10 17:13:35.973781 7f01677fe700 20 get_system_obj_state: s-\u003eobj_tag was set empty\r\n","stream":"stdout","time":"2017-02-10T09:13:35.974518954Z"}
{"log":"2017-02-10 17:13:35.973784 7f01677fe700 10 cache get: name=default.rgw.data.root+.bucket.meta.registry:4325fe60-129e-4f42-9e10-7a573e843b7d.6194459.4 : hit (requested=17, cached=23)\r\n","stream":"stdout","time":"2017-02-10T09:13:35.974670863Z"}
{"log":"2017-02-10 17:13:35.973801 7f01677fe700 20 bucket index object: .dir.4325fe60-129e-4f42-9e10-7a573e843b7d.6194459.4\r\n","stream":"stdout","time":"2017-02-10T09:13:35.974690211Z"}
{"log":"2017-02-10 17:13:35.973838 7f01677fe700 1 -- 240.30.128.215:0/2429516846 --\u003e 240.30.128.213:6810/42173 -- osd_op(client.6194459.0:393040 46.f09eeda3 .dir.4325fe60-129e-4f42-9e10-7a573e843b7d.6194459.4 [call rgw.bucket_prepare_op] snapc 0=[] ondisk+write+known_if_redirected e2159) v7 -- ?+0 0x7f00cc078a80 con 0x7f031c0de120\r\n","stream":"stdout","time":"2017-02-10T09:13:35.974696526Z"}

[janaurka]

I see exactly the same error on my docker-registry and ceph installation. Anything I can do/provide to make this bug more easily solvable?

[bibby]

Same outcome here; layers push and write to ceph, but will retry until fails.

The last items in my registry log is HTTP 307, a temporary redirect that I suspect is not being followed.

INFO[0075] response completed
go.version=go1.7.3
http.request.host=bibby:443
http.request.id=416ab88c-21b0-444b-8cfa-debeb65beb38
http.request.method=HEAD
http.request.remoteaddr=172.16.43.4:34386
http.request.uri=/v2/busybox/blobs/sha256:04176c8b224aa0eb9942af765f66dae866f436e75acef028fe44b8a98e045515
http.request.useragent=docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/3.13.0-113-generic os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \(linux\))
http.response.contenttype=application/octet-stream
http.response.duration=2.607499ms
http.response.status=307
http.response.written=0 instance.id=0a4c5bcc-d7e1-48a3-ad39-982207f02031 version=v2.6.1

172.16.43.4 - - [12/May/2017:15:58:16 +0000] "HEAD /v2/busybox/blobs/sha256:04176c8b224aa0eb9942af765f66dae866f436e75acef028fe44b8a98e045515
http/1.1" 307 0 "" "docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/3.13.0-113-generic os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))"

[gogeof]

I just want to know how is it going? Does it already resolved?

[fenghui2013]

@gogeof using swift. It worked.

[gogeof]

谢谢，但还是不支持 ceph s3 接口对么？咱这边使用的是哪个版本呢。

[fenghui2013]

我们使用的harbor是v1.6.0 ceph是0.94.10，使用swift协议。 s3好像是还不支持的，后面没再继续研究。

[gogeof]

好的，谢谢。再咨询一下，咱这边是在生产环境上使用，还是只是底下自己做的测试验证？

如果是在生产环境上使用，能说一下已经使用多久了么，谢谢。

[fenghui2013]

生产环境上，目前运行两个多月了。运行稳定。

[dimm0]

Is it solved or still pending?

[damoon]

As far as I know, it is not solved.
As a workaround I found to place minio in between the registry and the rados gateway of ceph.
Minio can talk a registry compatible protocol at the front and "translate" it to the protocol version of ceph as the storage backend.

Example https://gitlab.com/utopia-planitia/storage/blob/master/config-templates/minio.yaml.j2

[Raven888888]

+1

We still need this issue resolved, so that we can use registry with ceph s3. Thank you.