Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Registry becomes permanently unavailable after 30 seconds, using S3 driver #2292

Closed
oms opened this issue May 30, 2017 · 27 comments
Closed

Registry becomes permanently unavailable after 30 seconds, using S3 driver #2292

oms opened this issue May 30, 2017 · 27 comments

Comments

@oms
Copy link

@oms oms commented May 30, 2017

When we spin up a registry on an EC2-instance with an attached IAM instance role (giving access to the backing S3 bucket), the registry becomes consistently unavailable after about 30 seconds. During the initial "grace-period" it seems to work fine, and thereafter it only responds with HTTP - 503.

The above happens every time with registry:2.6.1, have not been able to reproduce so far with registry:2.5.1

I suspect this is the same problem being described in #2283 also.

The specifics of our setup:

config.yml:

version: 0.1
log:
  level: debug
  fields:
    service: registry
storage:
  cache:
    blobdescriptor: inmemory
http:
  addr: :5000
  secret: REDACTED
  debug:
    addr: :5001
headers:
    X-Content-Type-Options: [nosniff]
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3

The invocation:

docker run -d -p 5000:5000 -p 5001:5001 --restart=always --name registry \
 -v `pwd`/config.yml:/etc/docker/registry/config.yml \
 -e "REGISTRY_STORAGE=s3" \
 -e "REGISTRY_STORAGE_S3_REGION=eu-west-1" \
 -e "REGISTRY_STORAGE_S3_BUCKET=REDACTED" \
 registry:2.6.1

Right after running the container or restarting it we see this:

$ curl -v http://localhost:5000
* Rebuilt URL to: http://localhost:5000/
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 5000 (#0)
> GET / HTTP/1.1
> Host: localhost:5000
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Cache-Control: no-cache
< Date: Tue, 30 May 2017 15:20:21 GMT
< Content-Length: 0
< Content-Type: text/plain; charset=utf-8
<
* Connection #0 to host localhost left intact

$ curl -v http://localhost:5001/debug/health
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 5001 (#0)
> GET /debug/health HTTP/1.1
> Host: localhost:5001
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Length: 2
< Content-Type: application/json; charset=utf-8
< Date: Tue, 30 May 2017 15:20:25 GMT
<
* Connection #0 to host localhost left intact
{}
$

Then after about 30 seconds, the following:

$ curl -v http://localhost:5000
* Rebuilt URL to: http://localhost:5000/
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 5000 (#0)
> GET / HTTP/1.1
> Host: localhost:5000
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 503 Service Unavailable
< Content-Type: application/json; charset=utf-8
< Date: Tue, 30 May 2017 14:56:51 GMT
< Content-Length: 125
<
{"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}

$ curl -v http://localhost:5001/debug/health
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 5001 (#0)
> GET /debug/health HTTP/1.1
> Host: localhost:5001
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 503 Service Unavailable
< Content-Length: 47
< Content-Type: application/json; charset=utf-8
< Date: Tue, 30 May 2017 14:56:54 GMT
<
* Connection #0 to host localhost left intact
{"storagedriver_s3":"s3aws: Path not found: /"}

The registry logs:

$ docker logs --tail=200 -f -t registry
2017-05-30T14:40:33.327662022Z time="2017-05-30T14:40:33.327382019Z" level=info msg="redis not configured" go.version=go1.7.3 instance.id=42b8d328-f5cb-4bb3-b3f2-7a93d77febe0 service=registry version=v2.6.1
2017-05-30T14:40:33.336419060Z time="2017-05-30T14:40:33.336028387Z" level=info msg="debug server listening :5001"
2017-05-30T14:40:33.336430327Z time="2017-05-30T14:40:33.336173487Z" level=info msg="Starting upload purge in 45m0s" go.version=go1.7.3 instance.id=42b8d328-f5cb-4bb3-b3f2-7a93d77febe0 service=registry version=v2.6.1
2017-05-30T14:40:33.348947098Z time="2017-05-30T14:40:33.348727425Z" level=info msg="listening on [::]:5000" go.version=go1.7.3 instance.id=42b8d328-f5cb-4bb3-b3f2-7a93d77febe0 service=registry version=v2.6.1
2017-05-30T14:40:43.448078780Z time="2017-05-30T14:40:43.447797929Z" level=debug msg="s3aws.Stat(\"/\")" go.version=go1.7.3 instance.id=42b8d328-f5cb-4bb3-b3f2-7a93d77febe0 service=registry trace.duration=98.746543ms trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat" trace.id=11f43f27-dfb4-4fde-b67b-b40eb3b27d8e trace.line=137 version=v2.6.1
2017-05-30T14:40:53.238568918Z 172.17.0.1 - - [30/May/2017:14:40:53 +0000] "GET / HTTP/1.1" 200 0 "" "curl/7.47.0"
2017-05-30T14:40:53.493194665Z time="2017-05-30T14:40:53.492892001Z" level=debug msg="s3aws.Stat(\"/\")" go.version=go1.7.3 instance.id=42b8d328-f5cb-4bb3-b3f2-7a93d77febe0 service=registry trace.duration=143.883466ms trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat" trace.id=b25a47e5-f0ad-4f85-adc0-53eed3245b0d trace.line=137 version=v2.6.1
2017-05-30T14:40:54.873493591Z 172.17.0.1 - - [30/May/2017:14:40:54 +0000] "GET / HTTP/1.1" 200 0 "" "curl/7.47.0"
2017-05-30T14:40:55.717273385Z 172.17.0.1 - - [30/May/2017:14:40:55 +0000] "GET / HTTP/1.1" 200 0 "" "curl/7.47.0"
2017-05-30T14:40:56.392247414Z 172.17.0.1 - - [30/May/2017:14:40:56 +0000] "GET / HTTP/1.1" 200 0 "" "curl/7.47.0"
2017-05-30T14:41:00.510609780Z 172.17.0.1 - - [30/May/2017:14:41:00 +0000] "GET / HTTP/1.1" 200 0 "" "curl/7.47.0"
2017-05-30T14:41:03.380975062Z time="2017-05-30T14:41:03.380695246Z" level=debug msg="s3aws.Stat(\"/\")" go.version=go1.7.3 instance.id=42b8d328-f5cb-4bb3-b3f2-7a93d77febe0 service=registry trace.duration=31.651662ms trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat" trace.id=49db908e-074b-40f4-8471-3150ff080abf trace.line=137 version=v2.6.1
2017-05-30T14:41:06.701383438Z 172.17.0.1 - - [30/May/2017:14:41:06 +0000] "GET / HTTP/1.1" 503 125 "" "curl/7.47.0"
2017-05-30T14:41:13.379890009Z time="2017-05-30T14:41:13.379588757Z" level=debug msg="s3aws.Stat(\"/\")" go.version=go1.7.3 instance.id=42b8d328-f5cb-4bb3-b3f2-7a93d77febe0 service=registry trace.duration=30.570087ms trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat" trace.id=1d3db1d6-7e5b-4fa4-aee5-3d743a94e2e0 trace.line=137 version=v2.6.1
2017-05-30T14:41:23.377456357Z time="2017-05-30T14:41:23.377151322Z" level=debug msg="s3aws.Stat(\"/\")" go.version=go1.7.3 instance.id=42b8d328-f5cb-4bb3-b3f2-7a93d77febe0 service=registry trace.duration=28.111547ms trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat" trace.id=8d5fd464-e31f-495e-9dc6-fb3710f823a5 trace.line=137 version=v2.6.1

Registry logs after restart of container, it works in the beginning, and then eventually 503:

2017-05-30T15:20:18.474925398Z time="2017-05-30T15:20:18.474654839Z" level=info msg="redis not configured" go.version=go1.7.3 instance.id=84453db6-8e3d-429c-a98b-53cb599946ce service=registry version=v2.6.1
2017-05-30T15:20:18.484491091Z time="2017-05-30T15:20:18.484105064Z" level=info msg="debug server listening :5001"
2017-05-30T15:20:18.484502292Z time="2017-05-30T15:20:18.484249629Z" level=info msg="Starting upload purge in 8m0s" go.version=go1.7.3 instance.id=84453db6-8e3d-429c-a98b-53cb599946ce service=registry version=v2.6.1
2017-05-30T15:20:18.496314674Z time="2017-05-30T15:20:18.496090308Z" level=info msg="listening on [::]:5000" go.version=go1.7.3 instance.id=84453db6-8e3d-429c-a98b-53cb599946ce service=registry version=v2.6.1
2017-05-30T15:20:21.930669941Z 172.17.0.1 - - [30/May/2017:15:20:21 +0000] "GET / HTTP/1.1" 200 0 "" "curl/7.47.0"
2017-05-30T15:20:28.539811199Z time="2017-05-30T15:20:28.539504682Z" level=debug msg="s3aws.Stat(\"/\")" go.version=go1.7.3 instance.id=84453db6-8e3d-429c-a98b-53cb599946ce service=registry trace.duration=43.133214ms trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat" trace.id=c4dbf494-f9fa-4306-9847-cc50909347f5 trace.line=137 version=v2.6.1
2017-05-30T15:20:38.527759086Z time="2017-05-30T15:20:38.527461076Z" level=debug msg="s3aws.Stat(\"/\")" go.version=go1.7.3 instance.id=84453db6-8e3d-429c-a98b-53cb599946ce service=registry trace.duration=31.106894ms trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat" trace.id=be574080-bbe8-49f0-bc79-795ae328db68 trace.line=137 version=v2.6.1
2017-05-30T15:20:48.534808753Z time="2017-05-30T15:20:48.534495526Z" level=debug msg="s3aws.Stat(\"/\")" go.version=go1.7.3 instance.id=84453db6-8e3d-429c-a98b-53cb599946ce service=registry trace.duration=38.070819ms trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat" trace.id=4088eccc-a3fa-47ef-b56d-e56a156005b8 trace.line=137 version=v2.6.1
2017-05-30T15:20:58.528801376Z time="2017-05-30T15:20:58.528503898Z" level=debug msg="s3aws.Stat(\"/\")" go.version=go1.7.3 instance.id=84453db6-8e3d-429c-a98b-53cb599946ce service=registry trace.duration=32.12672ms trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat" trace.id=988f1141-e51a-4afa-a333-d21ad135ca1c trace.line=137 version=v2.6.1
2017-05-30T15:21:05.321744770Z 172.17.0.1 - - [30/May/2017:15:21:05 +0000] "GET / HTTP/1.1" 503 125 "" "curl/7.47.0"
2017-05-30T15:21:08.527621977Z time="2017-05-30T15:21:08.527319714Z" level=debug msg="s3aws.Stat(\"/\")" go.version=go1.7.3 instance.id=84453db6-8e3d-429c-a98b-53cb599946ce service=registry trace.duration=30.94061ms trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat" trace.id=773c73dc-b972-4bcd-9a39-8c688d768b3f trace.line=137 version=v2.6.1
2017-05-30T15:21:18.535229553Z time="2017-05-30T15:21:18.5349485Z" level=debug msg="s3aws.Stat(\"/\")" go.version=go1.7.3 instance.id=84453db6-8e3d-429c-a98b-53cb599946ce service=registry trace.duration=38.546592ms trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat" trace.id=83dfce3b-1eeb-4e38-ba86-1e7e2c04fc7b trace.line=137 version=v2.6.1

Some details about the docker install:

$ docker version
Client:
 Version:      17.03.1-ce
 API version:  1.27
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Mon Mar 27 17:14:09 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.1-ce
 API version:  1.27 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Mon Mar 27 17:14:09 2017
 OS/Arch:      linux/amd64
 Experimental: false


$ docker exec registry registry -v
registry github.com/docker/distribution v2.6.1



$ docker info
Containers: 4
 Running: 1
 Paused: 0
 Stopped: 3
Images: 4
Server Version: 17.03.1-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 20
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-1017-aws
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 990.7 MiB
Name: REDACTED
ID: REDACTED
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
@beniwtv
Copy link

@beniwtv beniwtv commented Jun 4, 2017

I have the exact same issue, but using Minio as the S3 backend with the following config:

"REGISTRY_STORAGE" = "s3"
"REGISTRY_STORAGE_S3_ACCESSKEY" = "xxxx"
"REGISTRY_STORAGE_S3_SECRETKEY" = "xxxx"
"REGISTRY_STORAGE_S3_REGIONENDPOINT" = "https://minio.xxxx.com/"
"REGISTRY_STORAGE_S3_REGION" = "europe"
"REGISTRY_STORAGE_S3_BUCKET" = "my-registry"
"REGISTRY_STORAGE_S3_ROOTDIRECTORY" = "/"

/debug/health gives:

{"storagedriver_s3":"s3aws: Path not found: /"}

In the logs I have lots of:

time="2017-06-04T14:50:52.994826233Z" level=debug msg="s3aws.Stat(\"/\")" go.version=go1.7.3 instance.id=792edd67-1849-4858-9e97-42bcfa1416c5 service=registry trace.duration=231.509999ms trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat" trace.id=4c11da5e-c66f-42f4-94ea-b695fed5ff8a trace.line=137 version=v2.6.1 
time="2017-06-04T14:51:03.003936963Z" level=debug msg="s3aws.Stat(\"/\")" go.version=go1.7.3 instance.id=792edd67-1849-4858-9e97-42bcfa1416c5 service=registry trace.duration=247.98118ms trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat" trace.id=a1b9f6e4-38f7-474c-af7a-f076653f0370 trace.line=137 version=v2.6.1 
time="2017-06-04T14:51:13.470571461Z" level=debug msg="s3aws.Stat(\"/\")" go.version=go1.7.3 instance.id=792edd67-1849-4858-9e97-42bcfa1416c5 service=registry trace.duration=714.607494ms trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat" trace.id=319bedf6-c7de-45d3-9e70-71b5e9b1345a trace.line=137 version=v2.6.1

Any ideas?

@pollosp
Copy link

@pollosp pollosp commented Jun 8, 2017

Same here.

@pollosp
Copy link

@pollosp pollosp commented Jun 8, 2017

I don't know if it is something related with S3 interface and IAM , but with this:

resource "aws_s3_bucket" "docker_registry" {
  bucket = "docker.registry.${data.aws_caller_identity.current.account_id}"
  acl = "private"
  force_destroy = "false"
  versioning {
    enabled = true
  }
}

# IAM

resource "aws_iam_user" "docker_registry" {
  name = "docker_registry"
  path = "/"
}

resource "aws_iam_access_key" "docker_registry" {
  user = "${aws_iam_user.docker_registry.name}"
}

resource "aws_iam_user_policy" "docker_registry" {
  name = "docker_registry_policy"
  user = "${aws_iam_user.docker_registry.name}"
  policy = "${data.aws_iam_policy_document.docker_registry.json}"
}

data "aws_iam_policy_document" "docker_registry" {
  statement {

    actions = [
      "s3:ListBucket",
      "s3:GetBucketLocation",
      "s3:ListBucketMultipartUploads"
    ]

    resources = [
      "arn:aws:s3:::${aws_s3_bucket.docker_registry.id}"
    ]
  }

  statement {

    actions = [
      "s3:PutObject",
      "s3:GetObject",
      "s3:DeleteObject",
      "s3:ListMultipartUploadParts",
      "s3:AbortMultipartUpload"
    ]

    resources = [
      "arn:aws:s3:::${aws_s3_bucket.docker_registry.id}/*"
    ]
  }

  statement {

    actions = [
      "kms:Encrypt",
      "kms:Decrypt",
      "kms:ReEncrypt*",
      "kms:GenerateDataKey*",
      "kms:DescribeKey"
    ]

    resources = [
      "*"
    ]
  }
}

output "docker_registry_access_key" {
  value = "${aws_iam_access_key.docker_registry.id}"
}

output "docker_registry_access_key_secret" {
  value = "${aws_iam_access_key.docker_registry.secret}"
}

output "docker_registry_bucket" {
  value = "${aws_s3_bucket.docker_registry.bucket}"
}

And this

REGISTRY_HTTP_ADDR=:5000
REGISTRY_AUTH=htpasswd
REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm
REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd
REGISTRY_STORAGE=s3
REGISTRY_STORAGE_S3_REGION=us-west-1
REGISTRY_STORAGE_S3_BUCKET=wadus-registry.084720738044
REGISTRY_STORAGE_S3_ACCESSKEY=XXXXXXXXXXX
REGISTRY_STORAGE_S3_SECRETKEY=XXXXXXXXXXXX
REGISTRY_STORAGE_CACHE_BLOBDESCRIPTOR=inmemory

It worked
docker run --env-file env -d -p 5000:5000 --name registry -vpwd/htpasswd:/auth/htpasswd registry:2

@beniwtv
Copy link

@beniwtv beniwtv commented Jun 12, 2017

So... I started the registry again through a Docker command (as opposed to Nomad), and suddenly it started working - even the Nomad one.

@liclac
Copy link

@liclac liclac commented Jul 11, 2017

Getting the same issue, using both S3 and OpenStack Swift. In both cases, /debug/health returns either {"storagedriver_s3":"s3aws: Path not found: /"} or {"storagedriver_swift":"swift: Path not found: /"}, depending on the driver in use.

Downgrading to 2.5 solves it for now.

@stevvooe
Copy link
Collaborator

@stevvooe stevvooe commented Jul 18, 2017

Looks like the health check against the backend is failing. You can disable the health check and this should go away.

The problem seems to be related to a regression from 8e915d6 and #2115. List was changed to a Stat to reduce load. We should probably consider a not found error as healthy, as that represents a response from the backend.

@deterralba
Copy link

@deterralba deterralba commented Aug 9, 2017

Same issue here with switf.

Btw for the new guys here (like me), to check if your problem is related : run your container with -p 5001:5001 -e REGISTRY_HTTP_DEBUG_ADDR=0.0.0.0:5001 and then curl localhost:5001/debug/health.

Disabling the health check (with -e REGISTRY_HEALTH_STORAGEDRIVER_ENABLED=false) fix the issue (unless you use Let'sEncrypt and they issued too many certs for you while you were debugging... but that's another story - any idea on how to avoid that?)

@liclac
Copy link

@liclac liclac commented Aug 13, 2017

For me, it seems to have been fixed by disabling health checks, pushing any image into the registry and then reenabling them. Using Swift as my backend.

docker pull busybox
docker tag busybox registry.default.svc.cluster.local/busybox
docker push registry.default.svc.cluster.local/busybox
@dumyan
Copy link

@dumyan dumyan commented Aug 18, 2017

Hit the same issue when using S3 as a storage backend and as @liclac described this happens only when the bucket is empty. After pushing any image the health checks can be re-enabled.

@stevvooe
Copy link
Collaborator

@stevvooe stevvooe commented Aug 21, 2017

#2377 should fix this, but we need a confirmation.

Sorry about the wait!

@oms
Copy link
Author

@oms oms commented Aug 30, 2017

@stevvooe Testing with with a custom registry build, including the #2377 changes I can no longer reproduce the issue, so it seems fixed. 👍 😄

@miquella
Copy link

@miquella miquella commented Sep 29, 2017

Additionally, if you're looking for an interim workaround, pushing an empty .keep file into the bucket resolved the issue for us.

@farshidtz
Copy link

@farshidtz farshidtz commented Oct 13, 2017

The issue still exists in Registry v2.6.2. I had to disable the health check until the bucket was populated with some files and then enabled it again.

@oms
Copy link
Author

@oms oms commented Oct 13, 2017

@farshidtz I believe the 2.6.2 was released a bit before this fix was merged, so a new release will probably be necessary.

@stevvooe
Copy link
Collaborator

@stevvooe stevvooe commented Oct 23, 2017

@dmcgowan Are we going to backport this to 2.6.2?

@ffedoroff
Copy link

@ffedoroff ffedoroff commented Feb 21, 2018

fyi: I had related problem when s3 bucket was empty. After I put sample /docker/sample.file into the bucket it starts works for me.

@Vad1mo
Copy link

@Vad1mo Vad1mo commented Feb 22, 2018

With 2.6.2 we see this error if the container/bucket is empty or not created. will there be a 2.6.3 @stevvooe?

The strange thing is that it did work in the past,

@Rocklviv
Copy link

@Rocklviv Rocklviv commented Feb 22, 2018

The issue with empty S3 bucket causes the 503 errors on any request to the registry.
Docker distribution version: 2.6.2

@ulm0
Copy link

@ulm0 ulm0 commented Apr 4, 2018

Same here, get 500's error as mentioned in minio/minio#1326

Debugging the image it says "error: unknown blob", though before i was able to push the same image when using filesystem storage.

$ docker push r.ulm0.xyz/cockroach:v1.1.6-slim
The push refers to repository [r.ulm0.xyz/cockroach]
ad26c81a8728: Pushed 
139438579314: Pushed 
received unexpected HTTP status: 500 Internal Server Error

Registry config:

version: 0.1
log:
  accesslog:
    disabled: true
  level: debug
  formatter: text
  fields:
    service: registry
    environment: production
  hooks:
    - type: mail
      disabled: true
      levels:
        - panic
      options:
        smtp:
          addr: mail.domain.com:587
          username: no-reply@domain.com
          password: [REDACTED]
          insecure: false
        from: no-reply@domain.com
        to:
          - mail@domain.com
storage:
  s3:
    accesskey: [REDACTED]
    secretkey: [REDACTED]
    region: cl-north-1
    regionendpoint: https://s.domain.com
    bucket: registry
    encrypt: false
    secure: true
    v4auth: true
    chunksize: 5242880
    rootdirectory: /
  delete:
    enabled: true
  redirect:
    disable: false
  cache:
    blobdescriptor: redis
  maintenance:
    uploadpurging:
      enabled: true
      age: 168h
      interval: 24h
      dryrun: false
    readonly:
      enabled: false
http:
  addr: :5000
  net: tcp
  secret: [REDACTED]
  host: https://r.ulm0.xyz
  headers:
    X-Content-Type-Options: [nosniff]
  http2:
    disabled: false
redis:
  addr: registry_cache:6379
  dialtimeout: 10ms
  readtimeout: 10ms
  writetimeout: 10ms
  pool:
    maxidle: 16
    maxactive: 64
    idletimeout: 300s
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3
  tcp:
    - addr: registry_cache:6379
      timeout: 3s
      interval: 10s
      threshold: 3

Minio config:

...
"credential": {
		"accessKey": "[REDACTED]",
		"secretKey": "[REDACTED]"
},
"region": "cl-north-1",
"browser": "on",
"domain": "s.domain.com",
...
@Rocklviv
Copy link

@Rocklviv Rocklviv commented Apr 4, 2018

@ulm0, The actual problem is with the docker distribution itself. In case if you are using newly created bucket you will get 500 error. Unfortunately, docker distribution hasn't fixed it, but there is a workaround.
Just upload an empty file into a root of your bucket. This will fix a problem with 500 error.

@ulm0
Copy link

@ulm0 ulm0 commented Apr 4, 2018

Apparently it was something related to the cache, had to flush redis...now 500's are gone for good.

@ketoiloihp
Copy link

@ketoiloihp ketoiloihp commented Jul 11, 2018

I also has the same problem when I run docker-registry with IAM role.
The problem here S3 folder is empty when docker-registry connect to s3 it get nothing. I tried to upload a blank file like Readme.md or index.html and the problem is resolved.

@dehypnosis
Copy link

@dehypnosis dehypnosis commented Sep 7, 2018

@Rocklviv You saved my day. thankkkks

@dtheodor
Copy link

@dtheodor dtheodor commented Sep 13, 2018

The swift workaround is to create an empty files file, then the healthcheck succeeds

@phs
Copy link

@phs phs commented Oct 8, 2018

Just upload an empty file into a root of your bucket.

I found I needed to upload it to the root of my rootdirectory prefix.

@cmeisinger
Copy link

@cmeisinger cmeisinger commented Oct 15, 2018

Confirming this is still an issue.

@palnabarun
Copy link

@palnabarun palnabarun commented Dec 4, 2018

@stevvooe The problem still persists. Can there be any fix other than workarounds?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

You can’t perform that action at this time.