docker-registry 0.7.3 : Disk filling up

[sahilsk]

Hi,

In merely 2 days of operation docker registry(v0.7.3) is filling up disk space to 100%. And when we restart the resgitry disk space go back 0, clearing up all cached stuff. Should not be its registry task to purge cache if disk reaching threshold value? Any help here would be great.

I'm running registry inside docker with redis lru cache (it's also containerized) enabled.

$ docker run  --restart=on-failure:10  -d -e SETTINGS_FLAVOR=prod  -e STORAGE_REDIRECT=true -v /opt/registry:/registry-conf  -e DOCKER_REGISTRY_CONFIG=/registry-conf/config.yml --link redis-cache:db --name registry-s3 -p 80:5000 registry:0.7.3

root@prod-registry--10-91-0-13:/tmp# docker ps
CONTAINER ID        IMAGE               COMMAND                CREATED             STATUS              PORTS                     NAMES
dfccc6c7dcsd       registry:0.7.3      "/bin/sh -c 'exec do   2 days ago          Up 2 days           10.x.x.x:80->5000/tcp   registry-s3
8b8389cecd4f       redis:latest        "/entrypoint.sh redi   2 days ago          Up 2 days           6379/tcp                  redis-cache

I am using s3 and my configuration looks like this:

# The `common' part is automatically included (and possibly overriden by all
# other flavors)
common:
    # Bucket for storage
    boto_bucket: REPLACEME

     # Amazon S3 Storage Configuration
    s3_access_key: REPLACEME
    s3_secret_key: REPLACEME
    s3_bucket: REPLACEME
    s3_encrypt: false
    s3_secure: false

    # Set a random string here
    secret_key: REPLACEME


# This is the default configuration when no flavor is specified
dev:
    standalone: true
    storage: file
    storage_path: /tmp/registry
    loglevel: debug

# To specify another flavor, set the environment variable SETTINGS_FLAVOR
# $ export SETTINGS_FLAVOR=prod
prod:
    storage: s3
    storage_path: "/"
    # Enabling LRU cache for small files. This speeds up read/write on small files
    # when using a remote storage backend (like S3).
    standalone: True
    cache:
        host: _env:DB_PORT_6379_TCP_ADDR
        port: 6379
    cache_lru:
        host: _env:DB_PORT_6379_TCP_ADDR
        port: 6379

# This flavor is automatically used by unit tests
test:
    storage: local
    storage_path: /tmp/test

[dmp42]

0.7 is a (quite) old version. You should consider using at least 0.8, or 0.9.

Also, can you investigate and point out what exactly is cluttering disk space? (if you are indeed using S3 there absolutely no reason for this, since there is no filesystem caching).

Thanks.

[sahilsk]

Hi @dmp42

can you help me understand these log lines? I am guessing that registry is caching images in tempfies so that it don't need to request them from s3. However, in doing so it's ignoring remaining disk space... somehow redis queue length is too big .....something like that. What do you suggest?

...
2015-03-07 00:32:03,479 INFO: ParallelKey: <Key: organizationimages,images/43fcdb249be8d2cdb8e46166fc4c399c2f5aa673a3ba90be6ad4d7c1a10a100a/layer>; size=193320751
2015-03-07 00:32:03,494 INFO: ParallelKey: <Key: organizationimages,images/43fcdb249be8d2cdb8e46166fc4c399c2f5aa673a3ba90be6ad4d7c1a10a100a/layer>; size=193320751
2015-03-07 00:32:03,520 INFO: ParallelKey: <Key: organizationimages,images/43fcdb249be8d2cdb8e46166fc4c399c2f5aa673a3ba90be6ad4d7c1a10a100a/layer>; size=193320751
2015-03-07 00:32:03,521 INFO: ParallelKey: <Key: organizationimages,images/43fcdb249be8d2cdb8e46166fc4c399c2f5aa673a3ba90be6ad4d7c1a10a100a/layer>; size=193320751

...
{"log":"2015-03-07 00:33:08,125 INFO: ParallelKey: \u003cKey: organizationimages,images/43fcdb249be8d2cdb8e46166fc4c399c2f5aa673a3ba90be6ad4d7c1a10a100a/l
ayer\u003e; buffering complete at 32.8% of the total transfer; now serving straight from the tempfile\n","stream":"stderr","time":"2015-03-07T00:33:08
.126475811Z"}
{"log":"2015-03-07 00:33:08,682 INFO: ParallelKey: \u003cKey: organizationimages,images/43fcdb249be8d2cdb8e46166fc4c399c2f5aa673a3ba90be6ad4d7c1a10a100a/l
ayer\u003e; buffering complete at 20.0% of the total transfer; now serving straight from the tempfile\n","stream":"stderr","time":"2015-03-07T00:33:08
.682632542Z"}
{"log":"2015-03-07 00:33:13,880 INFO: ParallelKey: \u003cKey: organizationimages,images/43fcdb249be8d2cdb8e46166fc4c399c2f5aa673a3ba90be6ad4d7c1a10a100a/l
ayer\u003e; buffering complete at 22.5% of the total transfer; now serving straight from the tempfile\n","stream":"stderr","time":"2015-03-07T00:33:13
.881879406Z"}
{"log":"2015-03-07 00:33:17,570 INFO: ParallelKey: \u003cKey: organizationimages,images/43fcdb249be8d2cdb8e46166fc4c399c2f5aa673a3ba90be6ad4d7c1a10a100a/l
ayer\u003e; buffering complete at 0.1% of the total transfer; now serving straight from the tempfile\n","stream":"stderr","time":"2015-03-07T00:33:17.
570786514Z"}
{"log":"2015-03-07 00:33:32,329 INFO: ParallelKey: \u003cKey: organizationimages,images/43fcdb249be8d2cdb8e46166fc4c399c2f5aa673a3ba90be6ad4d7c1a10a100a/l
ayer\u003e; buffering complete at 0.8% of the total transfer; now serving straight from the tempfile\n","stream":"stderr","time":"2015-03-07T00:33:32.
330254169Z"}
{"log":"2015-03-07 00:34:41,233 INFO: ParallelKey: \u003cKey: organizationimages,images/d884e0950871aa73997efed77675146931c6fd75fcf0e0606f1fd35b046e079e/l
ayer\u003e; buffering complete at 0.0% of the total transfer; now serving straight from the tempfile\n","stream":"stderr","time":"2015-03-07T00:34:41.
234784507Z"}
{"log":"2015-03-07 00:34:41,312 INFO: ParallelKey: \u003cKey: organizationimages,images/d884e0950871aa73997efed77675146931c6fd75fcf0e0606f1fd35b046e079e/l
ayer\u003e; buffering complete at 60.0% of the total transfer; now serving straight from the tempfile\n","stream":"stderr","time":"2015-03-07T00:34:41
.312638242Z"}
{"log":"2015-03-07 00:34:41,326 INFO: ParallelKey: \u003cKey: organizationimages,images/d884e0950871aa73997efed77675146931c6fd75fcf0e0606f1fd35b046e079e/l
ayer\u003e; buffering complete at 67.5% of the total transfer; now serving straight from the tempfile\n","stream":"stderr","time":"2015-03-07T00:34:41
.326851049Z"}
{"log":"2015-03-07 00:34:41,621 INFO: ParallelKey: \u003cKey: organizationimages,images/d884e0950871aa73997efed77675146931c6fd75fcf0e0606f1fd35b046e079e/l
ayer\u003e; buffering complete at 53.8% of the total transfer; now serving straight from the tempfile\n","stream":"stderr","time":"2015-03-07T00:34:41
.622623091Z"}

[dmp42]

The boto library uses parallel keys to multiplex requests to the s3 backend, and is apparently responsible for this...

I guess we should disable ParallelKeys entirely (see also #956)

[dmp42]

Still, you should really update your registry.