docker, containerd: image registry mirrors #1629

etungsten · 2021-06-23T19:48:04Z

Issue number:
Resolves #1577

Description of changes:

Author: Erikson Tung <etung@amazon.com>
Date:   Wed Jun 23 15:22:11 2021 -0700

    migrations: migrate `settings.container-registry`, and related models
    
    Adds migrations for the new `container-registry` setting and the new
    docker daemon config file and the new config file for host-containers
    and bootstrap containers.

Author: Erikson Tung <etung@amazon.com>
Date:   Mon Jul 26 22:52:09 2021 -0700

    host-ctr: add support for container image registries
    
    Adds support for configuring container image registrys for
    host-containers and bootstrap containers. A lot of the logic is
    borrowed from containerd-cri plugin's implementation of image registry
    mirrors.

Author: Erikson Tung <etung@amazon.com>
Date:   Tue Jun 22 15:12:14 2021 -0700

    docker, containerd: container image registry mirrors
    
    Adds a new setting `settings.container-registry.mirrors` that lets users set
    pull-through caches for container image registries.

Testing done:

Built AMIs and launched an ECS host and an K8S host.

Set up a pull-through cache for docker hub on a separate host:

sudo docker run \
  -d \
  -p 80:5000 \
  --restart=always \
  --name=through-cache \
  -e REGISTRY_PROXY_REMOTEURL="https://registry-1.docker.io" \
  -e REGISTRY_PROXY_USERNAME=snip \
  -e REGISTRY_PROXY_PASSWORD=snip \
  registry

Set the image registry mirror to the pull-through cache host on both ECS and K8S hosts:

apiclient set --json '{"container-registry":{"mirrors":{"docker.io":["http://my-pull-through-cache-host-ip"]}}}'

Tried running a nginx task on in the ECS cluster, observed that the image was pulled from the cache
The pull-through cache registry is responding to the requests for manifests and blobs with HTTP status code 200:

time="2021-06-23T20:40:09.738518303Z" level=info msg="response completed" go.version=go1.11.2 http.request.host=44.234.86.205 http.request.id=75a15a82-6b80-4508-ad7a-9cc17b1a1e84 http.request.method=GET http.request.remoteaddr="54.184.78.250:44052" http.request.uri="/v2/" http.request.useragent="docker/20.10.4 go/go1.16.3 git-commit/363e9a88a11be517d9e8c65c998ff56f774eb4dc kernel/5.10.35 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.016725ms http.response.status=200 http.response.written=2 
54.184.78.250 - - [23/Jun/2021:20:40:09 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "docker/20.10.4 go/go1.16.3 git-commit/363e9a88a11be517d9e8c65c998ff56f774eb4dc kernel/5.10.35 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)"
time="2021-06-23T20:40:10.494864674Z" level=info msg="response completed" go.version=go1.11.2 http.request.host=44.234.86.205 http.request.id=de6ec364-34d4-4648-8f30-9613f5f23512 http.request.method=HEAD http.request.remoteaddr="54.184.78.250:44054" http.request.uri="/v2/library/nginx/manifests/mainline" http.request.useragent="docker/20.10.4 go/go1.16.3 git-commit/363e9a88a11be517d9e8c65c998ff56f774eb4dc kernel/5.10.35 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)" http.response.contenttype="application/vnd.docker.distribution.manifest.list.v2+json" http.response.duration=753.789996ms http.response.status=200 http.response.written=1862 
54.184.78.250 - - [23/Jun/2021:20:40:09 +0000] "HEAD /v2/library/nginx/manifests/mainline HTTP/1.1" 200 1862 "" "docker/20.10.4 go/go1.16.3 git-commit/363e9a88a11be517d9e8c65c998ff56f774eb4dc kernel/5.10.35 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)"
54.184.78.250 - - [23/Jun/2021:20:40:10 +0000] "GET /v2/library/nginx/manifests/sha256:47ae43cdfc7064d28800bc42e79a429540c7c80168e8c8952778c0d5af1c09db HTTP/1.1" 200 1862 "" "docker/20.10.4 go/go1.16.3 git-commit/363e9a88a11be517d9e8c65c998ff56f774eb4dc kernel/5.10.35 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)"
time="2021-06-23T20:40:10.509265258Z" level=info msg="response completed" go.version=go1.11.2 http.request.host=44.234.86.205 http.request.id=b60c65c4-6909-4b6a-89dc-a5e95add0ec8 http.request.method=GET http.request.remoteaddr="54.184.78.250:44056" http.request.uri="/v2/library/nginx/manifests/sha256:47ae43cdfc7064d28800bc42e79a429540c7c80168e8c8952778c0d5af1c09db" http.request.useragent="docker/20.10.4 go/go1.16.3 git-commit/363e9a88a11be517d9e8c65c998ff56f774eb4dc kernel/5.10.35 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)" http.response.contenttype="application/vnd.docker.distribution.manifest.list.v2+json" http.response.duration=1.813035ms http.response.status=200 http.response.written=1862 
54.184.78.250 - - [23/Jun/2021:20:40:10 +0000] "GET /v2/library/nginx/manifests/sha256:2f1cd90e00fe2c991e18272bb35d6a8258eeb27785d121aa4cc1ae4235167cfd HTTP/1.1" 200 1570 "" "docker/20.10.4 go/go1.16.3 git-commit/363e9a88a11be517d9e8c65c998ff56f774eb4dc kernel/5.10.35 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)"
time="2021-06-23T20:40:10.525015041Z" level=info msg="response completed" go.version=go1.11.2 http.request.host=44.234.86.205 http.request.id=da96c670-692a-49aa-8eee-a20a3cf115cb http.request.method=GET http.request.remoteaddr="54.184.78.250:44058" http.request.uri="/v2/library/nginx/manifests/sha256:2f1cd90e00fe2c991e18272bb35d6a8258eeb27785d121aa4cc1ae4235167cfd" http.request.useragent="docker/20.10.4 go/go1.16.3 git-commit/363e9a88a11be517d9e8c65c998ff56f774eb4dc kernel/5.10.35 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)" http.response.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.response.duration=1.788932ms http.response.status=200 http.response.written=1570 
54.184.78.250 - - [23/Jun/2021:20:40:10 +0000] "GET /v2/library/nginx/blobs/sha256:b82f7f888feb03d38fed4dad68d7265a8b276f1f0c543d549fc6ef30b42c00eb HTTP/1.1" 200 667 "" "docker/20.10.4 go/go1.16.3 git-commit/363e9a88a11be517d9e8c65c998ff56f774eb4dc kernel/5.10.35 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)"
time="2021-06-23T20:40:10.550816892Z" level=info msg="response completed" go.version=go1.11.2 http.request.host=44.234.86.205 http.request.id=83e13e41-e792-4393-a72e-5dba761a479f http.request.method=GET http.request.remoteaddr="54.184.78.250:44072" http.request.uri="/v2/library/nginx/blobs/sha256:b82f7f888feb03d38fed4dad68d7265a8b276f1f0c543d549fc6ef30b42c00eb" http.request.useragent="docker/20.10.4 go/go1.16.3 git-commit/363e9a88a11be517d9e8c65c998ff56f774eb4dc kernel/5.10.35 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)" http.response.contenttype="application/octet-stream" http.response.duration=5.145482ms http.response.status=200 http.response.written=667 
time="2021-06-23T20:40:10.553786666Z" level=info msg="response completed" go.version=go1.11.2 http.request.host=44.234.86.205 http.request.id=844055df-4f8b-4f52-bbb6-cf76cd6e1699 http.request.method=GET http.request.remoteaddr="54.184.78.250:44066" http.request.uri="/v2/library/nginx/blobs/sha256:03e6a245275128e26fc650e724e3fc4510d81f8111bae35ece70242b0a638215" http.request.useragent="docker/20.10.4 go/go1.16.3 git-commit/363e9a88a11be517d9e8c65c998ff56f774eb4dc kernel/5.10.35 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)" http.response.contenttype="application/octet-stream" http.response.duration=5.994485ms http.response.status=200 http.response.written=895 
54.184.78.250 - - [23/Jun/2021:20:40:10 +0000] "GET /v2/library/nginx/blobs/sha256:03e6a245275128e26fc650e724e3fc4510d81f8111bae35ece70242b0a638215 HTTP/1.1" 200 895 "" "docker/20.10.4 go/go1.16.3 git-commit/363e9a88a11be517d9e8c65c998ff56f774eb4dc kernel/5.10.35 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)"
time="2021-06-23T20:40:10.55698674Z" level=info msg="response completed" go.version=go1.11.2 http.request.host=44.234.86.205 http.request.id=9b3cea70-112a-43ea-bcca-36e4f4107980 http.request.method=GET http.request.remoteaddr="54.184.78.250:44060" http.request.uri="/v2/library/nginx/blobs/sha256:4f380adfc10f4cd34f775ae57a17d2835385efd5251d6dfe0f246b0018fb0399" http.request.useragent="docker/20.10.4 go/go1.16.3 git-commit/363e9a88a11be517d9e8c65c998ff56f774eb4dc kernel/5.10.35 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)" http.response.contenttype="application/octet-stream" http.response.duration=23.66608ms http.response.status=200 http.response.written=7733 
54.184.78.250 - - [23/Jun/2021:20:40:10 +0000] "GET /v2/library/nginx/blobs/sha256:4f380adfc10f4cd34f775ae57a17d2835385efd5251d6dfe0f246b0018fb0399 HTTP/1.1" 200 7733 "" "docker/20.10.4 go/go1.16.3 git-commit/363e9a88a11be517d9e8c65c998ff56f774eb4dc kernel/5.10.35 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)"
time="2021-06-23T20:40:10.560957612Z" level=info msg="response completed" go.version=go1.11.2 http.request.host=44.234.86.205 http.request.id=2c20e6a6-e5e3-4d06-bd58-ef49fd6572ef http.request.method=GET http.request.remoteaddr="54.184.78.250:44068" http.request.uri="/v2/library/nginx/blobs/sha256:b21fed559b9f420d83f8e38ca08d1ac4f15298a3ae02c6de56f364bee2299f78" http.request.useragent="docker/20.10.4 go/go1.16.3 git-commit/363e9a88a11be517d9e8c65c998ff56f774eb4dc kernel/5.10.35 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)" http.response.contenttype="application/octet-stream" http.response.duration=7.948473ms http.response.status=200 http.response.written=602 
54.184.78.250 - - [23/Jun/2021:20:40:10 +0000] "GET /v2/library/nginx/blobs/sha256:b21fed559b9f420d83f8e38ca08d1ac4f15298a3ae02c6de56f364bee2299f78 HTTP/1.1" 200 602 "" "docker/20.10.4 go/go1.16.3 git-commit/363e9a88a11be517d9e8c65c998ff56f774eb4dc kernel/5.10.35 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)"
time="2021-06-23T20:40:10.56643432Z" level=info msg="response completed" go.version=go1.11.2 http.request.host=44.234.86.205 http.request.id=9bb0785e-9832-4ecf-89df-db7328c4f5ef http.request.method=GET http.request.remoteaddr="54.184.78.250:44070" http.request.uri="/v2/library/nginx/blobs/sha256:5430e98eba646ef4a34baff035f6f7483761c873711febd48fbcca38d7890c1e" http.request.useragent="docker/20.10.4 go/go1.16.3 git-commit/363e9a88a11be517d9e8c65c998ff56f774eb4dc kernel/5.10.35 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)" http.response.contenttype="application/octet-stream" http.response.duration=4.082108ms http.response.status=200 http.response.written=1397 
54.184.78.250 - - [23/Jun/2021:20:40:10 +0000] "GET /v2/library/nginx/blobs/sha256:5430e98eba646ef4a34baff035f6f7483761c873711febd48fbcca38d7890c1e HTTP/1.1" 200 1397 "" "docker/20.10.4 go/go1.16.3 git-commit/363e9a88a11be517d9e8c65c998ff56f774eb4dc kernel/5.10.35 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)"
54.184.78.250 - - [23/Jun/2021:20:40:10 +0000] "GET /v2/library/nginx/blobs/sha256:b4d181a07f8025e00e0cb28f1cc14613da2ce26450b80c54aea537fa93cf3bda HTTP/1.1" 200 27145851 "" "docker/20.10.4 go/go1.16.3 git-commit/363e9a88a11be517d9e8c65c998ff56f774eb4dc kernel/5.10.35 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)"
time="2021-06-23T20:40:10.754292088Z" level=info msg="response completed" go.version=go1.11.2 http.request.host=44.234.86.205 http.request.id=63d7ffbc-1b7d-47cb-8357-54672a3de54f http.request.method=GET http.request.remoteaddr="54.184.78.250:44062" http.request.uri="/v2/library/nginx/blobs/sha256:b4d181a07f8025e00e0cb28f1cc14613da2ce26450b80c54aea537fa93cf3bda" http.request.useragent="docker/20.10.4 go/go1.16.3 git-commit/363e9a88a11be517d9e8c65c998ff56f774eb4dc kernel/5.10.35 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)" http.response.contenttype="application/octet-stream" http.response.duration=220.138344ms http.response.status=200 http.response.written=27145851 
54.184.78.250 - - [23/Jun/2021:20:40:10 +0000] "GET /v2/library/nginx/blobs/sha256:edb81c9bc1f5416a41e5bea21748dc912772fedbd4bd90e5e3ebfe16b453edce HTTP/1.1" 200 26580118 "" "docker/20.10.4 go/go1.16.3 git-commit/363e9a88a11be517d9e8c65c998ff56f774eb4dc kernel/5.10.35 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)"
time="2021-06-23T20:40:10.767215744Z" level=info msg="response completed" go.version=go1.11.2 http.request.host=44.234.86.205 http.request.id=cdbbbb46-0c3e-42da-b1db-18717b747376 http.request.method=GET http.request.remoteaddr="54.184.78.250:44064" http.request.uri="/v2/library/nginx/blobs/sha256:edb81c9bc1f5416a41e5bea21748dc912772fedbd4bd90e5e3ebfe16b453edce" http.request.useragent="docker/20.10.4 go/go1.16.3 git-commit/363e9a88a11be517d9e8c65c998ff56f774eb4dc kernel/5.10.35 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)" http.response.contenttype="application/octet-stream" http.response.duration=233.865597ms http.response.status=200 http.response.written=26580118

Ran an nginx pod in my K8S cluster that has a single node:

kubectl --kubeconfig bottlerocket.kubeconfig run nginx --image=nginx

The image got pulled from the pull-through cache registry just like above:

18.236.180.8 - - [23/Jun/2021:19:36:37 +0000] "HEAD /v2/library/nginx/manifests/latest?ns=docker.io HTTP/1.1" 200 1862 "" "containerd/1.4.6+bottlerocket"
time="2021-06-23T19:36:37.944166513Z" level=info msg="response completed" go.version=go1.11.2 http.request.host=44.234.86.205 http.request.id=a2678ced-2d46-4fe3-9595-56d04ab029c4 http.request.method=HEAD http.request.remoteaddr="18.236.180.8:50412" http.request.uri="/v2/library/nginx/manifests/latest?ns=docker.io" http.request.useragent="containerd/1.4.6+bottlerocket" http.response.contenttype="application/vnd.docker.distribution.manifest.list.v2+json" http.response.duration=728.409568ms http.response.status=200 http.response.written=1862 
18.236.180.8 - - [23/Jun/2021:19:36:37 +0000] "GET /v2/library/nginx/manifests/sha256:47ae43cdfc7064d28800bc42e79a429540c7c80168e8c8952778c0d5af1c09db?ns=docker.io HTTP/1.1" 200 1862 "" "containerd/1.4.6+bottlerocket"
time="2021-06-23T19:36:37.960599554Z" level=info msg="response completed" go.version=go1.11.2 http.request.host=44.234.86.205 http.request.id=abbec0a6-6372-4aef-9ef3-d0c71d9c8a9b http.request.method=GET http.request.remoteaddr="18.236.180.8:23923" http.request.uri="/v2/library/nginx/manifests/sha256:47ae43cdfc7064d28800bc42e79a429540c7c80168e8c8952778c0d5af1c09db?ns=docker.io" http.request.useragent="containerd/1.4.6+bottlerocket" http.response.contenttype="application/vnd.docker.distribution.manifest.list.v2+json" http.response.duration=4.318429ms http.response.status=200 http.response.written=1862 
time="2021-06-23T19:36:38.220117889Z" level=info msg="Adding new scheduler entry for library/nginx@sha256:2f1cd90e00fe2c991e18272bb35d6a8258eeb27785d121aa4cc1ae4235167cfd with ttl=167h59m59.999997602s" go.version=go1.11.2 instance.id=1470de18-b2bd-4f24-b811-4f48dd848efb service=registry version=v2.7.1 
18.236.180.8 - - [23/Jun/2021:19:36:37 +0000] "GET /v2/library/nginx/manifests/sha256:2f1cd90e00fe2c991e18272bb35d6a8258eeb27785d121aa4cc1ae4235167cfd?ns=docker.io HTTP/1.1" 200 1570 "" "containerd/1.4.6+bottlerocket"
time="2021-06-23T19:36:38.220422328Z" level=info msg="response completed" go.version=go1.11.2 http.request.host=44.234.86.205 http.request.id=e4effd51-2b0f-4092-878b-c6c744d2815c http.request.method=GET http.request.remoteaddr="18.236.180.8:23923" http.request.uri="/v2/library/nginx/manifests/sha256:2f1cd90e00fe2c991e18272bb35d6a8258eeb27785d121aa4cc1ae4235167cfd?ns=docker.io" http.request.useragent="containerd/1.4.6+bottlerocket" http.response.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.response.duration=242.563687ms http.response.status=200 http.response.written=1570 
time="2021-06-23T19:36:38.57423413Z" level=info msg="response completed" go.version=go1.11.2 http.request.host=44.234.86.205 http.request.id=207b8564-62f8-4ddd-941d-0af91e6d30d3 http.request.method=GET http.request.remoteaddr="18.236.180.8:23923" http.request.uri="/v2/library/nginx/blobs/sha256:4f380adfc10f4cd34f775ae57a17d2835385efd5251d6dfe0f246b0018fb0399?ns=docker.io" http.request.useragent="containerd/1.4.6+bottlerocket" http.response.contenttype="application/octet-stream" http.response.duration=335.486858ms http.response.status=200 http.response.written=7733 
18.236.180.8 - - [23/Jun/2021:19:36:38 +0000] "GET /v2/library/nginx/blobs/sha256:4f380adfc10f4cd34f775ae57a17d2835385efd5251d6dfe0f246b0018fb0399?ns=docker.io HTTP/1.1" 200 7733 "" "containerd/1.4.6+bottlerocket"
time="2021-06-23T19:36:38.659022282Z" level=info msg="Adding new scheduler entry for library/nginx@sha256:4f380adfc10f4cd34f775ae57a17d2835385efd5251d6dfe0f246b0018fb0399 with ttl=167h59m59.999998257s" go.version=go1.11.2 instance.id=1470de18-b2bd-4f24-b811-4f48dd848efb service=registry version=v2.7.1 
time="2021-06-23T19:36:38.956560519Z" level=info msg="Adding new scheduler entry for library/nginx@sha256:5430e98eba646ef4a34baff035f6f7483761c873711febd48fbcca38d7890c1e with ttl=167h59m59.999997352s" go.version=go1.11.2 instance.id=1470de18-b2bd-4f24-b811-4f48dd848efb service=registry version=v2.7.1 
time="2021-06-23T19:36:38.976072094Z" level=info msg="response completed" go.version=go1.11.2 http.request.host=44.234.86.205 http.request.id=be13ba01-f5ab-4f85-ae70-d7e6225b55f8 http.request.method=GET http.request.remoteaddr="18.236.180.8:23923" http.request.uri="/v2/library/nginx/blobs/sha256:5430e98eba646ef4a34baff035f6f7483761c873711febd48fbcca38d7890c1e?ns=docker.io" http.request.useragent="containerd/1.4.6+bottlerocket" http.response.contenttype="application/octet-stream" http.response.duration=368.70882ms http.response.status=200 http.response.written=1397 
18.236.180.8 - - [23/Jun/2021:19:36:38 +0000] "GET /v2/library/nginx/blobs/sha256:5430e98eba646ef4a34baff035f6f7483761c873711febd48fbcca38d7890c1e?ns=docker.io HTTP/1.1" 200 1397 "" "containerd/1.4.6+bottlerocket"
time="2021-06-23T19:36:39.213741826Z" level=info msg="response completed" go.version=go1.11.2 http.request.host=44.234.86.205 http.request.id=f01a0bd3-1985-4c4b-a888-728486d11566 http.request.method=GET http.request.remoteaddr="18.236.180.8:18627" http.request.uri="/v2/library/nginx/blobs/sha256:03e6a245275128e26fc650e724e3fc4510d81f8111bae35ece70242b0a638215?ns=docker.io" http.request.useragent="containerd/1.4.6+bottlerocket" http.response.contenttype="application/octet-stream" http.response.duration=580.83924ms http.response.status=200 http.response.written=895 
18.236.180.8 - - [23/Jun/2021:19:36:38 +0000] "GET /v2/library/nginx/blobs/sha256:03e6a245275128e26fc650e724e3fc4510d81f8111bae35ece70242b0a638215?ns=docker.io HTTP/1.1" 200 895 "" "containerd/1.4.6+bottlerocket"
time="2021-06-23T19:36:39.281922202Z" level=info msg="Adding new scheduler entry for library/nginx@sha256:03e6a245275128e26fc650e724e3fc4510d81f8111bae35ece70242b0a638215 with ttl=167h59m59.999983565s" go.version=go1.11.2 instance.id=1470de18-b2bd-4f24-b811-4f48dd848efb service=registry version=v2.7.1 
time="2021-06-23T19:36:39.30334434Z" level=info msg="response completed" go.version=go1.11.2 http.request.host=44.234.86.205 http.request.id=0a70ac32-9cac-40b9-9810-12ba77fe6607 http.request.method=GET http.request.remoteaddr="18.236.180.8:56809" http.request.uri="/v2/library/nginx/blobs/sha256:b21fed559b9f420d83f8e38ca08d1ac4f15298a3ae02c6de56f364bee2299f78?ns=docker.io" http.request.useragent="containerd/1.4.6+bottlerocket" http.response.contenttype="application/octet-stream" http.response.duration=660.705942ms http.response.status=200 http.response.written=602 
18.236.180.8 - - [23/Jun/2021:19:36:38 +0000] "GET /v2/library/nginx/blobs/sha256:b21fed559b9f420d83f8e38ca08d1ac4f15298a3ae02c6de56f364bee2299f78?ns=docker.io HTTP/1.1" 200 602 "" "containerd/1.4.6+bottlerocket"
time="2021-06-23T19:36:39.311026941Z" level=info msg="response completed" go.version=go1.11.2 http.request.host=44.234.86.205 http.request.id=b98667dd-0051-4aef-a026-b7b18b610932 http.request.method=GET http.request.remoteaddr="18.236.180.8:38323" http.request.uri="/v2/library/nginx/blobs/sha256:b82f7f888feb03d38fed4dad68d7265a8b276f1f0c543d549fc6ef30b42c00eb?ns=docker.io" http.request.useragent="containerd/1.4.6+bottlerocket" http.response.contenttype="application/octet-stream" http.response.duration=660.928532ms http.response.status=200 http.response.written=667 
18.236.180.8 - - [23/Jun/2021:19:36:38 +0000] "GET /v2/library/nginx/blobs/sha256:b82f7f888feb03d38fed4dad68d7265a8b276f1f0c543d549fc6ef30b42c00eb?ns=docker.io HTTP/1.1" 200 667 "" "containerd/1.4.6+bottlerocket"
time="2021-06-23T19:36:39.316264076Z" level=info msg="Adding new scheduler entry for library/nginx@sha256:b21fed559b9f420d83f8e38ca08d1ac4f15298a3ae02c6de56f364bee2299f78 with ttl=167h59m59.999995972s" go.version=go1.11.2 instance.id=1470de18-b2bd-4f24-b811-4f48dd848efb service=registry version=v2.7.1 
time="2021-06-23T19:36:39.395778065Z" level=info msg="Adding new scheduler entry for library/nginx@sha256:b82f7f888feb03d38fed4dad68d7265a8b276f1f0c543d549fc6ef30b42c00eb with ttl=167h59m59.999997262s" go.version=go1.11.2 instance.id=1470de18-b2bd-4f24-b811-4f48dd848efb service=registry version=v2.7.1 
time="2021-06-23T19:36:39.717275884Z" level=info msg="response completed" go.version=go1.11.2 http.request.host=44.234.86.205 http.request.id=887d2a5f-905d-41ec-9e89-44153fe491d9 http.request.method=GET http.request.remoteaddr="18.236.180.8:25135" http.request.uri="/v2/library/nginx/blobs/sha256:edb81c9bc1f5416a41e5bea21748dc912772fedbd4bd90e5e3ebfe16b453edce?ns=docker.io" http.request.useragent="containerd/1.4.6+bottlerocket" http.response.contenttype="application/octet-stream" http.response.duration=1.101527344s http.response.status=200 http.response.written=26580118 
18.236.180.8 - - [23/Jun/2021:19:36:38 +0000] "GET /v2/library/nginx/blobs/sha256:edb81c9bc1f5416a41e5bea21748dc912772fedbd4bd90e5e3ebfe16b453edce?ns=docker.io HTTP/1.1" 200 26580118 "" "containerd/1.4.6+bottlerocket"
time="2021-06-23T19:36:39.891635154Z" level=info msg="response completed" go.version=go1.11.2 http.request.host=44.234.86.205 http.request.id=0c9a9d08-e289-4d07-ac2d-6c2df642f4d2 http.request.method=GET http.request.remoteaddr="18.236.180.8:24956" http.request.uri="/v2/library/nginx/blobs/sha256:b4d181a07f8025e00e0cb28f1cc14613da2ce26450b80c54aea537fa93cf3bda?ns=docker.io" http.request.useragent="containerd/1.4.6+bottlerocket" http.response.contenttype="application/octet-stream" http.response.duration=1.26689776s http.response.status=200 http.response.written=27145851 
18.236.180.8 - - [23/Jun/2021:19:36:38 +0000] "GET /v2/library/nginx/blobs/sha256:b4d181a07f8025e00e0cb28f1cc14613da2ce26450b80c54aea537fa93cf3bda?ns=docker.io HTTP/1.1" 200 27145851 "" "containerd/1.4.6+bottlerocket"

For host-containers and bootstrap-containers:
Started host-container that just runs nginx:

bash-5.0# apiclient set host-containers.nginx.source="docker.io/library/nginx:latest"
bash-5.0# apiclient set host-containers.nginx.enabled=true

As soon as the host-container started, I observed traffic through the pull-through cache from my instance:

time="2021-07-27T16:24:08.039378021Z" level=info msg="response completed" go.version=go1.11.2 http.request.host=34.222.201.232 http.request.id=e0293cee-59bd-4e5e-987c-f1c9321591b3 http.request.method=HEAD http.request.remoteaddr="34.222.177.43:49756" http.request.uri="/v2/library/nginx/manifests/latest?ns=docker.io" http.request.useragent="containerd/1.4.3+unknown" http.response.contenttype="application/vnd.docker.distribution.manifest.list.v2+json" http.response.duration=729.030479ms http.response.status=200 http.response.written=1862 
34.222.177.43 - - [27/Jul/2021:16:24:07 +0000] "HEAD /v2/library/nginx/manifests/latest?ns=docker.io HTTP/1.1" 200 1862 "" "containerd/1.4.3+unknown"
34.222.177.43 - - [27/Jul/2021:16:24:55 +0000] "HEAD /v2/library/nginx/manifests/latest?ns=docker.io HTTP/1.1" 200 1862 "" "containerd/1.4.3+unknown"
...

For bootstrap-containers, I defined a bootstrap container that just runs nginx along with the image registry config in my userdata.
Launched instance and bootstrap container started as expected and I observed corresponding traffic through the pull through cache.

Tested migrations and they work as expected

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

sources/models/shared-defaults/docker-services.toml

README.md

packages/docker-engine/docker-engine.spec

sources/models/shared-defaults/kubernetes-containerd.toml

sources/models/src/aws-ecs-1/mod.rs

packages/containerd/containerd-config-toml_k8s

etungsten · 2021-06-23T21:33:33Z

Taking into draft since I forgot to write a new migration for the setting.

sources/api/migration/migrations/v1.1.3/image-registry-mirrors/src/main.rs

README.md

etungsten · 2021-06-23T23:04:02Z

Push below and above addresses @tjkirch's comments.

tjkirch · 2021-06-23T23:12:23Z

LGTM assuming migrations test OK.

samuelkarp · 2021-06-23T23:41:44Z

README.md

@@ -407,6 +407,19 @@ These settings can be changed at any time.
  Supported values are `debug`, `info`, `warn`, `error`, and `crit`, and the default is `info`.
 * `settings.ecs.enable-spot-instance-draining`: If the instance receives a spot termination notice, the agent will set the instance's state to `DRAINING`, so the workload can be moved gracefully before the instance is removed. Defaults to `false`.

+#### Image registry settings


We talk about OS images in this readme too, so it might be worthwhile to call out that this is specific to container images.

Suggested change

#### Image registry settings

#### Container image registry settings

samuelkarp · 2021-06-23T23:42:00Z

README.md

@@ -407,6 +407,19 @@ These settings can be changed at any time.
  Supported values are `debug`, `info`, `warn`, `error`, and `crit`, and the default is `info`.
 * `settings.ecs.enable-spot-instance-draining`: If the instance receives a spot termination notice, the agent will set the instance's state to `DRAINING`, so the workload can be moved gracefully before the instance is removed. Defaults to `false`.

+#### Image registry settings
+
+The following setting is optional and allows you to configure image registry mirrors and pull-through caches for your orchestrated containers.


Does this only cover orchestrated containers and not host containers? Should it cover both?

Right, this currently only covers orchestrated containers (containers via cri-containerd).

For host-containers it's not immediately clear to me how I can configure the containerd client to use registry mirrors. I suspect I'd have to modify resolver to accomplish that? I can create a separate issue to track registry mirrors for host-containers.

Yes, this would be something that would have to be done in host-ctr. I'm not sure if we should have the setting called settings.container-registry.mirrors if it only controls orchestrated containers though; how would we change this to handle host containers in the future?

It seems like it's just a bug that host-ctr and host containers wouldn't support the mirror settings - same as if a component doesn't support the system-wide proxy settings.

README.md

etungsten · 2021-07-14T18:28:36Z

Push above rebases onto develop and fixes conflicts

etungsten · 2021-07-14T20:12:40Z

Push above addresses @bcressey 's comment.

"image-registry" -> "container-registry"

tested things and they still work.

etungsten · 2021-07-19T21:31:12Z

Missed some dangling references to image_registries in models. Push above fixes that.

etungsten · 2021-07-22T22:57:30Z

Rebases onto develop and fixes conflicts with migrations.

packages/containerd/containerd-config-toml_k8s

packages/docker-engine/docker-engine.spec

etungsten · 2021-07-28T02:21:17Z

Force push addresses @samuelkarp 's comments.

Changes the host-ctr-regsitry-config format from TOML to JSON
Adds unit tests for registryHosts
Adds checks for endpoints without an URL scheme and prefixes them with a default scheme
Adds comments explaining images from private ECR repositories cannot use registry configuration.

Force push fixes whitespacing issues with the JSON config template and other small typos.

etungsten · 2021-07-28T04:13:04Z

Push above reverts the change to the configuration file format from TOML to JSON.
The handlebars rust crate does not have feature parity with handlebarsjs. It currently does not support @last, @first, @index in #if or #unless helpers. Which makes it impossible to conditionally render a , for items in a list in JSON, so for multiple items in a list, the resulting JSON would be always invalid.

For example, for the following template:

{
    {{~#if settings.container-registry.mirrors}}
    "mirrors": {
        {{~#each settings.container-registry.mirrors}}
        "{{@key}}": {
            "endpoints": [{{join_array ", " this }}]
        }{{~#unless @last}}, {{/unless}}
        {{~/each}}
    }
    {{~/if}}
}

The , always gets rendered even though it shouldn't have.

{
    "mirrors": {
        "docker.io": {
            "endpoints": ["http://my-pull-through-cache-host-ip"]
        },
    }
}

I tried a bunch of variations of setting the condition, but it just comes down to #if/#unless not working with @index/@last/@first

We technically can write a new helper that conditionally renders the ,. But for simplicity's sake, I think we should just stick with TOML. The only downside is that we're taking on an additional go-toml dependency.

Tested changes and everything still work as described in the PR testing description.

webern · 2021-07-28T16:11:36Z

We technically can write a new helper that conditionally renders the ,.

I already wrote one I though.

Release.toml

samuelkarp

Looks good. A couple nits, but nothing that need to block this.

sources/host-ctr/cmd/host-ctr/main_test.go

sources/host-ctr/cmd/host-ctr/main.go

etungsten · 2021-07-30T00:54:43Z

Push above addresses @arnaldo2792 and @samuelkarp 's comments. Will rebase shortly to fix conflicts

etungsten · 2021-07-30T00:59:22Z

Push above rebases onto develop and fixes conflicts.

bcressey · 2021-07-30T03:25:28Z

Release.toml

+"(1.1.4, 1.2.0)" = [
+    "migrate_v1.2.0_container-registry-mirrors.lz4",
+    "migrate_v1.2.0_container-registry-config-restarts.lz4",
+]


Are we planning to fix up migrations in a different commit? If not then these migrations should be with the hostname ones.

bcressey · 2021-07-30T03:28:45Z

packages/os/host-ctr-registry-config

@@ -0,0 +1,6 @@
+{{~#if settings.container-registry.mirrors}}


nit: it might be better to call this file host-ctr.toml to make room for other potential settings - but I'm also happy to defer that until we need more settings.

sources/host-ctr/cmd/host-ctr/main.go

sources/host-ctr/cmd/host-ctr/main_test.go

Adds a new setting `settings.container-registry.mirrors` that lets users set pull-through caches for container image registries.

etungsten · 2021-08-02T17:20:35Z

Push above just rebases onto develop and fixes conflicts.

Adds support for configuring container image registrys for host-containers and bootstrap containers. A lot of the logic is borrowed from containerd-cri plugin's implementation of image registry mirrors.

etungsten · 2021-08-02T17:29:01Z

Push above addresses comments from @samuelkarp and @bcressey

I changed the name of the host-ctr config file from host-ctr-registry-config to host-ctr-toml to make it more generic.

Adds migrations for the new `container-registry` setting and the new docker daemon config file and the new config file for host-containers and bootstrap containers.

etungsten · 2021-08-02T17:34:41Z

Fixed up Release.toml migrations grouping.

etungsten requested review from samuelkarp and bcressey June 23, 2021 19:48

tjkirch reviewed Jun 23, 2021

View reviewed changes

etungsten marked this pull request as draft June 23, 2021 21:29

etungsten force-pushed the image-registry-cache branch 2 times, most recently from 41fc9ed to 676865a Compare June 23, 2021 22:50

tjkirch reviewed Jun 23, 2021

View reviewed changes

sources/api/migration/migrations/v1.1.3/image-registry-mirrors/src/main.rs Outdated Show resolved Hide resolved

tjkirch reviewed Jun 23, 2021

View reviewed changes

README.md Outdated Show resolved Hide resolved

etungsten force-pushed the image-registry-cache branch from 676865a to 7c5460c Compare June 23, 2021 23:02

etungsten force-pushed the image-registry-cache branch from 7c5460c to 87b6393 Compare June 23, 2021 23:04

etungsten marked this pull request as ready for review June 23, 2021 23:19

samuelkarp reviewed Jun 23, 2021

View reviewed changes

webern approved these changes Jul 2, 2021

View reviewed changes

etungsten force-pushed the image-registry-cache branch from 87b6393 to 663d141 Compare July 2, 2021 21:07

etungsten requested review from tjkirch and samuelkarp July 2, 2021 21:10

tjkirch approved these changes Jul 2, 2021

View reviewed changes

bcressey reviewed Jul 14, 2021

View reviewed changes

README.md Outdated Show resolved Hide resolved

etungsten force-pushed the image-registry-cache branch from 663d141 to 82ac125 Compare July 14, 2021 18:28

etungsten force-pushed the image-registry-cache branch from 82ac125 to 21a1abe Compare July 14, 2021 20:09

etungsten force-pushed the image-registry-cache branch from 21a1abe to 18c21bd Compare July 19, 2021 21:30

etungsten force-pushed the image-registry-cache branch from 18c21bd to e10e82e Compare July 22, 2021 22:57

bcressey approved these changes Jul 25, 2021

View reviewed changes

packages/containerd/containerd-config-toml_k8s Outdated Show resolved Hide resolved

packages/docker-engine/docker-engine.spec Outdated Show resolved Hide resolved

etungsten force-pushed the image-registry-cache branch 2 times, most recently from a185a00 to 7d949ee Compare July 28, 2021 04:12

etungsten requested a review from samuelkarp July 28, 2021 04:13

etungsten force-pushed the image-registry-cache branch from 7d949ee to a05aabb Compare July 28, 2021 16:03

arnaldo2792 reviewed Jul 28, 2021

View reviewed changes

Release.toml Outdated Show resolved Hide resolved

samuelkarp approved these changes Jul 29, 2021

View reviewed changes

sources/host-ctr/cmd/host-ctr/main_test.go Show resolved Hide resolved

sources/host-ctr/cmd/host-ctr/main.go Outdated Show resolved Hide resolved

sources/host-ctr/cmd/host-ctr/main.go Outdated Show resolved Hide resolved

etungsten force-pushed the image-registry-cache branch from a05aabb to 2243f84 Compare July 30, 2021 00:54

etungsten force-pushed the image-registry-cache branch from 2243f84 to ea32d34 Compare July 30, 2021 00:59

etungsten requested a review from arnaldo2792 July 30, 2021 00:59

bcressey approved these changes Jul 30, 2021

View reviewed changes

samuelkarp reviewed Jul 30, 2021

View reviewed changes

docker, containerd: container image registry mirrors

56062ea

Adds a new setting `settings.container-registry.mirrors` that lets users set pull-through caches for container image registries.

etungsten force-pushed the image-registry-cache branch from ea32d34 to 6c72595 Compare August 2, 2021 17:20

host-ctr: add support for container image registries

98a3796

Adds support for configuring container image registrys for host-containers and bootstrap containers. A lot of the logic is borrowed from containerd-cri plugin's implementation of image registry mirrors.

etungsten force-pushed the image-registry-cache branch from 6c72595 to 3d45ba3 Compare August 2, 2021 17:28

etungsten requested a review from samuelkarp August 2, 2021 17:29

migrations: migrate settings.container-registry, and related models

62e69aa

Adds migrations for the new `container-registry` setting and the new docker daemon config file and the new config file for host-containers and bootstrap containers.

etungsten force-pushed the image-registry-cache branch from 3d45ba3 to 62e69aa Compare August 2, 2021 17:34

samuelkarp approved these changes Aug 2, 2021

View reviewed changes

etungsten merged commit 1984358 into bottlerocket-os:develop Aug 2, 2021

etungsten deleted the image-registry-cache branch August 2, 2021 18:11

tjkirch mentioned this pull request Aug 3, 2021

Update default variant to aws-k8s-1.21 #1686

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docker, containerd: image registry mirrors #1629

docker, containerd: image registry mirrors #1629

etungsten commented Jun 23, 2021 •

edited

etungsten commented Jun 23, 2021

etungsten commented Jun 23, 2021

tjkirch commented Jun 23, 2021

samuelkarp Jun 23, 2021

samuelkarp Jun 23, 2021

etungsten Jun 24, 2021

samuelkarp Jul 23, 2021

bcressey Jul 25, 2021

etungsten commented Jul 14, 2021

etungsten commented Jul 14, 2021

etungsten commented Jul 19, 2021

etungsten commented Jul 22, 2021

etungsten commented Jul 28, 2021

etungsten commented Jul 28, 2021

webern commented Jul 28, 2021

samuelkarp left a comment

etungsten commented Jul 30, 2021

etungsten commented Jul 30, 2021

bcressey Jul 30, 2021

bcressey Jul 30, 2021

etungsten commented Aug 2, 2021

etungsten commented Aug 2, 2021

etungsten commented Aug 2, 2021

	#### Image registry settings
	#### Container image registry settings

docker, containerd: image registry mirrors #1629

docker, containerd: image registry mirrors #1629

Conversation

etungsten commented Jun 23, 2021 • edited

etungsten commented Jun 23, 2021

etungsten commented Jun 23, 2021

tjkirch commented Jun 23, 2021

samuelkarp Jun 23, 2021

Choose a reason for hiding this comment

samuelkarp Jun 23, 2021

Choose a reason for hiding this comment

etungsten Jun 24, 2021

Choose a reason for hiding this comment

samuelkarp Jul 23, 2021

Choose a reason for hiding this comment

bcressey Jul 25, 2021

Choose a reason for hiding this comment

etungsten commented Jul 14, 2021

etungsten commented Jul 14, 2021

etungsten commented Jul 19, 2021

etungsten commented Jul 22, 2021

etungsten commented Jul 28, 2021

etungsten commented Jul 28, 2021

webern commented Jul 28, 2021

samuelkarp left a comment

Choose a reason for hiding this comment

etungsten commented Jul 30, 2021

etungsten commented Jul 30, 2021

bcressey Jul 30, 2021

Choose a reason for hiding this comment

bcressey Jul 30, 2021

Choose a reason for hiding this comment

etungsten commented Aug 2, 2021

etungsten commented Aug 2, 2021

etungsten commented Aug 2, 2021

etungsten commented Jun 23, 2021 •

edited