Caching foreign layers

[dimuskin]

Hi,

We have small team that develops .NET applications. Often have to install windows docker containers.
Because the size of microsoft/windowsservercore image more than 8GB we decided to install local cache registry to save time to download.
Unfortunately the registry does not cache foreign layers or layers which contains urls parameter in manifest file. (btw, its no problem with any other images)

Example manifest (microsoft/nanoserver/10.0.14393.953):

"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 582,
"digest": "sha256:18a0d32a4b98e8a3e1ab7eb33b2be75b4826cbf43754961db08101b52bce0840"
},
"layers": [{
"mediaType": "application/vnd.docker.image.rootfs.foreign.diff.tar.gzip",
"size": 252691002,
"digest": "sha256:bce2fbc256ea437a87dadac2f69aabd25bed4f56255549090056c1131fad0277",
"urls": [
"https://go.microsoft.com/fwlink/?linkid=837858"
]
},
{
"mediaType": "application/vnd.docker.image.rootfs.foreign.diff.tar.gzip",
"size": 114925341,
"digest": "sha256:58f68fa0ceda734a980c12dedf782342f892e218bba3c74ded58bfabed652ba1",
"urls": [
"https://go.microsoft.com/fwlink/?linkid=844835"
]
}
]
}

Is this expected behavior and if so, are there any plans to support caching these items for windows? Is any other way to create cache for this type on images?

Best Regards.

[thaJeztah]

I wonder if this would help with that; #33151

Also; ping @friism

[dimuskin]

@thaJeztah Thank you for response, there the problem is discussed, but as I understood there is no working solution

[cpuguy83]

@dimuskin The feature @thaJeztah mentioned is going to be what you want.
Basically you tell docker "allow foreign layers to be pushed to registry "

This will be in the 17.06 release.

[sandersaares]

Can you explain the intended workflow in depth? It is not entirely obvious to me how this feature relates to the standard "registry-mirrors" cache system (if at all) nor what is required to make it all cooperate nice and proper. What is the sequence of actions needed to get from a clean Docker and custom Docker registry install to one where microsoft/windowsservercore is pulled from the custom registry?

[friism]

@sandersaares There's documentation coming here: https://github.com/docker/docker.github.io/pull/3485/files#diff-460d63bcfd2c1c08c13d056dfcc10b6aR521

I don't think registry-mirrors will work for this (I don't think so), but @nwt might remember.

[sandersaares]

Here are the questions that come to mind.

• Does Docker that is configured without the "push foreign" setting still attempt to download foreign layers from cache? (I think "yes" but want to confirm)
• If yes, is such a Docker install that downloads Microsoft layers from cache (without "push foreign" setting) able to see new/updated foreign-layer images that are not cached? (I have no idea; hopefully "yes")
• Is a Docker install that uploads foreign layers to a registry able to see new/updated foreign-layer images that are not in registry? (I have no idea; hopefully "yes")

I will experiment to discover the answers. My ideal scenario is one where there is one Docker install pre-seeding a cache registry with foreign layers, with other Docker installs able to pull from the cache without any special configuration.

[dimuskin]

Technically, I found a way to manually upload foreign images directly into the cache (aka. local registry mirror).
The method is very dummy, but it works for me.
(for example how insert into cache microsoft/windowsservercore:10.0.14393.1770 layrers)

• at first find the foreign images url in the manifest file:

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 774,
      "digest": "sha256:ce27208ad6786bb8df2cae183e51b636c91861c049d4797d27c38bb40921a87c"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.foreign.diff.tar.gzip",
         "size": 4069985900,
         "digest": "sha256:3889bb8d808bbae6fa5a33e07093e65c31371bcf9e4c38c21be6b9af52ad1548",
         "urls": [
            "https://go.microsoft.com/fwlink/?linkid=837859"
         ]
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.foreign.diff.tar.gzip",
         "size": 1280521235,
         "digest": "sha256:8df8e568af76c1c311a1251f6f7402e2a09d14629840c97091beb9ba55419464",
         "urls": [
            "https://go.microsoft.com/fwlink/?linkid=860053"
         ]
      }
   ]
}

• download and save it with correct name:

cd /tmp wget -O 3889bb8d808bbae6fa5a33e07093e65c31371bcf9e4c38c21be6b9af52ad1548 https://go.microsoft.com/fwlink/?linkid=837859

wget -O 8df8e568af76c1c311a1251f6f7402e2a09d14629840c97091beb9ba55419464 https://go.microsoft.com/fwlink/?linkid=860053

• move files your registry blobs directory

cd /var/lib/registr/docker/registry/v2/blobs/sha256 (path INSIDE container)
mkdir -p 38 8d (first 2 chars from filenames)
mv /tmp/3889bb8d808bbae6fa5a33e07093e65c31371bcf9e4c38c21be6b9af52ad1548 38/
mv /tmp/8df8e568af76c1c311a1251f6f7402e2a09d14629840c97091beb9ba55419464 8d/

• create link files in _layers directory

cd /var/lib/registr/docker/registry/v2/repositories/microsoft/windowsservercore/_layers/sha256

mkdir 3889bb8d808bbae6fa5a33e07093e65c31371bcf9e4c38c21be6b9af52ad1548

mkdir 8df8e568af76c1c311a1251f6f7402e2a09d14629840c97091beb9ba55419464

printf "sha256:3889bb8d808bbae6fa5a33e07093e65c31371bcf9e4c38c21be6b9af52ad1548" > 3889bb8d808bbae6fa5a33e07093e65c31371bcf9e4c38c21be6b9af52ad1548/link

printf "sha256:8df8e568af76c1c311a1251f6f7402e2a09d14629840c97091beb9ba55419464" > 8df8e568af76c1c311a1251f6f7402e2a09d14629840c97091beb9ba55419464/link

Now if you pull microsoft/windowsservercore:10.0.14393.1770, docker will download it from your local mirror repository.

[kassanmoor]

Could we reopen this? although @dimuskin solution works, is not very usable cc @thaJeztah @friism

[dimuskin]

@kassanmoor technically you can use simple python script like that:

#!/usr/bin/python
import os
import json
import urllib2
import requests
import wget

reponame = "microsoft/windowsservercore"

tags = ["latest","1709"]

# set your actual path
reg_path = "/docker-cache/data"
blob_path = reg_path + '/docker/registry/v2/blobs/sha256'
link_path = reg_path + '/docker/registry/v2/repositories/' + reponame + '/_layers/sha256'

def getToken():
        response = urllib2.urlopen('https://auth.docker.io/token?service=registry.docker.io&scope=repository:' + reponame + ':pull,push')
        html = response.read()
        token = json.loads(html)['token']
        return token

def getLayers(token):
        headers = {'Authorization' : 'Bearer ' + token, 'Accept' : 'application/vnd.docker.distribution.manifest.v2+json'}
        req = urllib2.Request('https://index.docker.io/v2/' + reponame + '/manifests/' + tag, None, headers = headers)
        response = urllib2.urlopen(req)
        resp = response.read()
        layers = json.loads(resp)['layers']
        return layers

def processLinkFile(dir, digest):
        file = dir + '/link'
        if os.path.exists(file) == True:
                print "Local link file exist..."
                with open (file, 'r') as file:
                        data = file.read()
                        if data == digest:
                                print "Local link file content OK !"
                        else:
                                print "FATAL ERROR, LINK FILE EXISTS WITH WRONG CONTENT !!!"
        else: # creating link file
                print "Creating link file directory .."
                os.makedirs(dir)
                print "Creating link file .."
                f = open(file, 'w+')
                f.write(digest)
                f.close
                print "Link file created !"

def processBlobFile(dir, hash, url, size):
        file = dir + '/data'

        if os.path.exists(file) == True:
                print "Local blob file exist .."
                if os.path.getsize(file) == size:
                        print "Local blob file size is OK !"
                else:
                        print "FATAL ERROR, BLOB FILE EXIST WITH WRONG SIZE !!!"
        else: # lets download file
                if not os.path.exists(dir):
                        print "Creating blob directory .." + dir
                        os.makedirs(dir)
                print "Starting to download blob file .."
                wget.download(url, file, bar=None)
                print "Downloaded url:" + url

token = getToken()
for tag in tags:
        print "Processing image:" + reponame + ':' + tag
        layers = getLayers(token)
        for layer in layers:
                url = layer['urls'][0] # I take first url
                size =  layer['size']
                digest =  layer['digest']
                hash = digest[7:]
                print "Processing layer with digest: " + digest
                blob_file_dir  = blob_path + '/' + hash[0:2] + '/' + hash
                link_file_dir  = link_path + '/' + hash

                processLinkFile(link_file_dir, digest)
                processBlobFile(blob_file_dir, hash, url, size)