mac docker-compose build fails to pull from private registry (works on linux)

[hiphamster]

I have a compose file with declared services, one of which has 'FROM xxx' in its Dockerfile, and the 'xxx' is an image located in my private registry in gitlab.

Expectation:
Whey i run docker-composer -f compose-file build, i expect the image from private registry to be pulled and used.

Actual:
On linux (ubuntu 18.04 under virtualbox) it works as expected - the image is pulled
On mac i get an error:
Building panda_app Step 1/26 : FROM registry.gitlab.com/xxx/xxx/xxx:3.7.4 ERROR: Service 'xxx_app' failed to build: Get https://registry.gitlab.com/v2/xxx/xxx/xxx/manifests/3.7.4: denied: access forbidden

I tried logging out / logging it, that didn't help. Below is additional info i got from running with --verbose flag:

  1 compose.cli.verbose_proxy.proxy_callable: docker close <- ()
  2 compose.cli.verbose_proxy.proxy_callable: docker close -> None
  3 compose.service.build: Building xxxxx_app
  4 compose.cli.verbose_proxy.proxy_callable: docker build <- (
  5  path='/Users/axxxxx/dev/xxxxx/build/xxxxx_app',
  6  tag='registry.gitlab.com/xxxxx/xxxxx/xxxxx_app:dev',
  7  rm=True,
  8  forcerm=False,
  9  pull=False,
 10  nocache=False,
 11  dockerfile=None,
 12  cache_from=None,
 13  labels=None,
 14  buildargs={'xxxxx',
 15  'sentry_dsn': 'xxxxxxx',
 16  'sqlalchemy_uri': 'mysql+pymysql://xxxxx'},
 17  network_mode=None,
 18  target=None,
 19  shmsize=None,
 20  extra_hosts=None,
 21  container_limits={'memory': None},
 22  gzip=False,
 23  isolation=None,
 24  platform=None)
 25 docker.api.build._set_auth_headers: Looking for auth config
 26 docker.auth._resolve_authconfig_credstore: Looking for auth entry for 'https://registry.gitlab.com'
 27 docker.api.build._set_auth_headers: Sending auth config ('https://registry.gitlab.com', 'registry.gitlab.com')
 28 urllib3.connectionpool._make_request:
 29   http://localhost:None
 30   "POST /v1.38/build?t=registry.gitlab.com%2Fxxxxx%2Fxxxxx%2Fxxxxx_app%3Adev&
 31   q=False&
 32   nocache=False&
 33   rm=True&
 34   forcerm=False&
 35   pull=False&
 36   buildargs=xxxxxxxxxx HTTP/1.1" 200 None
 37 compose.cli.verbose_proxy.proxy_callable: docker build -> <generator object APIClient._stream_helper at 0x10c36be08>
 38 Step 1/26 : FROM registry.gitlab.com/xxxxx/xxxxx/xxxxx_py:3.7.4
 39 compose.cli.main.main: Service 'xxxxx_app' failed to build:
 40 Get https://registry.gitlab.com/v2/xxxxx/xxxxx/xxxxx_py/manifests/3.7.4: denied: access forbidden

[charles-ferguson]

I seem to be having a similar issue as well. However, the error message is different as docker-compose doesn't seem to be sending the auth headers at all when trying to fetch the latest manifest.

docker.api.build._set_auth_headers: Looking for auth config
docker.auth._resolve_authconfig_credstore: Looking for auth entry for 'https://clio-peng-docker-dev-local.jfrog.io'
docker.auth._resolve_authconfig_credstore: Looking for auth entry for 'https://clio-product-docker-dev-local.jfrog.io'
docker.auth._resolve_authconfig_credstore: Looking for auth entry for 'https://index.docker.io/v1/'
docker.api.build._set_auth_headers: Sending auth config ('https://clio-product-docker-dev-local.jfrog.io', 'clio-product-docker-dev-local.jfrog.io', 'clio-peng-docker-dev-local.jfrog.io', 'https://clio-peng-docker-dev-local.jfrog.io', 'https://index.docker.io/v1/')
urllib3.connectionpool._make_request: http://localhost:None "POST
/v1.38/build?t=grow&q=False&nocache=False&rm=True&forcerm=False&pull=True&dockerfile=.%2Fdocker%2Fdevelopment%2FDockerfile&buildargs=%7B%22ARTIFACTORY_API_KEY%22<REDACTED>+%22ARTIFACTORY_USERNAME%22%3A+%22<REDACTED>%22%7D&cachefrom=%5B%22clio-product-docker-dev-local.jfrog.io%2Fgrow%3Alatest%22%2C+%22grow%3Alatest%22%5D HTTP/1.1" 200 None
compose.cli.verbose_proxy.proxy_callable: docker build -> <generator object APIClient._stream_helper at 0x111594048>
Step 1/16 : FROM clio-peng-docker-dev-local.jfrog.io/ruby-2.4.3:latest
ERROR: compose.cli.main.main: Service 'app' failed to build: Get https://clio-peng-docker-dev-local.jfrog.io/v2/ruby-2.4.3/manifests/latest: unknown: Authentication is required

pulling the image using docker and then building without the --pull flag is workaround that is working for me for now.

[fluetm]

We're experiencing this also. I can correct the situation by removing the auths entries from .docker/config.json after logging in to docker and then doing the docker-compose build.
So this config.json will fail:

	"auths": {
		"some.registry.name": {},
	},
	"HttpHeaders": {
		"User-Agent": "Docker-Client/19.03.1 (darwin)"
	},
	"credsStore": "osxkeychain"

But this one will succeed:

	"auths": {},
	"HttpHeaders": {
		"User-Agent": "Docker-Client/19.03.1 (darwin)"
	},
	"credsStore": "osxkeychain"

However, logging out and logging back in replaces the auth entries and then we're broken again.

This started breaking for us in docker-compose 1.24.0-rc1. There's no real obvious change that jumps out in the commit list for that release, but I'm suspecting it may have come in from the docker-py upgrade here: ab0a0d6

There was a decent refactoring around the auth stuff in that release: docker/docker-py@bc5d7c8#diff-6b81f0b56fd2b004af477b65a6990177

[fluetm]

A couple of other workarounds that I've confirmed will avoid this issue, but they require encoded credentials to be stored on disk...the problems seems to be tied specifically to the usage of the keychain:

1. Add the encoded auth credentials directly to config.json without a credsStore:

{
	"auths": {
		"some.registry.name": {
			"auth": "<base64 encoded credentials>"
		}
	},
	"HttpHeaders": {
		"User-Agent": "Docker-Client/19.03.1 (darwin)"
	}
}

2. In Docker Desktop preferences, uncheck Securely store Docker logins in macOS keychain and then use the desktop credsStore in config.json instead of the osxkeychain (this will end up writing the encoded credentials to .docker/plaintext-passwords.json:

{
	"auths": {
		"some.registry.name": {},
	},
	"HttpHeaders": {
		"User-Agent": "Docker-Client/19.03.1 (darwin)"
	},
	"credsStore": "desktop"
}

[fluetm]

The root cause in docker-py: docker/docker-py#2402

[chrisjohnson]

We are struggling with this at the moment and trying to figure out if it's worth us working around the issue or waiting for the fix. Can anybody give me a rough timeframe for how long it would take for this fix to bubble its way all the way up to the homebrew release of docker-compose?

[so-jelly]

@chrisjohnson is stripping the auths section not working for you? here's a oneliner i've been using (sharing with my teams) TMPVAR=$(jq 'del(.auths[])' ~/.docker/config.json) && echo $TMPVAR >| ~/.docker/config.json

[chrisjohnson]

That is one workaround, another is just disabling the keychain. The workarounds do work, but we have large teams, so if the workarounds are only going to be in place for a few weeks, it's less work to just wait for the proper fix. If it's more like months, we'll try to automate the workarounds.

[isaiahgrant]

Also experiencing this issue.

Dockerfile:

FROM <insert_private_registry>/test-image:latest

CLI

Both commands fail without a login to the private registry (expected)

    $ docker-compose up
    Building app
    Step 1/2 : FROM <insert_private_registry>/test-image:latest
    ERROR: Service 'app' failed to build: Get https://<insert_private_registry>/v2/test-image/manifests/latest: denied: access forbidden

    $ docker pull <insert_private_registry>/test-image:latest
    Error response from daemon: Get https://<insert_private_registry>/test-image/manifests/latest: denied: access forbidden

After logging in, a docker pull ... works while the docker-compose up fails to pull the image:

    $ docker login <insert_private_registry>
    Username: <insert>
    Password: <insert>
    Login Succeeded

    $ docker-compose up
    Building app
    Step 1/2 : FROM <insert_private_registry>/test-image:latest
    ERROR: Service 'app' failed to build: Get https://<insert_private_registry>/v2/test-image/manifests/latest: denied: access forbidden

    $ docker pull <insert_private_registry>/test-image:latest
    latest: Pulling from <insert_private_image_path>/test-image
    ...
    Status: Downloaded newer image for <insert_private_registry>/test-image:latest

Current Solution

Our current workaround is to explicitly pull the image prior to running the docker-compose containers:

    docker pull <insert_private_registry>/test-image:latest
    latest: Pulling from <insert_private_image_path>/test-image
    ...
    Status: Downloaded newer image for <insert_private_registry>/test-image:latest

    $ docker-compose up
    Building app
    Step 1/2 : FROM <insert_private_registry>/test-image:latest
    ...

[ndeloof]

Closing this issue as a duplicate of #6713