New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pushing to an insecure repository is problematic on OSX High Sierra #2392

Open
TeaSeaLancs opened this Issue Jan 10, 2018 · 28 comments

Comments

Projects
None yet
@TeaSeaLancs

TeaSeaLancs commented Jan 10, 2018

Expected behaviour

When I push an image to our private insecure repository on our network, the image should push without issues.

Actual behaviour

Each layer being pushed keeps on retrying. After pausing for 20 seconds for a retry, it errors out with a HTTP 503. The docker repo logs indicate that everything is fine.

This is only affecting our internal registry, uploading to docker cloud works fine. It's also not reproducible for all users of our internal registry. This issue started just after upgrading to High Sierra.

Information

Diagnostic ID: A5B5DC82-4183-4367-BA3B-9B6A66B2A259

Steps to reproduce the behavior

  1. Try to push a container to an insecure registry
@inkel

This comment has been minimized.

inkel commented Jan 10, 2018

Same is happening with macOS Sierra 10.12.6. Adding the insecure registry in the preferences window now gives an issue when applying the changes, the daemon won't start, and the diagnosis fails to end.

@maxgorovenko

This comment has been minimized.

maxgorovenko commented Jan 10, 2018

Same problem, guys! Need hotfix ))

@TeaSeaLancs

This comment has been minimized.

TeaSeaLancs commented Jan 10, 2018

Just to clarify, my issue only affects pushing. I can pull fine :\

@maxgorovenko

This comment has been minimized.

maxgorovenko commented Jan 10, 2018

The same for me. Downgraded docker to Version 17.09.0-ce-mac33 (19543), works well again

@djs55

This comment has been minimized.

Contributor

djs55 commented Jan 10, 2018

Thanks for the report. I have reproduced this internally and am working on a fix.

@djs55

This comment has been minimized.

Contributor

djs55 commented Jan 10, 2018

I have a prototype fix now which seems to fix the problem. I'm working on adding the fix to a stable update.

In the meantime if you'd like to try it then

  • download vpnkit.zip
  • check sha1sum vpnkit.zip gives 89aeb8ae1501162543c8279e99a375820294ca38
  • unzip vpnkit.zip produces a single vpnkit binary
  • cp vpnkit /Applications/Docker.app/Contents/Resources/bin/vpnkit
  • restart the app
@TeaSeaLancs

This comment has been minimized.

TeaSeaLancs commented Jan 10, 2018

@djs55 Confirmed fixed with your prototype at this end. Thanks so much for the speedy response!

@vincent-dm

This comment has been minimized.

vincent-dm commented Jan 10, 2018

I can confirm, this fix works!

@joonathan

This comment has been minimized.

joonathan commented Jan 11, 2018

@djs55 did not work for me, what's the correct way to pull from private repository exposed via ssh tunnel on the Host? Previously going through exposed port on docker.for.mac.http.internal worked, but right now that nor localhost work.
Also it seems that "Bypass proxy" settings might not be honored when proxy is set?

@djs55

This comment has been minimized.

Contributor

djs55 commented Jan 11, 2018

@joonathan thanks for the update. Could you try 'docker.for.mac.localhost' instead of 'docker.for.mac.http.internal'. The reason being that these map to separate IPs: one for host passthrough and the other for internal services. This avoids masking too many ports on the real localhost when we add more internal services.

Failing that, could you tell me what your docker proxy settings are set to (approximately) and what your system settings are (if you're using the system proxy). Regarding "bypass proxy" could you describe what you mean? (Sorry there are now tooany proxy settings IMHO and I get confused!)

Thanks for your help!

@joonathan

This comment has been minimized.

joonathan commented Jan 11, 2018

@djs55 thanks, just tried three different ways:
docker login on 'docker.for.mac.localhost' and I can't see any traffic on the tunnel but receive "Error response from daemon: login attempt to http://docker.for.mac.localhost:5888/v2/ failed with status: 503 Service Unavailable"
When trying login on 'docker.for.mac.http.internal ' I get "Error response from daemon: Get https://docker.for.mac.http.internal:5888/v2/: Service Unavailable" (so slightly different).
When trying login on '127.0.0.1:5888' I get "Error response from daemon: Get http://127.0.0.1:5888/v2/: dial tcp 127.0.0.1:5888: getsockopt: connection refused"

These tests have been done without any Proxies defined (Use system proxy + no proxies defined in system network)

I'll test and try to write down the inability to bypass proxy issue I am seeing as well.

@djs55

This comment has been minimized.

Contributor

djs55 commented Jan 11, 2018

@joonathan thanks for the info. I've just fixed one possible cause of the "503 Service Unavailable" -- it just passed CI and has been merged. We're hoping to do a stable update with the fixes we have so far very soon now. I'll ping you when it's ready -- it'll probably be worth re-testing then.

@nfwbc

This comment has been minimized.

nfwbc commented Jan 11, 2018

I also can confirm "vpnkit.zip" hotfix above works! Thanks much!

@tdterry

This comment has been minimized.

tdterry commented Jan 11, 2018

I'm getting the same behavior on Sierra after upgrading Docker to 17.12.0-ce-mac46.

When I upgraded, there was an error, and the installer said it had to revert to factory defaults. I let it do that, and then I added my private, insecure registry to the list. It restarted fine, but when I try to push, I get "retrying" for all of the new layers. This happens for a 20-30 seconds, and then it ends with a "503 Service Unavailable"

$ docker push private-repo:5000/app:tag
The push refers to repository [private-repo:5000/app]
834b7dff8b87: Retrying in 3 seconds
59fdc91a882a: Retrying in 2 seconds
f373d1650803: Retrying in 2 seconds
f659fd85dcb1: Retrying in 2 seconds
2fbee56e53e3: Retrying in 3 seconds
e72d35aaf00e: Waiting
f33b551a42b9: Waiting
4b78214a873e: Waiting
17bf9b56042d: Waiting
ad9fa7c9d0aa: Waiting
a3cf95a0a769: Waiting
8b375be9de19: Waiting
0f8361255952: Waiting
71ce2dc7f761: Waiting
0d960f1d4fba: Waiting

It retries serval times and then exits with:

received unexpected HTTP status: 503 Service Unavailable
@TeaSeaLancs

This comment has been minimized.

TeaSeaLancs commented Jan 11, 2018

@tdterry See the response from @djs55 above for a workaround until an official patch is released.

@inkel

This comment has been minimized.

inkel commented Jan 12, 2018

I've installed the version you just released and it doesn't work for me. Not only that, adding the registry address in the preferences window broke Docker For Mac and it won't restart, and when I click Reset Docker to factory defaults it just hangs forever. Same if I click Diagnose & Feedback.

@joonathan

This comment has been minimized.

joonathan commented Jan 15, 2018

@djs55 Testing with Version 17.12.0-ce-mac47 (21805) and trying docker login docker.for.mac.localhost:5888 would still result in 503 Service Unavailable for me.

@evenson419

This comment has been minimized.

evenson419 commented Jan 15, 2018

@djs55 I've installed Docker Community Edition 17.12.0-ce-mac46,when run "docker run -d --name zookeeper --publish 2181:2181 --volume /etc/localtime:/etc/localtime wurstmeister/zookeeper:latest",there was an error

Error response from daemon: Mounts denied:
The path /etc/localtime
is not shared from OS X and is not known to Docker.
You can configure shared paths from Docker -> Preferences... -> File Sharing.
See https://docs.docker.com/docker-for-mac/osxfs/#namespaces for more info.

Downgraded docker to Version 17.09.1-ce-mac42, works well again!

@herbrandson

This comment has been minimized.

herbrandson commented Jan 17, 2018

@djs55 I upgraded to 17.12.0-ce-mac47 this morning and have been getting this error since. What info can I provide you to help diagnose the issue?

@ajeetraina

This comment has been minimized.

ajeetraina commented Feb 3, 2018

Tested it with Docker 18.02 RC2 build today morning and still facing the same issue:

docker push docker.for.mac.localhost:5000/shellcheck
The push refers to repository [docker.for.mac.localhost:5000/shellcheck]
40ca87475646: Retrying in 5 seconds
0e5242ddb369: Retrying in 5 seconds
6f5999d2f598: Retrying in 5 seconds
523000cf10a0: Retrying in 5 seconds
34ac85e258a0: Retrying in 5 seconds
020914655625: Waiting
57da2adfdb40: Waiting
cd7100a72410: Waiting

FYI.. I am using docker.for.mac.localhost:5000 under Daemon.

@barbarello

This comment has been minimized.

barbarello commented Feb 16, 2018

@djs55
your vpnkit fix did not work for me.

Docker Version 17.12.0-ce-mac49 (21995)

docker push registry:5000/alpine
The push refers to repository [registry:5000/alpine]
28b0187a4b92: Retrying in 1 second
1903490fcd5c: Retrying in 1 second
6398a5e25dd1: Retrying in 1 second
ca0637b9f6ed: Retrying in 1 second
b620754093cf: Retrying in 1 second
10ba7a7bd6b3: Waiting
11104dbc6848: Waiting
cd7100a72410: Waiting
received unexpected HTTP status: 503 Service Unavailable

docker push docker.for.mac.localhost:5000/alpine
The push refers to repository [docker.for.mac.localhost:5000/alpine]
Get https://docker.for.mac.localhost:5000/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

@inkel

This comment has been minimized.

inkel commented Feb 19, 2018

I've restarted my computer today, and there was a new version to install (17.12.0-ce-mac49). I tried it and it didn't even restart, which is the same pattern I've been observing when setting an insecure registry in the preferences. Resetting to factory defaults doesn't work either. I had to keep using 17.09.0-ce-mac33 in order to work with insecure registries.

@shawmanmohan

This comment has been minimized.

shawmanmohan commented Feb 26, 2018

This started happening when i upgraded docker to Docker 17.12.0-ce-mac49

@akira

This comment has been minimized.

akira commented Apr 12, 2018

Is there any update on this ticket? This preventing us from using latest Docker and interacting with registry on laptop. The issue is happening after upgrading to docker past version 17.09.0-ce:

Connecting to docker-registry (port 5055)...
retrieving latest docker image...
Error response from daemon: Get http://<redacted>:5055/v2/: net/http: request canceled (Client.Timeout exceeded while awaiting headers)```
@YRM64

This comment has been minimized.

YRM64 commented May 21, 2018

Would like to thank djs55 for his comments on the '503 Service Unavailable' fix, mappings to separate IPs, and especially the information on the 'prototyping' fix.

For developers using ssh, make sure your ssh-agent has your keys/loaded. If not, you have to add them to the agent, and to do that, run in the terminal:

ssh-add -l (to check if keys are loaded)
ssh-add <path_to_key> (to add keys to agent)

@docker-for-desktop-robot

This comment has been minimized.

Collaborator

docker-for-desktop-robot commented Aug 20, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30d of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

@yxleung

This comment has been minimized.

yxleung commented Sep 6, 2018

@TeaSeaLancs @djs55 Hi all, I got the same problem;
mac version:macOS High Sierra 10.36.6(17G2208)
docker version:18.06.1-ce, build e68fc7a

I try to use @djs55 vpnkit.zip to fix it ,but doesn't work; even docker service can't be started; please help me, thx!

@Labibme

This comment has been minimized.

Labibme commented Oct 20, 2018

Hello,
I have same issue, but in my case i have almost more than 4 registry all works beside one of nexus repo:

The push refers to repository [rct-registry.idraa.ca-assurances.intra/showenv]
fe7a735e2238: Retrying in 2 seconds
5b9e0c831fdd: Retrying in 2 seconds
ce81a764015e: Retrying in 2 seconds
fc6d645e7de4: Retrying in 2 seconds
2c5d23bd5df8: Retrying in 2 seconds
6f7e56b5c1d6: Waiting
1502f4628706: Waiting
ff855d4efdbd: Waiting
f7dc68d0140a: Waiting
b7b45276eab1: Waiting
bca93b504abd: Waiting
de0633edd635: Waiting
0c8e3eadebad: Waiting
d8e63a6eb30e: Waiting
c1ab3a964f08: Waiting
8fad67424c4e: Waiting
received unexpected HTTP status: 503 Service Unavailable

DEBUG:

f347bc9f920a82ddf3f60229de (sha256:f2b6b4884fc8b2f1fcef843f92f7c82c9c149df85ac77e5f0de7a342ae442412) in rct-registry.idraa.ca-assurances.intra/showenv"
Oct 20 12:07:33 vl-a-drx-35 dockerd: time="2018-10-20T12:07:33.380307620+02:00" level=debug msg="Failed to check for presence of layer sha256:d8e63a6eb30eee153f6e90666b64f98fd5c15fa07e7a126633a0e660a7a9d222 (sha256:6e0e41c52c70a0be891e3c033156b95b2b3183dadfd9f725f93f45e25e35bc73) in rct-registry.idraa.ca-assurances.intra/showenv" error="received unexpected HTTP status: 503 Service Unavailable"
Oct 20 12:07:33 vl-a-drx-35 dockerd: time="2018-10-20T12:07:33.380398636+02:00" level=debug msg="Pushing layer: sha256:d8e63a6eb30eee153f6e90666b64f98fd5c15fa07e7a126633a0e660a7a9d222"
Oct 20 12:07:33 vl-a-drx-35 dockerd: time="2018-10-20T12:07:33.388423700+02:00" level=debug msg="Failed to check for presence of layer sha256:c1ab3a964f08f0e920db03936892dc97ea863f3c9b5483d3c195ebc1943ba69c (sha256:8db887c458002053abb80fe7da3ed9071bad3ba45e982f07779927d7baf62bad) in rct-registry.idraa.ca-assurances.intra/showenv" error="received unexpected HTTP status: 503 Service Unavailable"
Oct 20 12:07:33 vl-a-drx-35 dockerd: time="2018-10-20T12:07:33.388490506+02:00" level=debug msg="Pushing layer: sha256:c1ab3a964f08f0e920db03936892dc97ea863f3c9b5483d3c195ebc1943ba69c"
Oct 20 12:07:33 vl-a-drx-35 dockerd: time="2018-10-20T12:07:33.407317249+02:00" level=debug msg="Failed to check for presence of layer sha256:8fad67424c4e7098f255513e160caa00852bcff347bc9f920a82ddf3f60229de (sha256:f2b6b4884fc8b2f1fcef843f92f7c82c9c149df85ac77e5f0de7a342ae442412) in rct-registry.idraa.ca-assurances.intra/showenv" error="received unexpected HTTP status: 503 Service Unavailable"
Oct 20 12:07:33 vl-a-drx-35 dockerd: time="2018-10-20T12:07:33.407453397+02:00" level=debug msg="Pushing layer: sha256:8fad67424c4e7098f255513e160caa00852bcff347bc9f920a82ddf3f60229de"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment