docker_push not working

[chaitanya9186]

Build rules:

go_binary(
    name = "greeter_http_server",
    srcs = ["greeter_http.go"],
    deps = [":helloworld_proto_go"] + GRPC_GATEWAY_DEPS,
)

docker_build(
    name = "greeter_http_server_docker",
    base = "@joeshaw_busybox_nonroot//image:image.tar",
    cmd = ["./greeter_http_server"],
    files = [":greeter_http_server"],
    ports = ["8080"],
    repository = "registry.gitlab.com/<blah>",
    symlinks = {
        "/usr/bin/greeter_http_server": "greeter_http_server",
    },
)

docker_push(
    name = "greeter_http_server_docker_push",
    image = ":greeter_http_server_docker",
    registry = "registry.gitlab.com",
    repository = "<blah>/samples/helloworld",
    tag = "dev",
)

Error:

$ bazeldev run samples/helloworld:greeter_http_server_docker_push
INFO: Found 1 target...
Target //samples/helloworld:greeter_http_server_docker_push up-to-date:
  bazel-bin/samples/helloworld/greeter_http_server_docker_push
INFO: Elapsed time: 0.544s, Critical Path: 0.29s

INFO: Running command line: bazel-bin/samples/helloworld/greeter_http_server_docker_push
+ ../pusher/file/pusher.par --name=registry.gitlab.com/<blah>/samples/helloworld:dev --tarball=samples/helloworld/greeter_http_server_docker.tar
ERROR:root:Error during upload of: registry.gitlab.com/<blah>/samples/helloworld:dev
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "../pusher/file/pusher.par/__main__.py", line 71, in <module>
  File "../pusher/file/pusher.par/__main__.py", line 66, in main
  File "../pusher/file/pusher.par/containerregistry/client/v2_2/docker_session_.py", line 294, in upload
  File "../pusher/file/pusher.par/concurrent/futures/_base.py", line 398, in result
  File "../pusher/file/pusher.par/concurrent/futures/thread.py", line 55, in run
  File "../pusher/file/pusher.par/containerregistry/client/v2_2/docker_session_.py", line 264, in _upload_one
  File "../pusher/file/pusher.par/containerregistry/client/v2_2/docker_session_.py", line 186, in _put_blob
  File "../pusher/file/pusher.par/containerregistry/client/v2_2/docker_session_.py", line 125, in _monolithic_upload
  File "../pusher/file/pusher.par/containerregistry/client/v2_2/docker_http_.py", line 333, in Request
containerregistry.client.v2_2.docker_http_.V2DiagnosticException: response: {'status': '202', 'content-length': '0', 'content-security-policy': "object-src 'none'; script-src 'self' 'unsafe-inline' 'unsafe-eval' piwik.gitlab.com https://www.google.com/recaptcha/ https://www.gstatic.com/recaptcha/; style-src 'self' 'unsafe-inline'; img-src * data: blob:; frame-src 'self' https://www.google.com/recaptcha/; frame-ancestors 'none'; connect-src 'self' wss://gitlab.com; report-uri https://sentry-infra.gitlap.com/api/3/csp-report/?sentry_key=a664fdde83424b43a991f25fa7c78987", 'x-content-type-options': 'nosniff', 'docker-upload-uuid': '9e142cbb-38be-4023-a995-3973871bb035', 'server': 'nginx', 'range': '0-0', 'docker-distribution-api-version': 'registry/2.0', 'location': 'https://registry.gitlab.com/v2/<blah>/samples/helloworld/blobs/uploads/9e142cbb-38be-4023-a995-3973871bb035?_state=e7GilJ0V9PBDIMONe0vaApbNxhosgpwZNLyj1rpjph17Ik5hbWUiOiJjaGFpdGFueWE5MTg2L3NyZWxsaWszL3NhbXBsZXMvaGVsbG93b3JsZCIsIlVVSUQiOiI5ZTE0MmNiYi0zOGJlLTQwMjMtYTk5NS0zOTczODcxYmIwMzUiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTctMDUtMDFUMjM6MDI6MjEuMTkxMzE3MTE1WiJ9', 'date': 'Mon, 01 May 2017 23:02:21 GMT', 'content-type': 'text/plain; charset=utf-8'}
: None
ERROR: Non-zero return code '1' from command: Process exited with status 1.

Env details:

$ docker -v
Docker version 17.05.0-ce-rc2, build c57fdb2a14cfba584686ddad909e3006284d10aa

$ sudo cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=17.04
DISTRIB_CODENAME=zesty
DISTRIB_DESCRIPTION="Ubuntu 17.04"
NAME="Ubuntu"
VERSION="17.04 (Zesty Zapus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 17.04"
VERSION_ID="17.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=zesty
UBUNTU_CODENAME=zesty

[chaitanya9186]

Forgot to mention:
I able to use docker_build rule to build and tag the docker image and able to push the same using docker push <image> command.

[mattmoor]

Thanks for filing the issue!

I will try to reproduce this myself, but it smells like an incompatibility with Gitlab (I'll have to find a way to set up an account to test pushing).

[mattmoor]

Also, for completeness, have you tried docker_pull with Gitlab and it is working?

[chaitanya9186]

I have tried docker_pull, it looks like working to me.

[mattmoor]

Ah, I know what the problem is.

This is using a variant of the specification that is more efficient (for the sizes of layers typically produced by Bazel), but that docker/distribution does not implement, but that I implemented in GCR.

Basically, for blob uploads there are three modes:

1. POST w/ layer payload as body -- Unsupported by docker/distribution
2. POST/PUT w/ layer payload as body of PUT -- pre-1.8
3. POST/PATCH/PUT w/ layer payload as body of PATCH -- 1.8+

Cross-repo mounting is sort of like a 1b., but it falls back on 3. when it fails.

The client we use has dormant functionality to implement the other modes (that I implemented for benchmarking which was best), but it'll take a little doing to update the unit testing with the new requests to expect.

My apologies, I had forgotten we used this under the hood. I'll try to get a fix in and a new release cut in the next few days.

[chaitanya9186]

Awesome. Thanks for the update and the details.

[mattmoor]

This comment spells it out the best, and you are the unfortunate victim of my TODO :)
https://github.com/google/containerregistry/blob/master/client/v2_2/docker_session_.py#L183

[mattmoor]

tl;dr I have a couple changes out for (internal) review to the underlying tooling that performs the push, which I believe will resolve this.

With these changes I have already successfully pushed an image to DockerHub and Gitlab, but...

After setting up an account on registry.gitlab.com my initial attempt at using it to push there has resulted in a failure similar to one I initially encountered with JFrog's bintray, which was a result of them using a feature of HTTPS called SNI.

The problem is that the version of python 2.7 bundled with the latest Ubuntu LTS is a couple minor versions short of where this is added. You can see my more detailed comment here.

The good news (for you) is that if docker_pull was successful and docker_push got as far as the failure you experienced, then you are already past this and shouldn't have this problem.

I'll add a limited pull test for Gitlab (similar to the above bintray) to the Travis configuration shortly.

[mattmoor]

@chaitanya9186 Can you try using the commit in the linked PR and confirm that this resolves your problem?

[chaitanya9186]

@mattmoor I have just tried it. It is working.

Thanks for the quick fix.

[mattmoor]

Awesome, I'll merge the PR then. LMK if you have any other issues.