Error pushing container to Harbor repo with OIDC SSO auth

[dioguerra]

Hello!

I am having some problems pushing a container from singularity to my Harbor registry. At this time i think the problem is in the Headers being droped by my nginx reverse-proxy.

Although i suspect this, i can't validate for sure because if i try locally, i have no info on any Headers passed by Singularity, and documentation online is scarce.

My SetUp
I Have a kubernetes cluster running Harbor. The registry is behind a main load balancer that maps to the nginx-ingress in the cluster. The nginx in turn maps the requests to the pods.

Auth is managed by our internal SSO, so clients to push images must submit username and token (obtained from the harbor registry). This implementation is working nicely for ML models (using ORMB) and docker push, but not for singularity.

Testing
If i try with demo harbor repo it works successfully:

singularity pull docker://alpine
export SINGULARITY_DOCKER_USERNAME=test01
export SINGULARITY_DOCKER_PASSWORD=Test01(password)
singularity push alpine_latest.sif oras://demo.goharbor.io/test01/alpine:latest
INFO:    Upload complete

But with my deployment:

export SINGULARITY_DOCKER_USERNAME=dioguerra
export SINGULARITY_DOCKER_PASSWORD=REDACTED(token)
singularity push alpine_latest.sif oras://registry.domain.com/dioguerra/alpine:latest
FATAL:   Unable to push image to oci registry: unable to push: failed commit on ref "unknown-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855": unexpected status: 500 Internal Server Error

Tracking Problem (trying)
Now, for those following, the resulting SHA256 is actually the hash from an empty file.
Bellow is the output of the Harbor Core server:

2020-10-23T12:28:20Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 82971b669aa6b57d8b17c8aabb63e137 to the logger for the request HEAD /v2/dioguerra/alpine/blobs/sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2020-10-23T12:28:20Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:52]: In artifact info middleware, url: /v2/dioguerra/alpine/blobs/sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2020-10-23T12:28:20Z [DEBUG] [/server/middleware/security/unauthorized.go:29][requestID="82971b669aa6b57d8b17c8aabb63e137"]: an unauthorized security context generated for request HEAD /v2/dioguerra/alpine/blobs/sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2020-10-23T12:28:20Z [DEBUG] [/lib/http/error.go:59]: {"errors":[{"code":"UNAUTHORIZED","message":"authorize header needed to send HEAD to repository: authorize header needed to send HEAD to repository"}]}
2020-10-23T12:28:20Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 6d6722bf59f654a67acbed51f6b88583 to the logger for the request HEAD /v2/dioguerra/alpine/blobs/sha256:0ab22bfbf12f2928869de32502442f74cc237fe701c9ff61ff172885c37f5380
2020-10-23T12:28:20Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:52]: In artifact info middleware, url: /v2/dioguerra/alpine/blobs/sha256:0ab22bfbf12f2928869de32502442f74cc237fe701c9ff61ff172885c37f5380
2020-10-23T12:28:20Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id c5aca6ab834bb4f866cec89c39cb5fb3 to the logger for the request PUT /v2/dioguerra/alpine/blobs/uploads/3cc3da6c-75cc-4973-be03-0e205b1a8cbe
2020-10-23T12:28:20Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:52]: In artifact info middleware, url: /v2/dioguerra/alpine/blobs/uploads/3cc3da6c-75cc-4973-be03-0e205b1a8cbe?_state=REDACTED
2020-10-23T12:28:20Z [DEBUG] [/server/middleware/quota/put_blob_upload.go:50][action="request" middleware="quota" requestID="c5aca6ab834bb4f866cec89c39cb5fb3" url="/v2/dioguerra/alpine/blobs/uploads/3cc3da6c-75cc-4973-be03-0e205b1a8cbe"]: blob size is 0
2020-10-23T12:28:20Z [DEBUG] [/server/middleware/quota/quota.go:102][action="request" middleware="quota" requestID="c5aca6ab834bb4f866cec89c39cb5fb3" url="/v2/dioguerra/alpine/blobs/uploads/3cc3da6c-75cc-4973-be03-0e205b1a8cbe"]: no resources request for this http request, so direct to next handler

See:

1. https://groups.google.com/a/lbl.gov/forum/embed/#!topic/singularity/bMZ_Hv47XgU
2. return 500 when push a txt file as oras goharbor/harbor#11239

The bit of code that actually triggers the error is here:

		if a.target == repository && req.Header.Get(authHeader) == "" && req.Method == http.MethodHead { // make sure 401 is returned for CLI HEAD, see #11271
			return getChallenge(req, al), fmt.Errorf("authorize header needed to send HEAD to repository")

Now, this can come have 2 root causes:

1. The singularity client is not sending back the received SSO token, or is conclicting with SSO in some way.
2. Headers are being dropped in the nginx reverse-proxy because req.Header.Get(authHeader) matches empty string.

I checked with nginx and, headers are dropped if they start with underscore. I tried to check if singularity would send any headers like this but to my surprise, all the headers on all requests are empty (tho this is not a very sure way of validating)

I got this from the python server that i ran locally

INFO:root:Starting httpd...

Host: localhost:8080
User-Agent: containerd/1.4.0+unknown
Accept: application/vnd.sylabs.sif.config.v1+json, */*


INFO:root:HEAD request,
Path: /v2/dtomasgu/alpine.sif/blobs/sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Headers:
Host: localhost:8080
User-Agent: containerd/1.4.0+unknown
Accept: application/vnd.sylabs.sif.config.v1+json, */*



127.0.0.1 - - [30/Oct/2020 18:56:38] "HEAD /v2/dtomasgu/alpine.sif/blobs/sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 HTTP/1.1" 200 -
Host: localhost:8080
User-Agent: containerd/1.4.0+unknown
Accept: appliciation/vnd.sylabs.sif.layer.tar, */*


INFO:root:HEAD request,
Path: /v2/dtomasgu/alpine.sif/blobs/sha256:c43c3ff660e01b55eb5d041d15cdcbd74a93e7d7d48fd14de03becd9bf42c979
Headers:
Host: localhost:8080
User-Agent: containerd/1.4.0+unknown
Accept: appliciation/vnd.sylabs.sif.layer.tar, */*



127.0.0.1 - - [30/Oct/2020 18:56:38] "HEAD /v2/dtomasgu/alpine.sif/blobs/sha256:c43c3ff660e01b55eb5d041d15cdcbd74a93e7d7d48fd14de03becd9bf42c979 HTTP/1.1" 200 -
Host: localhost:8080
User-Agent: containerd/1.4.0+unknown
Accept: application/vnd.oci.image.manifest.v1+json, */*


INFO:root:HEAD request,
Path: /v2/dtomasgu/alpine.sif/manifests/latest
Headers:
Host: localhost:8080
User-Agent: containerd/1.4.0+unknown
Accept: application/vnd.oci.image.manifest.v1+json, */*



127.0.0.1 - - [30/Oct/2020 18:56:38] "HEAD /v2/dtomasgu/alpine.sif/manifests/latest HTTP/1.1" 200 -
Host: localhost:8080
User-Agent: containerd/1.4.0+unknown
Content-Length: 410
Content-Type: application/vnd.oci.image.manifest.v1+json
Accept-Encoding: gzip


INFO:root:POST request,
Path: /v2/dtomasgu/alpine.sif/manifests/latest
Headers:
Host: localhost:8080
User-Agent: containerd/1.4.0+unknown
Content-Length: 410
Content-Type: application/vnd.oci.image.manifest.v1+json
Accept-Encoding: gzip



Body:
{"schemaVersion":2,"config":{"mediaType":"application/vnd.sylabs.sif.config.v1+json","digest":"sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","size":0},"layers":[{"mediaType":"appliciation/vnd.sylabs.sif.layer.tar","digest":"sha256:c43c3ff660e01b55eb5d041d15cdcbd74a93e7d7d48fd14de03becd9bf42c979","size":2760704,"annotations":{"org.opencontainers.image.title":"alpine_latest.sif"}}]}

127.0.0.1 - - [30/Oct/2020 18:56:38] "PUT /v2/dtomasgu/alpine.sif/manifests/latest HTTP/1.1" 200 -

I'm trying to understand which headers are sent from singularity, but until now i can't understand this.
Does anybody had the same problem? How did you solve it? Some pointers here would be appreciated...

[dioguerra]

Track from Harbor: goharbor/harbor#13420

[vsoch]

hey @mrd2 ! This looks like an issue with Singularity not being compliant for the OCI distribution spec, and this is indeed true. The repository here is no longer maintained - it's the old documentation site for Singularity containers, which eventually was ported to the sylabs organization, and then hpcng. You should likely close the issue here and open it up at https://github.com/hpcng/singularity/issues. I too would very much like to see the singularity command line client be OCI compliant for a registry, but it's likely a tough push because the company that runs Singularity has their own proprietary "library" client and server, and having OCI compliance would then make it easy to push to any registry and thus lose customers.

If you don't get help over there, we could work on a simple script (go lang would work) that would serve to push/pull and otherwise interact with the registry for singularity, of course it would still be annoying that Singularity can't just handle it directly. But it would work easily because the SIF image is just a kind of blob, and we'd just need to artificially create a config blob and then the manifest. Feel free to ping me on the issue so I can follow along. Good luck!

[dioguerra]

Closing this in favor of: apptainer/singularity#5691

[dtrudg]

hey @mrd2 ! This looks like an issue with Singularity not being compliant for the OCI distribution spec, and this is indeed true.

I too would very much like to see the singularity command line client be OCI compliant for a registry, but it's likely a tough push because the company that runs Singularity has their own proprietary "library" client and server, and having OCI compliance would then make it easy to push to any registry and thus lose customers.

This is not the case. Singularity includes functionality for pushing to OCI registries that implement ORAS, and we routinely test this against docker/registry and know it to work against Azure Container Registry. Singularity requires a target registry to support ORAS and accept the singularity media types. ORAS is a young protocol, and our code is not heavily used yet, so we can expect some issues with some registries, but we are interested in fixing these when they arise. We cannot push to OCI registries without ORAS support as the Singularity container format is not the same as OCI containers. We do not have the same manifest/config/layers structure - everything is within a single file.

Additionally, please note that Singularity is an open source project that gladly includes contributions in this area. Recently, improved authentication handling for OCI registries has been merged for the upcoming 3.7.0 release.