Singularity cannot push to harbor

[dioguerra]

$ singularity version
3.6.2

Hello!

I am having some problems pushing a container from singularity to my Harbor registry. At this time i think the problem is in the Headers being droped by my nginx reverse-proxy.

Although i suspect this, i can't validate for sure because if i try locally, i have no info on any Headers passed by Singularity, and documentation online is scarce.

My SetUp
I Have a kubernetes cluster running Harbor. The registry is behind a main load balancer that maps to the nginx-ingress in the cluster. The nginx in turn maps the requests to the pods.

Auth is managed by our internal SSO, so clients to push images must submit username and token (obtained from the harbor registry). This implementation is working nicely for ML models (using ORMB) and docker push, but not for singularity.

Testing
If i try with demo harbor repo it works successfully:

singularity pull docker://alpine
export SINGULARITY_DOCKER_USERNAME=test01
export SINGULARITY_DOCKER_PASSWORD=Test01(password)
singularity push alpine_latest.sif oras://demo.goharbor.io/test01/alpine:latest
INFO:    Upload complete

But with my deployment:

export SINGULARITY_DOCKER_USERNAME=dioguerra
export SINGULARITY_DOCKER_PASSWORD=REDACTED(token)
singularity push alpine_latest.sif oras://registry.domain.com/dioguerra/alpine:latest
FATAL:   Unable to push image to oci registry: unable to push: failed commit on ref "unknown-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855": unexpected status: 500 Internal Server Error

Tracking Problem (trying)
Now, for those following, the resulting SHA256 is actually the hash from an empty file.
Bellow is the output of the Harbor Core server:

2020-10-23T12:28:20Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 82971b669aa6b57d8b17c8aabb63e137 to the logger for the request HEAD /v2/dioguerra/alpine/blobs/sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2020-10-23T12:28:20Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:52]: In artifact info middleware, url: /v2/dioguerra/alpine/blobs/sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2020-10-23T12:28:20Z [DEBUG] [/server/middleware/security/unauthorized.go:29][requestID="82971b669aa6b57d8b17c8aabb63e137"]: an unauthorized security context generated for request HEAD /v2/dioguerra/alpine/blobs/sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2020-10-23T12:28:20Z [DEBUG] [/lib/http/error.go:59]: {"errors":[{"code":"UNAUTHORIZED","message":"authorize header needed to send HEAD to repository: authorize header needed to send HEAD to repository"}]}
2020-10-23T12:28:20Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 6d6722bf59f654a67acbed51f6b88583 to the logger for the request HEAD /v2/dioguerra/alpine/blobs/sha256:0ab22bfbf12f2928869de32502442f74cc237fe701c9ff61ff172885c37f5380
2020-10-23T12:28:20Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:52]: In artifact info middleware, url: /v2/dioguerra/alpine/blobs/sha256:0ab22bfbf12f2928869de32502442f74cc237fe701c9ff61ff172885c37f5380
2020-10-23T12:28:20Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id c5aca6ab834bb4f866cec89c39cb5fb3 to the logger for the request PUT /v2/dioguerra/alpine/blobs/uploads/3cc3da6c-75cc-4973-be03-0e205b1a8cbe
2020-10-23T12:28:20Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:52]: In artifact info middleware, url: /v2/dioguerra/alpine/blobs/uploads/3cc3da6c-75cc-4973-be03-0e205b1a8cbe?_state=REDACTED
2020-10-23T12:28:20Z [DEBUG] [/server/middleware/quota/put_blob_upload.go:50][action="request" middleware="quota" requestID="c5aca6ab834bb4f866cec89c39cb5fb3" url="/v2/dioguerra/alpine/blobs/uploads/3cc3da6c-75cc-4973-be03-0e205b1a8cbe"]: blob size is 0
2020-10-23T12:28:20Z [DEBUG] [/server/middleware/quota/quota.go:102][action="request" middleware="quota" requestID="c5aca6ab834bb4f866cec89c39cb5fb3" url="/v2/dioguerra/alpine/blobs/uploads/3cc3da6c-75cc-4973-be03-0e205b1a8cbe"]: no resources request for this http request, so direct to next handler

See:

1. https://groups.google.com/a/lbl.gov/forum/embed/#!topic/singularity/bMZ_Hv47XgU
2. return 500 when push a txt file as oras goharbor/harbor#11239

The bit of code that actually triggers the error is here:

		if a.target == repository && req.Header.Get(authHeader) == "" && req.Method == http.MethodHead { // make sure 401 is returned for CLI HEAD, see #11271
			return getChallenge(req, al), fmt.Errorf("authorize header needed to send HEAD to repository")

Now, this can come have 2 root causes:

1. The singularity client is not sending back the received SSO token, or is conclicting with SSO in some way.
2. Headers are being dropped in the nginx reverse-proxy because req.Header.Get(authHeader) matches empty string.

I checked with nginx and, headers are dropped if they start with underscore. I tried to check if singularity would send any headers like this but to my surprise, all the headers on all requests are empty (tho this is not a very sure way of validating)

I got this from the python server that i ran locally

INFO:root:Starting httpd...

Host: localhost:8080
User-Agent: containerd/1.4.0+unknown
Accept: application/vnd.sylabs.sif.config.v1+json, */*


INFO:root:HEAD request,
Path: /v2/dtomasgu/alpine.sif/blobs/sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Headers:
Host: localhost:8080
User-Agent: containerd/1.4.0+unknown
Accept: application/vnd.sylabs.sif.config.v1+json, */*



127.0.0.1 - - [30/Oct/2020 18:56:38] "HEAD /v2/dtomasgu/alpine.sif/blobs/sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 HTTP/1.1" 200 -
Host: localhost:8080
User-Agent: containerd/1.4.0+unknown
Accept: appliciation/vnd.sylabs.sif.layer.tar, */*


INFO:root:HEAD request,
Path: /v2/dtomasgu/alpine.sif/blobs/sha256:c43c3ff660e01b55eb5d041d15cdcbd74a93e7d7d48fd14de03becd9bf42c979
Headers:
Host: localhost:8080
User-Agent: containerd/1.4.0+unknown
Accept: appliciation/vnd.sylabs.sif.layer.tar, */*



127.0.0.1 - - [30/Oct/2020 18:56:38] "HEAD /v2/dtomasgu/alpine.sif/blobs/sha256:c43c3ff660e01b55eb5d041d15cdcbd74a93e7d7d48fd14de03becd9bf42c979 HTTP/1.1" 200 -
Host: localhost:8080
User-Agent: containerd/1.4.0+unknown
Accept: application/vnd.oci.image.manifest.v1+json, */*


INFO:root:HEAD request,
Path: /v2/dtomasgu/alpine.sif/manifests/latest
Headers:
Host: localhost:8080
User-Agent: containerd/1.4.0+unknown
Accept: application/vnd.oci.image.manifest.v1+json, */*



127.0.0.1 - - [30/Oct/2020 18:56:38] "HEAD /v2/dtomasgu/alpine.sif/manifests/latest HTTP/1.1" 200 -
Host: localhost:8080
User-Agent: containerd/1.4.0+unknown
Content-Length: 410
Content-Type: application/vnd.oci.image.manifest.v1+json
Accept-Encoding: gzip


INFO:root:POST request,
Path: /v2/dtomasgu/alpine.sif/manifests/latest
Headers:
Host: localhost:8080
User-Agent: containerd/1.4.0+unknown
Content-Length: 410
Content-Type: application/vnd.oci.image.manifest.v1+json
Accept-Encoding: gzip



Body:
{"schemaVersion":2,"config":{"mediaType":"application/vnd.sylabs.sif.config.v1+json","digest":"sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","size":0},"layers":[{"mediaType":"appliciation/vnd.sylabs.sif.layer.tar","digest":"sha256:c43c3ff660e01b55eb5d041d15cdcbd74a93e7d7d48fd14de03becd9bf42c979","size":2760704,"annotations":{"org.opencontainers.image.title":"alpine_latest.sif"}}]}

127.0.0.1 - - [30/Oct/2020 18:56:38] "PUT /v2/dtomasgu/alpine.sif/manifests/latest HTTP/1.1" 200 -

I'm trying to understand which headers are sent from singularity, but until now i can't understand this.
Does anybody had the same problem? How did you solve it? Some pointers here would be appreciated...

[dioguerra]

Do we have any roadmap for this?
Is this something that is planned?

@vsoch Ping as requested

[dtrudg]

What version of harbor are you using? Singularity will push oras without issue to other registries including docker/registry, and the Azure Container Registry so it is not a universal issue pushing to an OCI registry.

[dioguerra]

I'm using Harbor 2.1 installed with Helm Chart v1.5.0

[dtrudg]

Thanks - we'll look into this further. @ikaneshiro - any thoughts off the top of your head?

[dtrudg]

@mrd2 - just came across this comment: goharbor/harbor#13420 (comment)

I tried to upload with admin:password. Still didn't work. I get the exact same error.

In this case I think we would have the situation where:

• It works against oras://demo.goharbor.io/ with a username/password (reported in your top comment here)
• It doesn't work against your harbor deployment with token or a direct account username/password pairing

However this is assuming your harbor deployment admin:password login is not going via OIDC SSO?

Can you confirm whether or not that is the case? We need to get a matching setup to diagnose this. If you can try with a non-SSO / OIDC configured harbor instance with a username/password that would be helpful.

Singularity uses the github.com/deislabs/oras module for ORAS pushes. This in turn uses the github.com/containers/image module for communication with the registry. It's possible that there may be an issue with OIDC/SSO auth not being supported there, within or between those modules, - which isn't something we can directly address in Singularity. Alternatively we might not be setting things up quite right. We need to isolate the reproducer first, though.

[dioguerra]

I updated What i said. I tried admin:pass still with the OSS set up assuming that Harbor's internal override of the auth mode to db_auth would make it work.

I will try with a set up without SSO to check this furder. It will also help to completly discard a proxy problem.
Will post here when i have news.

[dioguerra]

So, new evidence comes to light that this is probably an SSO problem and/or derived from github.com/containers/image.

One of our users that is actively pushing for this just gave me a glimpse of an error when pushing to gitlab registry that presents the exact same error as with harbor.

I will try to search for some issue in gitlab and update

@vsoch for follow ref
I'm thinking that if this is a reply from the SSO redirect (that creates the empty file) maybe we can drop the push of said file

[dioguerra]

Hey. I'm closing in on the problem, this seems to be a CLI side error.
With access to the same registry, i can use ormb to upload model images with no problem.
As both this projects, singularity and ormb, use the same oras package AND as singularity oras package in my tests is more recent than ormb i conclude that this is a CLI side error.

To test this, i am currently testing submitting the sif package manually into the remote registry. By following the manifest obtained before.

{
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.sylabs.sif.config.v1+json",
    "digest": "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "size": 0
  },
  "layers": [
    {
      "mediaType": "appliciation/vnd.sylabs.sif.layer.tar",
      "digest": "sha256:c43c3ff660e01b55eb5d041d15cdcbd74a93e7d7d48fd14de03becd9bf42c979",
      "size": 2760704,
      "annotations": {
        "org.opencontainers.image.title": "alpine_latest.sif"
      }
    }
  ]
}

Using the oras CLI:

oras version
Version:        0.7.0
Go version:     go1.12.4
Git commit:     7467008b2683c5eff5fb13fe91c5b17b2eec75a3
Git tree state: clean

I created a dir as follows:

../test/
├── alpine_latest.sif
└── config.json

Pushing with oras yields the same error.

oras push registry.foo.bar/dtomasgu/alpine:latest --manifest-config /dev/null:appliciation/vnd.sylabs.sif.layer.tar alpine_latest.sif
Uploading 93fbf43f73c4 alpine_latest.sif
Error: failed commit on ref "unknown-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855": unexpected status: 500 Internal Server Error

I suspect this is a manifest config problem. So now trying to upload with the configuration file as per oras docs:

This is the contents of my config.json

{
  "mediaType": "application/vnd.sylabs.sif.config.v1+json",
  "digest": "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
  "size": 0
}

Pushing:

oras push registry.foo.bar/dtomasgu/alpine:latest --manifest-config config.json:appliciation/vnd.sylabs.sif.layer.tar alpine_latest.sif
Uploading 93fbf43f73c4 alpine_latest.sif
Pushed registry.foo.bar/dtomasgu/alpine:latest
Digest: sha256:996bc74b7705fcc5659ce86549cbaef42b12891e32ae1b9b632320d4bd1ef630

Note the --manifest-config now includes the config.json file instead of /dev/null and the image uploads successfully.
This can be a user error (are we supposed to create the manifest) or is the manifest generated automatically? This can also point out that Harbor does not respond correctly to a /dev/null config file

At this point in time, i dont understand how the manifest config options work (i would apreciate some instruction here) but refactoring this it seems that we want something like this??

$oras push registry.foo.bar/dtomasgu/alpine:latest --manifest-config config.json:application/vnd.sylabs.sif.config.v1+json alpine_latest.sif
Uploading 93fbf43f73c4 alpine_latest.sif
Pushed registry.foo.bar/dtomasgu/alpine:latest
Digest: sha256:0032663c2712dbce3418b30574db24239e15072038bfedf42aed29808ad73aa9

~/singularity/test$ cat config.json 
{
  "mediaType": "appliciation/vnd.sylabs.sif.layer.tar"
}

oras push registry.foo.bar/dtomasgu/alpine:latest alpine_latest.sif:appliciation/vnd.sylabs.sif.layer.tar
Uploading 93fbf43f73c4 alpine_latest.sif
Pushed registry.foo.bar/dtomasgu/alpine:latest
Digest: sha256:a2b2c2158d7935706090d05930fba0dcf0af12498c8e2800a1b9c7d8df931526

Because Harbor may say that this is fixed in goharbor/harbor@033d6da
i replicated the exact same operations with my registry-staging repository which is currently running Harbor image v2.2.0-rc2 where this commit is present. Results below:

oras push registry-staging.foo.bar/dtomasgu/alpine:latest --manifest-config /dev/null:appliciation/vnd.sylabs.sif.layer.tar alpine_latest.sif
Uploading 93fbf43f73c4 alpine_latest.sif
Error: failed commit on ref "unknown-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855": unexpected status: 500 Internal Server Error

oras push registry-staging.foo.bar/dtomasgu/alpine:latest --manifest-config config.json:appliciation/vnd.sylabs.sif.layer.tar alpine_latest.sif
Uploading 93fbf43f73c4 alpine_latest.sif
Pushed registry-staging.foo.bar/dtomasgu/alpine:latest
Digest: sha256:996bc74b7705fcc5659ce86549cbaef42b12891e32ae1b9b632320d4bd1ef630

$ oras push registry-staging.foo.bar/dtomasgu/alpine:latest --manifest-config config.json:application/vnd.sylabs.sif.config.v1+json alpine_latest.sif
Uploading 93fbf43f73c4 alpine_latest.sif
Pushed registry-staging.foo.bar/dtomasgu/alpine:latest
Digest: sha256:0032663c2712dbce3418b30574db24239e15072038bfedf42aed29808ad73aa9

~/singularity/test$ cat config.json 
{
  "mediaType": "appliciation/vnd.sylabs.sif.layer.tar"
}

$ oras push registry-staging.foo.bar/dtomasgu/alpine:latest alpine_latest.sif:appliciation/vnd.sylabs.sif.layer.tar
Uploading 93fbf43f73c4 alpine_latest.sif
Pushed registry-staging.foo.bar/dtomasgu/alpine:latest
Digest: sha256:a2b2c2158d7935706090d05930fba0dcf0af12498c8e2800a1b9c7d8df931526

Same results back to back between Harbor v2.1.0 and v2.2.0-rc2.

Fetching from the registry, something is missing/wrong:

singularity pull oras://registry.foo.bar/dtomasgu/alpine:latest
FATAL:   While pulling image from oci registry: error fetching image to cache: failed to get checksum for oras://registry.foo.bar/dtomasgu/alpine:latest: no layer found corresponding to SIF image

[dioguerra]

$~/singularity/test$ oras push registry.foo.bar/dtomasgu/alpine:latest --manifest-config config.json alpine_latest.sif:appliciation/vnd.sylabs.sif.layer.tar
Uploading 93fbf43f73c4 alpine_latest.sif
Pushed registry.foo.bar/dtomasgu/alpine:latest
Digest: sha256:d2b50c174801f6838998b89a0785569afb2f29df9a607d13d0ea79958f538843

$~/singularity/test$ cat config.json 
{
  "mediaType": "application/vnd.sylabs.sif.config.v1+json",
  "digest": "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
  "size": 0
}

$ singularity pull oras://registry.foo.bar/dtomasgu/alpine:latest
INFO:    Downloading oras image

[dtrudg]

Thanks for all of this information. Does harbor accept a push using oras if the config.json you use is an empty JSON object:

$ cat config.json
{}

If it does accept that, it appears we should probably just use that instead of /dev/null to allow it to work with harbor.

However, it still seems like harbor should probably accept the completely empty /dev/null config. The oras repo gives explicit examples for this in its README (presumably these don't work with harbor either):

https://github.com/deislabs/oras#pushing-artifacts-with-single-files

It is also noted there that the config is optional...

The OCI distribution-spec provides for storing optional config objects.

If this is within-spec use of oras it seems this is a harbor conformance problem - and I note that you are following that route too.

[dioguerra]

Apparently, there are 3 conformance tests failing with Harbor. I don't know if this is specific of Harbor being behind the OIDC.

With config.json empty and pushing with
oras push registry.foo.bar/dtomasgu/alpine:latest --manifest-config config.json alpine_latest.sif

and it fails with

singularity pull oras://registry.foo.bar/dtomasgu/alpine:latest
FATAL:   While pulling image from oci registry: error fetching image to cache: failed to get checksum for oras://registry.foo.bar/dtomasgu/alpine:latest: no layer found corresponding to SIF image

But, with the same image file if i set the :format, it works

$ oras push registry.foo.bar/dtomasgu/alpine:latest --manifest-config config.json alpine_latest.sif:appliciation/vnd.sylabs.sif.layer.tar
Uploading 93fbf43f73c4 alpine_latest.sif
Pushed registry.foo.bar/dtomasgu/alpine:latest
Digest: sha256:b4818870eee6682ae4b6caa2e118d9de538af495a6c0062f6f9cd6683477ea7f

$~/singularity$ singularity cache clean 
This will delete everything in your cache (containers from all sources and OCI blobs). 
Hint: You can see exactly what would be deleted by canceling and using the --dry-run option.
Do you want to continue? [N/y] y
INFO:    No cached files to remove at /home/dtomasgu/.singularity/cache/blob
INFO:    No cached files to remove at /home/dtomasgu/.singularity/cache/library
INFO:    No cached files to remove at /home/dtomasgu/.singularity/cache/oci-tmp
INFO:    No cached files to remove at /home/dtomasgu/.singularity/cache/shub
INFO:    No cached files to remove at /home/dtomasgu/.singularity/cache/oras
INFO:    No cached files to remove at /home/dtomasgu/.singularity/cache/net

$~/singularity$ singularity pull oras://registry.foo.bar/dtomasgu/alpine:latest
INFO:    Downloading oras image

[dtrudg]

Thanks @dioguerra - since harbor is getting more and more used I'll look at putting the workaround in our code even if this appears to be a conformance issue on the harbor side for bare single file ORAS pushes.

I have a harbor install I can test against here, once I turn my home dev servers machines back on. Am in Texas where we are still having to conserve power due to winter storms for a bit. ❄️

[dioguerra]

@dtrudg When you have something, ping me i can test

[dtrudg]

@dioguerra - I just sat down to implement the {} config potential fix, but tried this against harbor Version v2.1.3-b6de84c5 using the master branch of Singularity and don't see any issue:

08:41 AM $ export SINGULARITY_DOCKER_USERNAME=XXXXXX
08:41 AM $ export SINGULARITY_DOCKER_PASSWORD=XXXXXX
08:41 AM $ singularity push alpine_latest.sif oras://harbor.k8s.randomroad.net/dtrudg/dtrudg:latest
INFO:    Upload complete

Do you have any issue if you compile singualrity from the master branch? There has been an update to the oras package we are using... now at 0.10.0

[dioguerra]

Hey @dtrudg, hope everything is well with you... News where not kind with Texas. :(

As far as i know, with plain Harbor this is a non issue. It only happens with Harbor behind OIDC.
I did found https://storage.googleapis.com/harbor-conformance-test/report.html on their docs. But still didnt validate it myself.
Im in the process of upgrading Harbor to version 2.2 so i can try this again.

Also, @dcsouthwick found another problem that i also need to investigate.

[dcsouthwick]

I'll jump in here, as it was me pestering @dioguerra in the first place trying to push .sif 😄

Indeed using the above oras work-around, I can manually push "small" sif images to OIDC harbor - however once we got that working, I found pushing a larger image (for example, library://dcsouthwick/iotools/root-f31) to OIDC harbor results in timeout -
Error: failed commit on ref "unknown-sha256:9e951d68cadb2d703aa62784714f29ec5b0d255abf94fd481bf4ed4516311a53": unexpected status: 504 Gateway Timeout

I'm guessing either the oras module singularity uses, or the default configuration for harbor isn't happy with large single-layer images. Will try and investigate more...

[dtrudg]

Hi all... okay, I'll put this on the 'more investigation required' pile. The oras module is known working for me on multi GB images (albeit to other registries), so the 504 is likely the harbor install.

Sorry I missed that this config issue only happens with OIDC... that's really odd that the authentication mechanism would impact behavior with / without a config. I'll need to setup something... or we may already have a test instance hooked up to OIDC that I can find somewhere.

[dtrudg]

Okay... got something now with harbor authenticating with OIDC / keycloak...

04:58 PM $ singularity push lolcow_latest.sif oras://xxx.xxx.xxx.xx/dt/lolcow:latest
FATAL:   Unable to push image to oci registry: unable to push: failed commit on ref "unknown-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855": unexpected status: 500 Internal Server Error

Will be able to try out the {} config instead of /dev/null soon.

[dioguerra]

Hey @dtrudg, I have news!

This image being uploaded was actually failing because of issues on our side (related to connection drops)
I managed to upload the image, and all works well, but now i'm getting MIME type errors when unpacking with singularity.

• Upload works ok

Reproduce with:

oras push registry-staging.foo.bar/dtomasgu/root-f31:latest --manifest-config config.json root-f31_latest.sif:appliciation/vnd.sylabs.sif.layer.tar &

With config.json:

{
    "mediaType": "appliciation/vnd.sylabs.sif.layer.tar",
    "digest": "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "size": 0
}

• Download

singularity pull docker://registry-staging.foo.bar/dtomasgu/root-f31:vnd
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob ff6d6448d232 [=====>--------------------------------] 96.5MiB / 584.3MiB
FATAL:   While making image from oci registry: error fetching image to cache: while building 
SIF from layers: conveyor failed to get: Error initializing source 
oci:/home/dtomasgu/.singularity/cache/blob:d9da75af027a7231a89b16c834fb24018ec1000aa0e3c082811d1b4fd04682c3: Error creating an updated image manifest: Error preparing updated manifest, layer "sha256:af9330580191eed899d45eaae96a1d68b677627fd57e16067127942e96f7237e": 
unsupported MIME type for compression:appliciation/vnd.sylabs.sif.layer.tar

I think we can try the work around? Hope this helps!
Cheers!

[dioguerra]

Hey @dtrudg,

Sorry to be pesky. Should i have a go at this?

[dtrudg]

Hi @dioguerra - if you'd like to have a go, please do. Sorry - it's on my to-do list and I will get to it before 3.8 but can't guarantee when that'd be.

[dcsouthwick]

I can add a small update to this:

Pushing can be done with command line oras cli binary to Harbor 2.2 behind OIDC login, without the above manual manifest include using a different OCI type:
Examples:

oras login -u <username> -p <OIDC CLI TOKEN> registry.foo.bar
Login Succeeded

oras push registry.foo.bar/iotools/root-f31:latest root-f31.sif:application/vnd.sylabs.sif.layer.v1.sif
Uploading 9e951d68cadb root-f31.sif
Pushed registry.foo.bar/iotools/root-f31:latest
Digest: sha256:3da44d1209efef7b15a963ab211815c330d641076c98b26c28aa7a8f9d92fd51

Pulling natively:

singularity pull oras://registry.foo.bar/iotools/root-f31:latest
INFO:    Downloading oras image

Notes:

Pushing with ORAS requires the OCI SIF metatype or the pushed image will not be pull-able.
:application/vnd.sylabs.sif.layer.v1.sif

Pulling with singularity requires the tag/hash. Singularity cannot resolve "latest" automatically for some reason from oras...

Pushing with singularity currently fails for unknown reasons, despite its reliance on ORAS natively.

[dtrudg]

@dcsouthwick - thanks. Sounds like something may have changed with Harbor 2.2 here? Singularity sets the type on the SIF application/vnd.sylabs.sif.layer.v1.sif ...

https://github.com/hpcng/singularity/blob/4958db5e421cae0f6ebff30216f6341005256e4e/internal/pkg/client/oras/oras.go#L189

But also provides a config which has a type, but is empty...

https://github.com/hpcng/singularity/blob/4958db5e421cae0f6ebff30216f6341005256e4e/internal/pkg/client/oras/oras.go#L180

This is working on the other registries we've tried, but harbor doesn't like it. I'm pretty sure the safest fix will be to provide some JSON for the config here. I have this on my list for today now, as we have a Harbor 2.2 instance with OIDC auth handy, and I'm working with it otherwise as well.

[dtrudg]

If you can give #5951 a try that'd be appreciated. I have verified against harbor 2.2. It's a very simple retry-based workaround, so a little ugly... but I don't see another way short of the complexity of identifying our destination registry as one in an 'incompatible' list for the config media type. I don't believe we should invest in that, given the registry supporting oras should accept the custom config media type.

[mahendrapaipuri]

I have noticed similar behaviour as posted by @dcsouthwick on GitLab container registry. I could push the image into registry using singularity push my-image.sif oras://registry.gitlab.com/my-image:latest but couldnt pull it.

Using oras push with :application/vnd.sylabs.sif.layer.v1.sif works and image can be pulled back.

Little issue I found on GitLab registry is that the image doesnt have creation time in the metadata. So, the registry UI says the image is published "just now" all the time.

Does singularity includes creation time in the image metadata? If not, is there a way that I can pass this such a way that GitLab UI can parse.

[dtrudg]

@mahendrapaipuri - please could you try with the referenced PR and see if it works to push/pull now to GitLab container registry? Anything that implements the ORAS specification properly should work. Further fixes are a case of needing workaround for registries that don't support ORAS fully for arbitrary types.

The metadata that the GitLab container registry is showing about images is part of the OCI container image formats. Singularity's SIF images that you are pushing are not an OCI container image, so they do not have this metadata at all. The ORAS specification doesn't have a generic metadata format, and the OCI registry only sees a single file blob... not an OCI container image with the metadata it can understand.

[dtrudg]

@mahendrapaipuri - I found a discussion around this here btw... https://gitlab.com/gitlab-org/gitlab/-/issues/324709

[mahendrapaipuri]

@dtrudg Actually, I just finished testing and YES, it works now on GitLab registry now. I could push and pull the image using singularity push and singularity pull. I dont know about harbor though.

Yes, thats what I understood about the metadata of the singularity's SIF image.

@mahendrapaipuri - I found a discussion around this here btw... https://gitlab.com/gitlab-org/gitlab/-/issues/324709

That discussion actually brought me here. That issue still exists though even with the referenced PR. I will check with ORAS, if I can add Creation attribute to their generic image metadata, it should solve the issue on GitLab.

[dcsouthwick]

Hi @dtrudg,

I pulled your patch into a build from master, and I can confirm that worked for me with Harbor! Both push & pull working as expected so far!

[dtrudg]

Thanks @dcsouthwick