Nydus-Snapshotter Auth Error

[sharique-masood]

Hi Team,

I’m currently integrating Nydus with Dragonfly in a Kubernetes cluster installed via Helm charts. While Dragonfly works seamlessly without Nydus, I’m encountering authentication issues when the two are integrated. The nydusd is running as a daemonset.

image: ghcr.io/containerd/nydus-snapshotter
tag: v0.9.0

Helm Chart:
https://github.com/dragonflyoss/helm-charts/blob/main/INSTALL.md

Nydus-Config:

{
"device": {
"backend": {
"type": "registry",
"config": {
"mirrors": [{"auth_through":false,"headers":{"X-Dragonfly-Registry":"http://harbor-registry.com/"},"host":"http://127.0.0.1:65001/","ping_url":"http://127.0.0.1:40901/server/ping"}],
"scheme": "http",
"skip_verify": true,
"timeout": 5,
"connect_timeout": 5,
"retry_limit": 5
}
},
"cache": {
"type": "blobcache",
"config": {
"work_dir": "/var/lib/nydus/cache/"
}
}
},
"mode": "direct",
"digest_validate": false,
"iostats_files": false,
"enable_xattr": true,
"fs_prefetch": {
"enable": true,
"threads_count": 8,
"merging_size": 1048576,
"prefetch_all": true
}
}

ContainerD Config:

[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://mirror.gcr.io/", "https://registry-1.docker.io/"]
[proxy_plugins]
[proxy_plugins.nydus]
type = "snapshot"
address = "/run/containerd-nydus/containerd-nydus-grpc.sock"
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."harbor-registry.com"]
endpoint = ["http://127.0.0.1:65001/","http://harbor-registry.com/"]
[plugins."io.containerd.grpc.v1.cri".registry.configs."127.0.0.1:65001".auth]
username = "username"
password = "mypassword"
auth = ""
identitytoken = ""

Error Message:
Nydus log:

Read from backend failed: Registry(Request(ErrorWithMsg("401 Unauthorized"))), retry count 10
[2023-11-15 12:08:22.991641 +00:00] INFO [storage/src/cache/cachedfile.rs:74] temporarily failed to get blob.meta, I/O error (os error 5)
[2023-11-15 12:08:23.003567 +00:00] ERROR [error/src/error.rs:21] Error: "failed to read metadata from backend(compressor is none), Registry(Request(ErrorWithMsg("401 Unauthorized")))" at storage/src/meta/mod.rs:659 note: enable RUST_BACKTRACE=1 env to display a backtrace

Pod Event:

Error: failed to create containerd container: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount28632954: read /var/lib/containerd/tmpmounts/containerd-mount28632954/etc/passwd: invalid argument

I have also tried doing the auth via the kubernetes service account by following the below doc:
https://github.com/containerd/nydus-snapshotter/blob/main/docs/configure_nydus.md#use-serviceaccount

Established a service account, configured registry authentication, and seamlessly applied the authentication in a DaemonSet through imagePullSecrets.

[sharique-masood]

This is now resolved; the auth section is functioning with the correct credentials.