Nydusd accesses Dragonfly in a mirror registry fashion

[changweige]

At present, nydusd can only access dragonfly as an HTTP proxy. It has some cons:

1. For HTTPS registry, must provide a copy of CA and private key to dragonfly dfdaemon.
2. Hardly support multiple mirrors
3. An extra HTTP connect is needed (a more round trip)

My proposal is to support accessing dragonfly as a pull-through cache.
To achieve it:

1. add more items to nydusd configuration file:
2. support filling HTTP header according to a configuration JSON file.

{
  "device": {
    "backend": {
      "type": "registry",
      "config": {
        "scheme": "http",
        "repo": "nydus",
        "host": "127.0.0.1:8000",
        "mirrors": [{"scheme":"https","host": "dragonfrly.dfdaemon.org:65001", "username": "itsme", "password": "PASS", "header":"X-dragonfaly-header"}]
      }
    },

Furthermore, we can try to configure multiple registries to nydusd when it works globally in snapshotter shared mode

[imeoer]

As per the offline discussion with @changweige and @zyfjeff , nydus can integrate dragonfly in three ways, proxy mode (2) and mirror mode.

1. Proxy Mode

Proxy mode integration documentation: https://d7y.io/zh/docs/setup/runtime/containerd/proxy/

For https image registry, we need to configure the https decryption certificate for d7y so that it can cache the blob data.

2. Proxy Mode with HTTP scheme

This is a workaround without configuring the certificate:

{
  "device": {
    "backend": {
      "type": "registry",
      "config": {
        "scheme": "https",
        "repo": "library/nginx",
        "host": "source-registry.com",
        "proxy": [
          {
            "url": "http://127.0.0.1:65001",
            "fallback": true,
            // use scheme `http` to request source registry with proxy, and fallback to `https` if proxy is unhealthy.
            "use_http": true
          }
        ]
      }
    }
  },
  ...
}

• For d7y dfdaemon, enable useHTTPS option, when d7y dfdaemon intercepts the http request, it will automatically replace the http with https to request the source image registry center, thus eliminating the need to decrypt https.

• For nydusd, when the option "use_http": true is enabled, need to use scheme http to request source registry with proxy, and fallback to https if proxy is unhealthy, this feature still needs to be implemented.

3. Mirror Mode

Mirror mode integration documentation: https://d7y.io/zh/docs/setup/runtime/containerd/mirror

Currently nydus can only support single mirror (d7y fixed configuration mirror url), but not support dynamic multi-mirrors, a feasible design to support multi-mirrors is as follows:

For example, the mirror requirement is:

• ctr pull source-registry1.com will be mirrored to 127.0.0.1:65001
• ctr pull source-registry2.com will be mirrored to 127.0.0.1:65001

Then when ctr pull source-registry1.com, the nydus snapshotter should generate the config.json file for nydusd like this:

{
  "device": {
    "backend": {
      "type": "registry",
      "config": {
        "scheme": "https",
        "repo": "library/nginx",
        "host": "source-registry1.com",
        "skip_verify": true,
        "mirrors": [
          {
            "scheme": "http",
            "host": "127.0.0.1:65001",
            "headers": {
              "X-Dragonfly-Registry": "https://source-registry1.com"
            }
          }
        ]
      }
    }
  },
  ...
}

Similarly, when ctr pull source-registry2.com, the generated nydusd config.json is replaced with "X-Dragonfly-Registry": "https://source-registry2.com"

This means that the nydus snapshotter should have a configuration of (just a possible config in toml format):

<snapshotter startup arguments... >

<nydusd configuration template... >

[[resolver.host. "source-registry1.com".mirrors]]
host = "http://127.0.0.1:65001"
headers = "X-Dragonfly-Registry": "https://source-registry1.com"
insecure = true

[[resolver.host. "source-registry2.com".mirrors]]
host = "http://127.0.0.1:65001"
headers = "X-Dragonfly-Registry": "https://source-registry2.com"
insecure = true

@zyfjeff is implementing feature 2.
#723

@sctb512 is implementing feature 3. (nydusd side)
#723

@raoxiang1996 is implementing feature 3. (snapshotter side)
containerd/nydus-snapshotter#176

[changweige]

@zyfjeff Is there any progress on this feature, I think it is instrumental when integrating with Dragonlfy.

[zyfjeff]

@zyfjeff Is there any progress on this feature, I think it is instrumental when integrating with Dragonlfy.

Sorry, I haven't implemented it yet, currently, nydus communicates with dragonfly via https, and it just works.

[changweige]

@zyfjeff Is there any progress on this feature, I think it is instrumental when integrating with Dragonlfy.

Sorry, I haven't implemented it yet, currently, nydus communicates with dragonfly via https, and it just works.

Thanks for feedback ~

[changweige]

@imeoer

For nydusd, when the option "use_http": true is enabled, need to use scheme http to request source registry with proxy, and fallback to https if proxy is unhealthy, this feature still needs to be implemented.

I guess you are suggesting using the original HTTP scheme to access the original registry rather than only https?
Registry host can also be configured as HTTP

[imeoer]

Refer to containerd/nydus-snapshotter#178

[changweige]

Let's close this issue since related PR has been merged into mainline