Configuration Conflict in containerd When Deploying Dragonfly

[liuyuxuan0723]

Description

When using Helm to deploy the Dragonfly service in a Kubernetes cluster, the containerd service fails to start and logs the following warning:

failed to load plugin io.containerd.grpc.v1.cri" error="invalid plugin config: `mirrors` cannot be set when `config_path` is provided

Even though this is a warning log, it prevents the containerd service from starting.

Environment

• Kubernetes version: v1.24
• Dragonfly Chart version: v1.1.65
• Containerd version: v1.6.20

Expected Behavior

The containerd service should start and run normally.

Actual Behavior

The containerd service fails to start and logs the following warning:

failed to load plugin io.containerd.grpc.v1.cri" error="invalid plugin config: `mirrors` cannot be set when `config_path` is provided

Containerd Error Log Screenshot

Containerd Configuration

My containerd configuration before installing Dragonfly was as follows:

[plugins.'io.containerd.grpc.v1.cri'.registry]
  [plugins.'io.containerd.grpc.v1.cri'.registry.mirrors]
    [plugins.'io.containerd.grpc.v1.cri'.registry.mirrors.'docker.io']
      endpoint = ['https://mirror.baidubce.com']

After installing Dragonfly, the following configuration was injected, causing containerd to fail and the node to show as NotReady:

[plugins.'io.containerd.grpc.v1.cri'.registry]
  config_path = "/etc/containerd/certs.d"
  [plugins.'io.containerd.grpc.v1.cri'.registry.mirrors]
    [plugins.'io.containerd.grpc.v1.cri'.registry.mirrors.'docker.io']
      endpoint = ['https://mirror.baidubce.com']

Removing the following configuration allows containerd to start normally:

[plugins.'io.containerd.grpc.v1.cri'.registry.mirrors]
  [plugins.'io.containerd.grpc.v1.cri'.registry.mirrors.'docker.io']
    endpoint = ['https://mirror.baidubce.com']

Question

1. Why is the configuration injected into containerd so abruptly?
2. Can I control the configuration injected into containerd? I could not find any configurable options in the values.yaml file.

cc @saltbo

[gaius-qi]

@liuyuxuan0723
Can you provide your entire containerd config before installing, dfinit entire config and dfinit verison?

If you don't know how to get dfinit entire config and dfinit verison, you can give me the helm chart config.

[liuyuxuan0723]

@gaius-qi
the complete containerd configuration before installing is as follows:

version = 2
disabled_plugins = []
imports = []
oom_score = -999
required_plugins = []
root = '/cce/containerd'
state = '/run/containerd'
[debug]
  address = '/run/containerd/debug.sock'
  level = 'info'
[plugins]
  [plugins.'io.containerd.grpc.v1.cri']
    enable_selinux = false
    enable_tls_streaming = false
    max_concurrent_downloads = 10
    sandbox_image = 'registry.baidubce.com/cce-public/pause:3.1'
    stream_server_address = '127.0.0.1'
    stream_server_port = '0'
    [plugins.'io.containerd.grpc.v1.cri'.cni]
      bin_dir = '/opt/cni/bin'
      conf_dir = '/etc/cni/net.d'
      conf_template = ''
    [plugins.'io.containerd.grpc.v1.cri'.containerd]
      default_runtime_name = 'runc'
      [plugins.'io.containerd.grpc.v1.cri'.containerd.runtimes]
        [plugins.'io.containerd.grpc.v1.cri'.containerd.runtimes.runc]
          container_annotations = []
          pod_annotations = []
          privileged_without_host_devices = false
          runtime_type = 'io.containerd.runc.v2'
    [plugins.'io.containerd.grpc.v1.cri'.registry]
      [plugins.'io.containerd.grpc.v1.cri'.registry.mirrors]
        [plugins.'io.containerd.grpc.v1.cri'.registry.mirrors.'docker.io']
          endpoint = ['https://mirror.baidubce.com']

The Helm chart configuration is similar to the following:

containerRuntime:
  containerd:
    enable: true
    injectConfigPath: true
    registries:
      - "docker.io"

  initContainer:
    image:
      repository: dragonflyoss/openssl
      tag: latest

scheduler:
  image:
    repository: dragonflyoss/scheduler
    tag: latest
  replicas: 1
  metrics:
    enable: true
  config:
    verbose: true
    pprofPort: 18066
    console: true
  initContainer:
    image:
      repository: dragonflyoss/busybox
      tag: latest

seedPeer:
  image:
    repository: dragonflyoss/dfdaemon
    tag: latest
  replicas: 3
  metrics:
    enable: true
  config:
    verbose: true
    pprofPort: 18066
    console: true
  initContainer:
    image:
      repository: dragonflyoss/busybox
      tag: latest

dfdaemon:
  image:
    repository: dragonflyoss/dfdaemon
    tag: latest
  metrics:
    enable: true
  config:
    verbose: true
    pprofPort: 18066
    console: true
    keepStorage: true
    scheduler:
      manager:
        seedPeer:
          enable: true
  initContainer:
    image:
      repository: dragonflyoss/busybox
      tag: latest

manager:
  enable: true
  image:
    repository: dragonflyoss/manager
    tag: latest
  replicas: 1
  metrics:
    enable: true
  config:
    verbose: true
    pprofPort: 18066
    console: true
  initContainer:
    image:
      repository: dragonflyoss/busybox
      tag: latest

[gaius-qi]

@liuyuxuan0723 I think you need use the rust client, refer to https://d7y.io/docs/next/getting-started/installation/helm-charts/.

[liuyuxuan0723]

@gaius-qi
I tried deploying using the Rust client and enabled dfinit initialization. When using mirror mode to inject the configuration, the containerd configuration is as follows:

[plugins."io.containerd.grpc.v1.cri".registry]
    [plugins."io.containerd.grpc.v1.cri".registry.mirrors.'docker.io']
      endpoint = ['https://mirror.baidubce.com']

[plugins."io.containerd.grpc.v1.cri".registry.mirrors."artifactory-registry.com"]
endpoint = ["http://127.0.0.1:4001", "https://artifactory-registry.com"]

However, when pulling images on the host, it seems that the client is not acting as a proxy. The logs of this task can be viewed in the client, but no cached pieces are found in /var/lib/dragonfly/content.

When dfinit injects the configuration using the config_path mode method, it correctly proxies the download requests. Cached pieces can be found in the /var/lib/dragonfly/content directory. The configuration is as follows:

/etc/containerd/config.toml:

[plugins."io.containerd.grpc.v1.cri".registry]
config_path = "/etc/containerd/certs.d"

/etc/containerd/certs.d/artifactory-registry.com/hosts.toml :

server = "https://artifactory-registry.com"

[host."http://127.0.0.1:4001"]
capabilities = ["pull", "resolve"]

[host."http://127.0.0.1:4001".header]
X-Dragonfly-Registry = "https://artifactory-registry.com"

I have one more question. Does the certs.d configuration need to include the following entry to ensure that images can still be pulled if the client goes down?

[host."https://artifactory-registry.com"]
capabilities = ["pull", "resolve"]