containerd can't use private repositity pause to create pod #813

hmtai · 2018-06-11T13:21:06Z

When I install kubernetes 1.10 with containerd and I config the kubelet.service --pod-infra-container-image=registry/mdelder/pause:3.0.After the whole cluster running,I found that containerd pull the k8s.gcr.io/pause:3.1to create pods.How I config to use my private repositity pause image?Did I make the wrong configuration？

The text was updated successfully, but these errors were encountered:

Random-Liu · 2018-06-11T17:47:58Z

You shouldn't configure kubelet flag --pod-infra-container-image. That flag is for dockershim. You should use sandbox_image in containerd config instead (see https://github.com/containerd/cri/blob/master/docs/config.md).
containerd doesn't support pulling pause image from private registry which requires authentication. Does your registry/mdelder registry require authentication?

hmtai · 2018-06-12T01:21:00Z

@Random-Liu Yes, my registry requires authentication,and this is my configuration for containerd:

[plugins]
  [plugins.cgroups]
    no_prometheus = false
  [plugins.cri]
    stream_server_address = ""
    stream_server_port = "10010"
    enable_selinux = false
    sandbox_image = "{{ k8s_pause_image }}"
    stats_collect_period = 10
    systemd_cgroup = false
    [plugins.cri.containerd]
      snapshotter = "overlayfs"
      [plugins.cri.containerd.default_runtime]
        runtime_type = "io.containerd.runtime.v1.linux"

sandbox_imageis my private registry pause image which I use ctr cri loadto load into contaienrd.But when the pod running,I inspect it,it use k8s.gcr.io/pause:3.1image.

crictl inspectp 97f9718d71e1e| grep info -A 15
  "info": {
    "pid": 2953,
    "processStatus": "running",
    "netNamespaceClosed": false,
    "image": "k8s.gcr.io/pause:3.1",
    "snapshotKey": "97f9718d71e1ea814985d356619bb1834ad0cb9bd016d5dd6c316c0ce4cd0839",
    "snapshotter": "overlayfs",
    "runtime": {
      "runtimeType": "io.containerd.runtime.v1.linux",
      "runtimeEngine": "",
      "runtimeRoot": ""
    },
    "config": {
      "metadata": {
        "name": "k8s-master-10.10.24.111",
        "uid": "000737af6f45ed5f4e62c3865f41391a",

Random-Liu · 2018-06-12T02:34:45Z

@hmtai Hm, Let me look into it tonight.

Random-Liu · 2018-06-12T20:43:49Z

I tried changing sandbox image to nginx, and it works for me.

$ cat /etc/containerd/config.toml 
disabled_plugins = ["restart"]
[plugins.linux]
  shim_debug = true
[plugins.cri]
  sandbox_image = "nginx"
$ crictl inspectp e | grep info -A 15
  "info": {
    "pid": 0,
    "processStatus": "deleted",
    "netNamespaceClosed": false,
    "image": "docker.io/library/nginx:latest",
    "snapshotKey": "e50a2079080a0f168e38dc93fd530c70d95adc0b68b1e826665c2ebd58f7f74d",
    "snapshotter": "overlayfs",
    "runtime": {
      "runtimeType": "io.containerd.runtime.v1.linux",
      "runtimeEngine": "",
      "runtimeRoot": ""
    },
    "config": {
      "metadata": {
        "name": "nginx-sandbox",
        "uid": "hdishd83djaidwnduwk28bcsb",

Can you show me your final containerd config on that node? (Without the template string).

hmtai · 2018-06-13T03:05:35Z

@Random-Liu Thanks for you help.I want to use /etc/containerd/config.toml to change default config,but it seems that the config.toml didn't work.Any other steps that I missed?

install containerd

tar -C / -zxf cri-containerd-1.1.0.linux-amd64.tar.gz
systemctl daemon-reload
systemctl start containerd

create the config.toml

vim /etc/containerd/config.toml

Random-Liu · 2018-06-13T03:07:59Z

Can you check 1) Whether your config is our at /etc/containerd/config.toml 2) containers log. The loaded log should be logged. You can check whether the config is correct there. 3) crictl info to check the config in the output.

…

On Tue, Jun 12, 2018 at 7:12 PM huimin ***@***.***> wrote: Thanks a lot :).I follow your steps and hard code in the config.toml but containerd use the old image. root = "/var/lib/containerd" state = "/run/containerd" oom_score = 0 [grpc] address = "/run/containerd/containerd. sock" uid = 0 gid = 0 max_recv_message_size = 16777216 max_send_message_size = 16777216 [debug] address = "" uid = 0 gid = 0 level = "" [metrics] address = "" grpc_histogram = false [cgroup] path = "" [plugins] [plugins.cgroups] no_prometheus = false [plugins.cri] stream_server_address = "" stream_server_port = "10010" enable_selinux = false sandbox_image = "registry.ng.bluemix.net/mdelder/pause:3.0" stats_collect_period = 10 systemd_cgroup = false [plugins.cri.containerd] snapshotter = "overlayfs" [plugins.cri.containerd.default_runtime] runtime_type = "io.containerd.runtime.v1.linux" runtime_engine = "" runtime_root = "" [plugins.cri.containerd.untrusted_workload_runtime] runtime_type = "" runtime_engine = "" runtime_root = "" [plugins.cri.cni] bin_dir = "/opt/cni/bin" conf_dir = "/etc/cni/net.d" [plugins.cri.registry] [plugins.cri.registry.mirrors] [plugins.cri.registry.mirrors."docker.io"] endpoint = ["https://registry-1.docker.io"] [plugins.diff-service] default = ["walking"] [plugins.linux] shim = "containerd-shim" runtime = "runc" runtime_root = "" no_shim = false shim_debug = false [plugins.scheduler] pause_threshold = 0.02 deletion_threshold = 0 mutation_threshold = 100 schedule_delay = "0s" startup_delay = "100ms" crictl inspectp 52e7efc8fd7a8 | grep info -A 10 "info": { "pid": 16512, "processStatus": "running", "netNamespaceClosed": false, "image": "k8s.gcr.io/pause:3.1", "snapshotKey": "52e7efc8fd7a813934b0aa7763dc1d79352f51a6c43c6c7a09f4caa320bb8239", "snapshotter": "overlayfs", "runtime": { "runtimeType": "io.containerd.runtime.v1.linux", "runtimeEngine": "", "runtimeRoot": "" — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#813 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFjVu8r7obDWIN_vIaIXT2Z4Hiq6LQRlks5t8HUEgaJpZM4UirvA> .

Random-Liu · 2018-06-13T03:08:51Z

Sorry, I mean containerd log.

…

On Tue, Jun 12, 2018 at 8:05 PM huimin ***@***.***> wrote: @Random-Liu <https://github.com/Random-Liu> Thanks for you help.I want to use /etc/containerd/config.toml to change default config,but it seems that the config.toml didn't work.Any other steps that I missed? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#813 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFjVu055CeTBJGtuQoTim2n2-ohi-B4oks5t8IGAgaJpZM4UirvA> .

hmtai · 2018-06-13T03:19:33Z

@Random-Liu Thanks,when I systemctl restart containerd it works well.

root@kvm-016081:/etc/containerd# crictl inspectp 2f037b9a2bf1d71ed68b851200fe61e77a94a612e7ee3fcdc16512b77a208a92 | grep info -A 10
  "info": {
    "pid": 0,
    "processStatus": "deleted",
    "netNamespaceClosed": false,
    "image": "registry.ng.bluemix.net/mdelder/pause:3.0",
    "snapshotKey": "2f037b9a2bf1d71ed68b851200fe61e77a94a612e7ee3fcdc16512b77a208a92",
    "snapshotter": "overlayfs",
    "runtime": {
      "runtimeType": "io.containerd.runtime.v1.linux",
      "runtimeEngine": "",
      "runtimeRoot": ""

Random-Liu · 2018-06-13T04:13:58Z

Hm? Why it works now?

…

On Tue, Jun 12, 2018 at 8:19 PM huimin ***@***.***> wrote: Closed #813 <#813>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#813 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFjVu7GSCuxpyS7EutVzxoI_GmvAPMHVks5t8ITQgaJpZM4UirvA> .

hmtai · 2018-06-13T05:26:47Z

@Random-Liu When I change the /etc/containerd/config.toml ,it seems I should restart containerd that containerd can use my new config.toml.

mikebrow · 2018-06-13T15:39:15Z

@Random-Liu that ole do we support live config updates question :-)

Random-Liu added the kind/question label Jun 11, 2018

hmtai closed this as completed Jun 13, 2018

Cynerva mentioned this issue Aug 28, 2019

reconfigure/restart kubelet on registry change charmed-kubernetes/charm-kubernetes-worker#29

Merged

SataQiu mentioned this issue Jun 15, 2022

--pod-infra-container-image does not work in 1.24.1 kubernetes/kubernetes#110594

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

containerd can't use private repositity pause to create pod #813

containerd can't use private repositity pause to create pod #813

hmtai commented Jun 11, 2018 •

edited

Loading

Random-Liu commented Jun 11, 2018 •

edited

Loading

hmtai commented Jun 12, 2018 •

edited

Loading

Random-Liu commented Jun 12, 2018 •

edited

Loading

Random-Liu commented Jun 12, 2018 •

edited

Loading

hmtai commented Jun 13, 2018 •

edited

Loading

Random-Liu commented Jun 13, 2018 via email

Random-Liu commented Jun 13, 2018 via email

hmtai commented Jun 13, 2018 •

edited

Loading

Random-Liu commented Jun 13, 2018 via email

hmtai commented Jun 13, 2018

mikebrow commented Jun 13, 2018

containerd can't use private repositity pause to create pod #813

containerd can't use private repositity pause to create pod #813

Comments

hmtai commented Jun 11, 2018 • edited Loading

Random-Liu commented Jun 11, 2018 • edited Loading

hmtai commented Jun 12, 2018 • edited Loading

Random-Liu commented Jun 12, 2018 • edited Loading

Random-Liu commented Jun 12, 2018 • edited Loading

hmtai commented Jun 13, 2018 • edited Loading

Random-Liu commented Jun 13, 2018 via email

Random-Liu commented Jun 13, 2018 via email

hmtai commented Jun 13, 2018 • edited Loading

Random-Liu commented Jun 13, 2018 via email

hmtai commented Jun 13, 2018

mikebrow commented Jun 13, 2018

hmtai commented Jun 11, 2018 •

edited

Loading

Random-Liu commented Jun 11, 2018 •

edited

Loading

hmtai commented Jun 12, 2018 •

edited

Loading

Random-Liu commented Jun 12, 2018 •

edited

Loading

Random-Liu commented Jun 12, 2018 •

edited

Loading

hmtai commented Jun 13, 2018 •

edited

Loading

hmtai commented Jun 13, 2018 •

edited

Loading