autopilot update does not import new images if k0s version remains the same

[cbodonnell]

Before creating an issue, make sure you've checked the following:

• You are running the latest released version of k0s
• Make sure you've searched for existing issues, both open and closed
• Make sure you've searched for PRs too, a fix might've been merged already
• You're looking at docs for the released version, "main" branch docs are usually ahead of released versions.

Platform

Linux 6.5.0-1016-gcp #16~22.04.1-Ubuntu SMP Sat Mar  9 00:58:37 UTC 2024 x86_64 GNU/Linux
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Version

v1.29.3+k0s.0

Sysinfo

`k0s sysinfo`

Machine ID: "eb37358b11ef42b9066501fe5d801ae439fb533d58d35002b876896ecddff908" (from machine) (pass)
Total memory: 3.8 GiB (pass)
Disk space available for /var/lib/k0s: 179.1 GiB (pass)
Name resolution: localhost: [127.0.0.1] (pass)
Operating system: Linux (pass)
  Linux kernel release: 6.5.0-1016-gcp (pass)
  Max. file descriptors per process: current: 1048576 / max: 1048576 (pass)
  AppArmor: active (pass)
  Executable in PATH: modprobe: /usr/sbin/modprobe (pass)
  Executable in PATH: mount: /usr/bin/mount (pass)
  Executable in PATH: umount: /usr/bin/umount (pass)
  /proc file system: mounted (0x9fa0) (pass)
  Control Groups: version 2 (pass)
    cgroup controller "cpu": available (is a listed root controller) (pass)
    cgroup controller "cpuacct": available (via cpu in version 2) (pass)
    cgroup controller "cpuset": available (is a listed root controller) (pass)
    cgroup controller "memory": available (is a listed root controller) (pass)
    cgroup controller "devices": available (device filters attachable) (pass)
    cgroup controller "freezer": available (cgroup.freeze exists) (pass)
    cgroup controller "pids": available (is a listed root controller) (pass)
    cgroup controller "hugetlb": available (is a listed root controller) (pass)
    cgroup controller "blkio": available (via io in version 2) (pass)
  CONFIG_CGROUPS: Control Group support: built-in (pass)
    CONFIG_CGROUP_FREEZER: Freezer cgroup subsystem: built-in (pass)
    CONFIG_CGROUP_PIDS: PIDs cgroup subsystem: built-in (pass)
    CONFIG_CGROUP_DEVICE: Device controller for cgroups: built-in (pass)
    CONFIG_CPUSETS: Cpuset support: built-in (pass)
    CONFIG_CGROUP_CPUACCT: Simple CPU accounting cgroup subsystem: built-in (pass)
    CONFIG_MEMCG: Memory Resource Controller for Control Groups: built-in (pass)
    CONFIG_CGROUP_HUGETLB: HugeTLB Resource Controller for Control Groups: built-in (pass)
    CONFIG_CGROUP_SCHED: Group CPU scheduler: built-in (pass)
      CONFIG_FAIR_GROUP_SCHED: Group scheduling for SCHED_OTHER: built-in (pass)
        CONFIG_CFS_BANDWIDTH: CPU bandwidth provisioning for FAIR_GROUP_SCHED: built-in (pass)
    CONFIG_BLK_CGROUP: Block IO controller: built-in (pass)
  CONFIG_NAMESPACES: Namespaces support: built-in (pass)
    CONFIG_UTS_NS: UTS namespace: built-in (pass)
    CONFIG_IPC_NS: IPC namespace: built-in (pass)
    CONFIG_PID_NS: PID namespace: built-in (pass)
    CONFIG_NET_NS: Network namespace: built-in (pass)
  CONFIG_NET: Networking support: built-in (pass)
    CONFIG_INET: TCP/IP networking: built-in (pass)
      CONFIG_IPV6: The IPv6 protocol: built-in (pass)
    CONFIG_NETFILTER: Network packet filtering framework (Netfilter): built-in (pass)
      CONFIG_NETFILTER_ADVANCED: Advanced netfilter configuration: built-in (pass)
      CONFIG_NF_CONNTRACK: Netfilter connection tracking support: built-in (pass)
      CONFIG_NETFILTER_XTABLES: Netfilter Xtables support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_REDIRECT: REDIRECT target support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_COMMENT: "comment" match support: module (pass)
        CONFIG_NETFILTER_XT_MARK: nfmark target and match support: module (pass)
        CONFIG_NETFILTER_XT_SET: set target and match support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_MASQUERADE: MASQUERADE target support: module (pass)
        CONFIG_NETFILTER_XT_NAT: "SNAT and DNAT" targets support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: "addrtype" address type match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_CONNTRACK: "conntrack" connection tracking match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_MULTIPORT: "multiport" Multiple port match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_RECENT: "recent" match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_STATISTIC: "statistic" match support: module (pass)
      CONFIG_NETFILTER_NETLINK: built-in (pass)
      CONFIG_NF_NAT: module (pass)
      CONFIG_IP_SET: IP set support: module (pass)
        CONFIG_IP_SET_HASH_IP: hash:ip set support: module (pass)
        CONFIG_IP_SET_HASH_NET: hash:net set support: module (pass)
      CONFIG_IP_VS: IP virtual server support: module (pass)
        CONFIG_IP_VS_NFCT: Netfilter connection tracking: built-in (pass)
        CONFIG_IP_VS_SH: Source hashing scheduling: module (pass)
        CONFIG_IP_VS_RR: Round-robin scheduling: module (pass)
        CONFIG_IP_VS_WRR: Weighted round-robin scheduling: module (pass)
      CONFIG_NF_CONNTRACK_IPV4: IPv4 connetion tracking support (required for NAT): unknown (warning)
      CONFIG_NF_REJECT_IPV4: IPv4 packet rejection: module (pass)
      CONFIG_NF_NAT_IPV4: IPv4 NAT: unknown (warning)
      CONFIG_IP_NF_IPTABLES: IP tables support: module (pass)
        CONFIG_IP_NF_FILTER: Packet filtering: module (pass)
          CONFIG_IP_NF_TARGET_REJECT: REJECT target support: module (pass)
        CONFIG_IP_NF_NAT: iptables NAT support: module (pass)
        CONFIG_IP_NF_MANGLE: Packet mangling: module (pass)
      CONFIG_NF_DEFRAG_IPV4: built-in (pass)
      CONFIG_NF_CONNTRACK_IPV6: IPv6 connetion tracking support (required for NAT): unknown (warning)
      CONFIG_NF_NAT_IPV6: IPv6 NAT: unknown (warning)
      CONFIG_IP6_NF_IPTABLES: IP6 tables support: module (pass)
        CONFIG_IP6_NF_FILTER: Packet filtering: module (pass)
        CONFIG_IP6_NF_MANGLE: Packet mangling: module (pass)
        CONFIG_IP6_NF_NAT: ip6tables NAT support: module (pass)
      CONFIG_NF_DEFRAG_IPV6: built-in (pass)
    CONFIG_BRIDGE: 802.1d Ethernet Bridging: module (pass)
      CONFIG_LLC: module (pass)
      CONFIG_STP: module (pass)
  CONFIG_EXT4_FS: The Extended 4 (ext4) filesystem: built-in (pass)
  CONFIG_PROC_FS: /proc file system support: built-in (pass)

What happened?

When performing an autopilot update that maintains the same k0s version, but includes updated application images for helm charts, the new images are not imported unless k0s controller is restarted. As a result, pods for the helm charts will fail to pull the new images unless it's restarted by the user.

Steps to reproduce

1. Do a k0s install in airgap mode
2. Execute an autopilot plan that maintains the same k0s version, but includes a new images bundle (see example below)
3. The plan completes, but the new images are not imported until the user restarts k0s controller

Expected behavior

The new images should be imported before the plan is marked as completed.

Actual behavior

k0s controller is not restarted, so the images are not imported until the user does this themselves.

Screenshots and logs

Autopilot Plan:

apiVersion: autopilot.k0sproject.io/v1beta2
kind: Plan
metadata:
  annotations:
    embedded-cluster.replicated.com/installation-name: "20240418130951"
  creationTimestamp: "2024-04-18T13:10:22Z"
  generation: 1
  name: autopilot
  resourceVersion: "6746"
  uid: fcfdbe7b-0213-4d90-a750-0c3c808e6b3d
spec:
  commands:
  - airgapupdate:
      platforms:
        linux-amd64:
          url: http://127.0.0.1:50000/images/images-amd64-20240418130951.tar
      version: v1.29.3+k0s.0
      workers:
        discovery:
          static:
            nodes:
            - craig-embedded-cluster-test
        limits:
          concurrent: 1
  - k0supdate:
      platforms:
        linux-amd64:
          sha256: 1320c4ac2ff15fc9442c8629739a55bf21e8951b5b244c791aa9c9990280ecce
          url: http://127.0.0.1:50000/bin/k0s-upgrade
      targets:
        controllers:
          discovery:
            static:
              nodes:
              - craig-embedded-cluster-test
          limits:
            concurrent: 1
        workers:
          discovery:
            static: {}
          limits:
            concurrent: 1
      version: v1.29.3+k0s.0
  id: ed80c4c6-c02c-4609-a41c-cfc3318c2b32
  timestamp: now
status:
  commands:
  - airgapupdate:
      workers:
      - lastUpdatedTimestamp: "2024-04-18T13:10:22Z"
        name: craig-embedded-cluster-test
        state: SignalCompleted
    id: 0
    state: Completed
  - id: 1
    k0supdate:
      controllers:
      - lastUpdatedTimestamp: "2024-04-18T13:10:27Z"
        name: craig-embedded-cluster-test
        state: SignalCompleted
    state: Completed
  state: Completed

The updated image bundle is pulled from the source file server and is present in /var/lib/k0s/images. However, running journalctl -u k0scontroller.service shows that images have not been imported after the plan is marked as completed. Upon restarting k0scontroller.service, we then see the imported image logs such as:

Apr 18 13:13:21 craig-embedded-cluster-test k0s[3466926]: time="2024-04-18 13:13:21" level=info msg="Imported image docker.io/bloomberg/goldpinger:3.9.0" component=OCIBundleReconciler
Apr 18 13:13:21 craig-embedded-cluster-test k0s[3466926]: time="2024-04-18 13:13:21" level=info msg="Imported image docker.io/library/busybox:1.36.1" component=OCIBundleReconciler
Apr 18 13:13:21 craig-embedded-cluster-test k0s[3466926]: time="2024-04-18 13:13:21" level=info msg="Imported image docker.io/library/busybox:latest" component=OCIBundleReconciler
...

Additional context

Is this intentional behavior? If so, what would the recommended procedure be for doing an autopilot update that maintains the same k0s version, but depends on new images for Helm charts?

[jnummelin]

The reason for this functionality is that k0s does not do file watch on the image bundles. It merely loads them up during start. The main use case for image bundles has been the "system" images. And those are pretty much involved when updating k0s binary hence we haven't implemented a watch over it.

I don't think having a watch on the bundle dir would be out of question.

[ricardomaraschini]

@jnummelin I have implemented a watcher on #4321 . Would you mind checking if it goes in the right direction and point me to things that I may be not seeing ?