[FEATURE] Container-Optimized OS support

[innobead]

Is your feature request related to a problem? Please describe (👍 if you like this request)

https://cloud.google.com/container-optimized-os/docs This is an evaluation task instead to see how Longhorn can integrate with it.

Describe the solution you'd like

A clear and concise description of what you want to happen

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Limitations https://cloud.google.com/container-optimized-os/docs/concepts/features-and-benefits

[c3y1huang]

According to the document statement:

Container-Optimized OS is the default node OS Image in Kubernetes Engine and other Kubernetes deployments on Google Cloud Platform.

Given that Longhorn is compatible with Google Kubernetes Engine (GKE), we can assume that it should also work on self-launched COS + Kubernetes.

TODO: manually create a COS cluster with Kubernetes and verify Longhorn passes the core testing.

[c3y1huang]

Seems like we've only tested GKE with UBUNTU_CONTAINERD. We've never done any testings with COS_CONTAINERD.

TODO: figure out how if it is possible to get Longhorn dependencies onto COS.

[c3y1huang]

TODO: figure out how if it is possible to get Longhorn dependencies onto COS.

COS_CONTAINERD doesn't ship with the package manager, and the majority of directories are set to read-only. GKE works around this by cloud-config in user-data and /home/kubernetes/containerized_mounter/rootfs that contains the dependencies and mounts within the chroot environment.

TODO: manually create a COS cluster with Kubernetes and verify Longhorn passes the core testing.

Testing with other orchestrators like K3s is challenging due to the absence of a support method for the environment pre-configuration.

The easiest approach is to use GKE which comes with the COS_CONTAINER + Kubernetes. We could potentially use the same rootfs for installing Longhorn dependencies and mounting for the data path. However, since the user-data is predefined in GKE, we can't directly utilize the cloud-config. Instead, a possible workaround is to use a daemonset that runs chroot and nsenter for Longhorn dependencies setup/installation.

ref: https://github.com/kubernetes/kubernetes/tree/v1.29.2/cluster/gce

[c3y1huang]

As evidence supporting the assumption mentioned above, below is the installed the open-iscsi via a daemonset and confirmed it is running on the host system:

> ps aux | grep iscsid
root       58264  0.0  0.0  25384   292 ?        Ss   06:21   0:00 /sbin/iscsid
root       58265  0.0  0.0  25888  5324 ?        S<Ls 06:21   0:00 /sbin/iscsid

[c3y1huang]

Given that we've done the majority of the work on the Talos support feature, it seems that this feature that only a small effort is needed for this feature. Primarily, providing users with the necessary dependency installation.

Additionally, in GKE, periodic updates are made to topology.kubernetes.io/zone to reflect the actual zone, causing failures of the replica_auto_balance_zone tests when such updates occur. To tackle this, we can modify the tests to refresh the zone labels and maintain the expected zone label simulation throughout each test run.

Below is the full PoC test result using 1.28.7-gke.1026000, COS_CONTAINERD cos-109-17800-66-78, S3 as the backup store.

= 355 passed, 19 skipped, 348 warnings in 53969.99s (14:59:29) =

Below is the core test result 1.28.7-gke.1026000, COS_CONTAINERD cos-109-17800-66-78, NFS as the backup store.

= 64 passed, 310 deselected, 44 warnings in 7587.32s (2:06:27) =

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at:

• Is there a workaround for the issue? If so, where is it documented?
  The workaround is at:

• Does the PR include the explanation for the fix or the feature?

• Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?
  The PR for the YAML change is at: feat(platform): support GKE Container-Optimized OS #8196
  The PR for the chart change is at:

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at

• Which areas/issues this PR might have potential impacts on?
  Area platform
  Issues

• If labeled: require/LEP Has the Longhorn Enhancement Proposal PR submitted?
  The LEP PR is at

• If labeled: area/ui Has the UI issue filed or ready to be merged (including backport-needed/*)?
  The UI issue/PR is at

• If labeled: require/doc Has the necessary document PR submitted or merged (including backport-needed/*)?
  The documentation issue/PR is at doc(1.7.0): support GKE Container-Optimized OS website#884

• If labeled: require/automation-e2e Has the end-to-end test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue (including backport-needed/*)
  The automation skeleton PR is at
  The automation test case PR is at test(integration/manual): support GKE Container-Optimized OS longhorn-tests#1819
  The issue of automation test case implementation is at (please create by the template)

• If labeled: require/automation-engine Has the engine integration test been merged (including backport-needed/*)?
  The engine automation PR is at

• If labeled: require/manual-test-plan Has the manual test plan been documented?
  The updated manual test plan is at test(integration/manual): support GKE Container-Optimized OS longhorn-tests#1819

• If the fix introduces the code for backward compatibility Has a separate issue been filed with the label release/obsolete-compatibility?
  The compatibility issue is filed at

[chriscchien]

Hi @c3y1huang , test case test_replica_auto_balance_node_duplicates_in_multiple_zones failed on master pipeline after longhorn/longhorn-tests#1819, could you take a look on that? thank you.

[c3y1huang]

Hi @c3y1huang , test case test_replica_auto_balance_node_duplicates_in_multiple_zones failed on master pipeline after longhorn/longhorn-tests#1819, could you take a look on that? thank you.

Thank you @chriscchien . Will be fixed in longhorn/longhorn-tests#1849.

[yangchiu]

After applied kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/prerequisite/longhorn-gke-cos-node-agent.yaml on cos_containerd, there are some error messages in the log:

$ kubectl get pods
NAME                                  READY   STATUS    RESTARTS        AGE
longhorn-gke-cos-node-agent-5n9xx     1/1     Running   1 (7h43m ago)   7h44m
longhorn-gke-cos-node-agent-8r7tx     1/1     Running   1 (7h43m ago)   7h44m
longhorn-gke-cos-node-agent-f28vz     1/1     Running   1 (7h43m ago)   7h44m

$ kubectl logs longhorn-gke-cos-node-agent-5n9xx
...
(23/24) Installing: systemd-249.17-150400.8.40.1.x86_64 [.......
Creating group systemd-journal with gid 485.
Creating group systemd-network with gid 484.
Creating user systemd-network (systemd Network Management) with uid 484 and gid 484.
Creating group systemd-timesync with gid 483.
Creating user systemd-timesync (systemd Time Synchronization) with uid 483 and gid 483.
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down
/usr/lib/tmpfiles.d/journal-nocow.conf:26: Failed to resolve specifier: uninitialized /etc detected, skipping
All rules containing unresolvable specifiers will be skipped.
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down
done]
(24/24) Installing: open-iscsi-2.1.9-150500.46.3.1.x86_64 [..
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down

Not sure if there's any issue here, but longhorn can be installed without problem. cc @c3y1huang

[c3y1huang]

After applied kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/prerequisite/longhorn-gke-cos-node-agent.yaml on cos_containerd, there are some error messages in the log:

$ kubectl get pods
NAME                                  READY   STATUS    RESTARTS        AGE
longhorn-gke-cos-node-agent-5n9xx     1/1     Running   1 (7h43m ago)   7h44m
longhorn-gke-cos-node-agent-8r7tx     1/1     Running   1 (7h43m ago)   7h44m
longhorn-gke-cos-node-agent-f28vz     1/1     Running   1 (7h43m ago)   7h44m

$ kubectl logs longhorn-gke-cos-node-agent-5n9xx
...
(23/24) Installing: systemd-249.17-150400.8.40.1.x86_64 [.......
Creating group systemd-journal with gid 485.
Creating group systemd-network with gid 484.
Creating user systemd-network (systemd Network Management) with uid 484 and gid 484.
Creating group systemd-timesync with gid 483.
Creating user systemd-timesync (systemd Time Synchronization) with uid 483 and gid 483.
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down
/usr/lib/tmpfiles.d/journal-nocow.conf:26: Failed to resolve specifier: uninitialized /etc detected, skipping
All rules containing unresolvable specifiers will be skipped.
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down
done]
(24/24) Installing: open-iscsi-2.1.9-150500.46.3.1.x86_64 [..
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down

Not sure if there's any issue here, but longhorn can be installed without problem.

When iSCSId is installed via package managers within a container, the package manager will try to initiate the ISCSi services managed by systemd, which causes problems because the container environment doesn't fully support systemd. Hence, the script manually starts the daemon instead of relying on systemd. Therefore, this kind of error should be safe to ignore in this context.

[yangchiu]

Verified passed on master-head (longhorn-manager f63611c) following the document to setup Longhorn on cos_containerd. Longhorn can be installed without problem, and run core test on it, test cases all passed.

[innobead]

Not sure if there's any issue here, but longhorn can be installed without problem.

When iSCSId is installed via package managers within a container, the package manager will try to initiate the ISCSi services managed by systemd, which causes problems because the container environment doesn't fully support systemd. Hence, the script manually starts the daemon instead of relying on systemd. Therefore, this kind of error should be safe to ignore in this context.

@c3y1huang can we mention this error as an expected warning in the doc? to prevent any confusion or later explanation?

[c3y1huang]

@c3y1huang can we mention this error as an expected warning in the doc? to prevent any confusion or later explanation?

Sure! Doc updated.