Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

k8s: guest-pull: initContainer with shared volume fails #9668

Open
fidencio opened this issue May 19, 2024 · 0 comments
Open

k8s: guest-pull: initContainer with shared volume fails #9668

fidencio opened this issue May 19, 2024 · 0 comments
Labels
area/guest-pull bug Incorrect behaviour needs-review Needs to be assessed by the team.

Comments

@fidencio
Copy link
Member

Another one related to initContainers (see also #9666 and #9664).

Bats output:

# Events:
#   Type     Reason     Age                From               Message
#   ----     ------     ----               ----               -------
#   Normal   Scheduled  90s                default-scheduler  Successfully assigned kata-containers-k8s-tests/initcontainer-shared-volume to 984fee00bd70.jf.intel.com
#   Normal   Pulling    82s                kubelet            Pulling image "quay.io/prometheus/busybox:latest"
#   Normal   Pulled     81s                kubelet            Successfully pulled image "quay.io/prometheus/busybox:latest" in 424ms (424ms including waiting)
#   Normal   Created    81s                kubelet            Created container first
#   Normal   Started    80s                kubelet            Started container first
#   Normal   Pulled     78s                kubelet            Successfully pulled image "quay.io/prometheus/busybox:latest" in 448ms (448ms including waiting)
#   Warning  Failed     77s                kubelet            Error: failed to create containerd task: failed to create shim task: the file sh was not found: unknown
#   Normal   Pulled     76s                kubelet            Successfully pulled image "quay.io/prometheus/busybox:latest" in 421ms (421ms including waiting)
#   Warning  Failed     75s                kubelet            Error: failed to create containerd task: failed to create shim task: failed to mount /run/kata-containers/dc969ac55b612bf095dff70ecdc0f7bfa659fcfc1084c16a8fa8eada56320169/rootfs to /run/kata-containers/last/rootfs, with error: ENOENT: No such file or directory: unknown
#   Normal   Pulled     62s                kubelet            Successfully pulled image "quay.io/prometheus/busybox:latest" in 434ms (434ms including waiting)
#   Normal   Pulling    31s (x4 over 79s)  kubelet            Pulling image "quay.io/prometheus/busybox:latest"
#   Normal   Pulled     31s                kubelet            Successfully pulled image "quay.io/prometheus/busybox:latest" in 437ms (437ms including waiting)
#   Normal   Created    31s (x4 over 78s)  kubelet            Created container last
#   Warning  Failed     31s (x2 over 62s)  kubelet            Error: failed to create containerd task: failed to create shim task: load image bundle
#
# Caused by:
#     No such file or directory (os error 2): unknown
@fidencio fidencio added bug Incorrect behaviour needs-review Needs to be assessed by the team. labels May 19, 2024
fidencio added a commit to fidencio/kata-containers that referenced this issue May 19, 2024
This is another one that is related to initContainers not being properly
handled with the guest image pulling.

Let's skip it for now as we have
kata-containers#9668 to track
it down.

Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
ChengyuZhu6 added a commit to ChengyuZhu6/kata-containers that referenced this issue May 22, 2024
Fix tests:
- k8s-credentials-secrets.bats
- k8s-file-volume.bats
- k8s-nested-configmap-secret.bats
- k8s-projected-volume.bats
- k8s-volume.bats

Fixes: kata-containers#9664 kata-containers#9666 kata-containers#9667 kata-containers#9668

Signed-off-by: ChengyuZhu6 <chengyu.zhu@intel.com>
ChengyuZhu6 added a commit to ChengyuZhu6/kata-containers that referenced this issue May 22, 2024
Revert

Fix tests:
- k8s-credentials-secrets.bats
- k8s-file-volume.bats
- k8s-nested-configmap-secret.bats
- k8s-projected-volume.bats
- k8s-volume.bats
- k8s-shared-volume.bats
- k8s-kill-all-process-in-container.bats
- k8s-sysctls.bats

Fixes: kata-containers#9664 kata-containers#9666 kata-containers#9667 kata-containers#9668

Signed-off-by: ChengyuZhu6 <chengyu.zhu@intel.com>
ChengyuZhu6 added a commit to ChengyuZhu6/kata-containers that referenced this issue May 22, 2024
Revert code logic in 462051b

Let me explain why:

In our previous approach, we implemented guest pull by passing PullImageRequest to the guest.
However, this method  resulted in the loss of specifications essential for running the container,
such as commands specified in YAML, during the CreateContainer stage. To address this,
it is necessary to integrate the OCI specifications and process information from the image’s configuration with the container in guest pull.

The snapshotter method does not care this issue. Nevertheless, a problem arises when two containers in the same pod attempt
to pull the same image, like InitContainer. This is because the system searches for the existing configuration, which resides in the
guest. The configuration, associated with <image name, cid>, is stored in the directory /run/kata-containers/<cid>. Consequently, when
the InitContainer finishes its task and terminates, the directory ceases to exist. As a result, during the creation of the application container,
the OCI spec and process information cannot be merged due to the absence of the expected configuration file.

Fix tests:
- k8s-credentials-secrets.bats
- k8s-file-volume.bats
- k8s-nested-configmap-secret.bats
- k8s-projected-volume.bats
- k8s-volume.bats
- k8s-shared-volume.bats
- k8s-kill-all-process-in-container.bats
- k8s-sysctls.bats

Fixes: kata-containers#9664 kata-containers#9666 kata-containers#9667 kata-containers#9668

Signed-off-by: ChengyuZhu6 <chengyu.zhu@intel.com>
@katacontainersbot katacontainersbot moved this from To do to In progress in Issue backlog May 22, 2024
ChengyuZhu6 added a commit to ChengyuZhu6/kata-containers that referenced this issue May 22, 2024
Let me explain why:

In our previous approach, we implemented guest pull by passing PullImageRequest to the guest.
However, this method  resulted in the loss of specifications essential for running the container,
such as commands specified in YAML, during the CreateContainer stage. To address this,
it is necessary to integrate the OCI specifications and process information from the image’s configuration with the container in guest pull.

The snapshotter method does not care this issue. Nevertheless, a problem arises when two containers in the same pod attempt
to pull the same image, like InitContainer. This is because the system searches for the existing configuration, which resides in the
guest. The configuration, associated with <image name, cid>, is stored in the directory /run/kata-containers/<cid>. Consequently, when
the InitContainer finishes its task and terminates, the directory ceases to exist. As a result, during the creation of the application container,
the OCI spec and process information cannot be merged due to the absence of the expected configuration file.

Fix tests:
- k8s-credentials-secrets.bats
- k8s-file-volume.bats
- k8s-nested-configmap-secret.bats
- k8s-projected-volume.bats
- k8s-volume.bats
- k8s-shared-volume.bats
- k8s-kill-all-process-in-container.bats
- k8s-sysctls.bats

Fixes: kata-containers#9664 kata-containers#9666 kata-containers#9667 kata-containers#9668

Signed-off-by: ChengyuZhu6 <chengyu.zhu@intel.com>
ChengyuZhu6 added a commit to ChengyuZhu6/kata-containers that referenced this issue May 22, 2024
Let me explain why:

In our previous approach, we implemented guest pull by passing PullImageRequest to the guest.
However, this method  resulted in the loss of specifications essential for running the container,
such as commands specified in YAML, during the CreateContainer stage. To address this,
it is necessary to integrate the OCI specifications and process information from the image’s configuration with the container in guest pull.

The snapshotter method does not care this issue. Nevertheless, a problem arises when two containers in the same pod attempt
to pull the same image, like InitContainer. This is because the image service searches for the existing configuration,
which resides in the guest. The configuration, associated with <image name, cid>, is stored in the directory /run/kata-containers/<cid>.
Consequently, when the InitContainer finishes its task and terminates, the directory ceases to exist.
As a result, during the creation of the application container, the OCI spec and process information cannot
be merged due to the absence of the expected configuration file.

Fix tests:
- k8s-credentials-secrets.bats
- k8s-file-volume.bats
- k8s-nested-configmap-secret.bats
- k8s-projected-volume.bats
- k8s-volume.bats
- k8s-shared-volume.bats
- k8s-kill-all-process-in-container.bats
- k8s-sysctls.bats

Fixes: kata-containers#9664 kata-containers#9666 kata-containers#9667 kata-containers#9668

Signed-off-by: ChengyuZhu6 <chengyu.zhu@intel.com>
ChengyuZhu6 added a commit to ChengyuZhu6/kata-containers that referenced this issue May 22, 2024
Revert code logic in 462051b

Let me explain why:

In our previous approach, we implemented guest pull by passing PullImageRequest to the guest.
However, this method  resulted in the loss of specifications essential for running the container,
such as commands specified in YAML, during the CreateContainer stage. To address this,
it is necessary to integrate the OCI specifications and process information from the image’s configuration with the container in guest pull.

The snapshotter method does not care this issue. Nevertheless, a problem arises when two containers in the same pod attempt
to pull the same image, like InitContainer. This is because the image service searches for the existing configuration,
which resides in the guest. The configuration, associated with <image name, cid>, is stored in the directory /run/kata-containers/<cid>.
Consequently, when the InitContainer finishes its task and terminates, the directory ceases to exist.
As a result, during the creation of the application container, the OCI spec and process information cannot
be merged due to the absence of the expected configuration file.

Fix tests:
- k8s-credentials-secrets.bats
- k8s-file-volume.bats
- k8s-nested-configmap-secret.bats
- k8s-projected-volume.bats
- k8s-volume.bats
- k8s-shared-volume.bats
- k8s-kill-all-process-in-container.bats
- k8s-sysctls.bats

Fixes: kata-containers#9664 kata-containers#9666 kata-containers#9667 kata-containers#9668

Signed-off-by: ChengyuZhu6 <chengyu.zhu@intel.com>
ChengyuZhu6 added a commit to ChengyuZhu6/kata-containers that referenced this issue May 22, 2024
Revert code logic in 462051b

Let me explain why:

In our previous approach, we implemented guest pull by passing PullImageRequest to the guest.
However, this method  resulted in the loss of specifications essential for running the container,
such as commands specified in YAML, during the CreateContainer stage. To address this,
it is necessary to integrate the OCI specifications and process information from the image’s configuration with the container in guest pull.

The snapshotter method does not care this issue. Nevertheless, a problem arises when two containers in the same pod attempt
to pull the same image, like InitContainer. This is because the image service searches for the existing configuration,
which resides in the guest. The configuration, associated with <image name, cid>, is stored in the directory /run/kata-containers/<cid>.
Consequently, when the InitContainer finishes its task and terminates, the directory ceases to exist.
As a result, during the creation of the application container, the OCI spec and process information cannot
be merged due to the absence of the expected configuration file.

Fix tests:
- k8s-credentials-secrets.bats
- k8s-file-volume.bats
- k8s-nested-configmap-secret.bats
- k8s-projected-volume.bats
- k8s-volume.bats
- k8s-shared-volume.bats
- k8s-kill-all-process-in-container.bats
- k8s-sysctls.bats

Fixes: kata-containers#9664   kata-containers#9666   kata-containers#9667   kata-containers#9668

Signed-off-by: ChengyuZhu6 <chengyu.zhu@intel.com>
ChengyuZhu6 added a commit to ChengyuZhu6/kata-containers that referenced this issue May 22, 2024
Revert code logic in 462051b

Let me explain why:

In our previous approach, we implemented guest pull by passing PullImageRequest to the guest.
However, this method  resulted in the loss of specifications essential for running the container,
such as commands specified in YAML, during the CreateContainer stage. To address this,
it is necessary to integrate the OCI specifications and process information from the image’s configuration with the container in guest pull.

The snapshotter method does not care this issue. Nevertheless, a problem arises when two containers in the same pod attempt
to pull the same image, like InitContainer. This is because the image service searches for the existing configuration,
which resides in the guest. The configuration, associated with <image name, cid>, is stored in the directory /run/kata-containers/<cid>.
Consequently, when the InitContainer finishes its task and terminates, the directory ceases to exist.
As a result, during the creation of the application container, the OCI spec and process information cannot
be merged due to the absence of the expected configuration file.

Fix tests:
- k8s-credentials-secrets.bats
- k8s-file-volume.bats
- k8s-nested-configmap-secret.bats
- k8s-projected-volume.bats
- k8s-volume.bats
- k8s-shared-volume.bats
- k8s-kill-all-process-in-container.bats
- k8s-sysctls.bats

Fixes: kata-containers#9664
Fixes: kata-containers#9666
Fixes: kata-containers#9667
Fixes: kata-containers#9668

Signed-off-by: ChengyuZhu6 <chengyu.zhu@intel.com>
fidencio pushed a commit to ChengyuZhu6/kata-containers that referenced this issue May 22, 2024
Revert code logic in 462051b

Let me explain why:

In our previous approach, we implemented guest pull by passing PullImageRequest to the guest.
However, this method  resulted in the loss of specifications essential for running the container,
such as commands specified in YAML, during the CreateContainer stage. To address this,
it is necessary to integrate the OCI specifications and process information from the image’s configuration with the container in guest pull.

The snapshotter method does not care this issue. Nevertheless, a problem arises when two containers in the same pod attempt
to pull the same image, like InitContainer. This is because the image service searches for the existing configuration,
which resides in the guest. The configuration, associated with <image name, cid>, is stored in the directory /run/kata-containers/<cid>.
Consequently, when the InitContainer finishes its task and terminates, the directory ceases to exist.
As a result, during the creation of the application container, the OCI spec and process information cannot
be merged due to the absence of the expected configuration file.

Fix tests:
- k8s-credentials-secrets.bats
- k8s-file-volume.bats
- k8s-nested-configmap-secret.bats
- k8s-projected-volume.bats
- k8s-volume.bats
- k8s-shared-volume.bats
- k8s-kill-all-process-in-container.bats
- k8s-sysctls.bats

Fixes: kata-containers#9664
Fixes: kata-containers#9666
Fixes: kata-containers#9667
Fixes: kata-containers#9668

Signed-off-by: ChengyuZhu6 <chengyu.zhu@intel.com>
ChengyuZhu6 added a commit to ChengyuZhu6/kata-containers that referenced this issue May 23, 2024
Revert code logic in 462051b

Let me explain why:

In our previous approach, we implemented guest pull by passing PullImageRequest to the guest.
However, this method  resulted in the loss of specifications essential for running the container,
such as commands specified in YAML, during the CreateContainer stage. To address this,
it is necessary to integrate the OCI specifications and process information from the image’s configuration with the container in guest pull.

The snapshotter method does not care this issue. Nevertheless, a problem arises when two containers in the same pod attempt
to pull the same image, like InitContainer. This is because the image service searches for the existing configuration,
which resides in the guest. The configuration, associated with <image name, cid>, is stored in the directory /run/kata-containers/<cid>.
Consequently, when the InitContainer finishes its task and terminates, the directory ceases to exist.
As a result, during the creation of the application container, the OCI spec and process information cannot
be merged due to the absence of the expected configuration file.

Fix tests:
- k8s-credentials-secrets.bats
- k8s-file-volume.bats
- k8s-nested-configmap-secret.bats
- k8s-projected-volume.bats
- k8s-volume.bats
- k8s-shared-volume.bats
- k8s-kill-all-process-in-container.bats
- k8s-sysctls.bats

Fixes: kata-containers#9664
Fixes: kata-containers#9666
Fixes: kata-containers#9667
Fixes: kata-containers#9668

Signed-off-by: ChengyuZhu6 <chengyu.zhu@intel.com>
ChengyuZhu6 added a commit to ChengyuZhu6/kata-containers that referenced this issue May 23, 2024
Revert code logic in 462051b

Let me explain why:

In our previous approach, we implemented guest pull by passing PullImageRequest to the guest.
However, this method  resulted in the loss of specifications essential for running the container,
such as commands specified in YAML, during the CreateContainer stage. To address this,
it is necessary to integrate the OCI specifications and process information from the image’s configuration with the container in guest pull.

The snapshotter method does not care this issue. Nevertheless, a problem arises when two containers in the same pod attempt
to pull the same image, like InitContainer. This is because the image service searches for the existing configuration,
which resides in the guest. The configuration, associated with <image name, cid>, is stored in the directory /run/kata-containers/<cid>.
Consequently, when the InitContainer finishes its task and terminates, the directory ceases to exist.
As a result, during the creation of the application container, the OCI spec and process information cannot
be merged due to the absence of the expected configuration file.

Fix tests:
- k8s-credentials-secrets.bats
- k8s-file-volume.bats
- k8s-nested-configmap-secret.bats
- k8s-projected-volume.bats
- k8s-volume.bats
- k8s-shared-volume.bats
- k8s-sysctls.bats

Fixes: kata-containers#9666
Fixes: kata-containers#9667
Fixes: kata-containers#9668

Signed-off-by: ChengyuZhu6 <chengyu.zhu@intel.com>
ChengyuZhu6 added a commit to ChengyuZhu6/kata-containers that referenced this issue May 23, 2024
Revert code logic in 462051b

Let me explain why:

In our previous approach, we implemented guest pull by passing PullImageRequest to the guest.
However, this method  resulted in the loss of specifications essential for running the container,
such as commands specified in YAML, during the CreateContainer stage. To address this,
it is necessary to integrate the OCI specifications and process information from the image’s configuration with the container in guest pull.

The snapshotter method does not care this issue. Nevertheless, a problem arises when two containers in the same pod attempt
to pull the same image, like InitContainer. This is because the image service searches for the existing configuration,
which resides in the guest. The configuration, associated with <image name, cid>, is stored in the directory /run/kata-containers/<cid>.
Consequently, when the InitContainer finishes its task and terminates, the directory ceases to exist.
As a result, during the creation of the application container, the OCI spec and process information cannot
be merged due to the absence of the expected configuration file.

Fix tests:
- k8s-credentials-secrets.bats
- k8s-file-volume.bats
- k8s-nested-configmap-secret.bats
- k8s-projected-volume.bats
- k8s-volume.bats
- k8s-shared-volume.bats
- k8s-sysctls.bats

Fixes: kata-containers#9666
Fixes: kata-containers#9667
Fixes: kata-containers#9668

Signed-off-by: ChengyuZhu6 <chengyu.zhu@intel.com>
ChengyuZhu6 added a commit to ChengyuZhu6/kata-containers that referenced this issue May 23, 2024
Revert code logic in 462051b

Let me explain why:

In our previous approach, we implemented guest pull by passing PullImageRequest to the guest.
However, this method  resulted in the loss of specifications essential for running the container,
such as commands specified in YAML, during the CreateContainer stage. To address this,
it is necessary to integrate the OCI specifications and process information from the image’s configuration with the container in guest pull.

The snapshotter method does not care this issue. Nevertheless, a problem arises when two containers in the same pod attempt
to pull the same image, like InitContainer. This is because the image service searches for the existing configuration,
which resides in the guest. The configuration, associated with <image name, cid>, is stored in the directory /run/kata-containers/<cid>.
Consequently, when the InitContainer finishes its task and terminates, the directory ceases to exist.
As a result, during the creation of the application container, the OCI spec and process information cannot
be merged due to the absence of the expected configuration file.

Fix tests:
- k8s-credentials-secrets.bats
- k8s-file-volume.bats
- k8s-nested-configmap-secret.bats
- k8s-projected-volume.bats
- k8s-volume.bats
- k8s-shared-volume.bats
- k8s-sysctls.bats

Fixes: kata-containers#9666
Fixes: kata-containers#9667
Fixes: kata-containers#9668

Signed-off-by: ChengyuZhu6 <chengyu.zhu@intel.com>
ChengyuZhu6 added a commit to ChengyuZhu6/kata-containers that referenced this issue May 23, 2024
Refactor code in guest pull

Fix tests:
- k8s-credentials-secrets.bats
- k8s-file-volume.bats
- k8s-nested-configmap-secret.bats
- k8s-projected-volume.bats
- k8s-volume.bats
- k8s-shared-volume.bats
- k8s-sysctls.bats

Fixes: kata-containers#9666
Fixes: kata-containers#9667
Fixes: kata-containers#9668

Signed-off-by: ChengyuZhu6 <chengyu.zhu@intel.com>
ChengyuZhu6 added a commit to ChengyuZhu6/kata-containers that referenced this issue May 23, 2024
Refactor code in guest pull

Fix tests:
- k8s-credentials-secrets.bats
- k8s-file-volume.bats
- k8s-nested-configmap-secret.bats
- k8s-projected-volume.bats
- k8s-volume.bats
- k8s-shared-volume.bats
- k8s-sysctls.bats

Fixes: kata-containers#9666
Fixes: kata-containers#9667
Fixes: kata-containers#9668

Signed-off-by: ChengyuZhu6 <chengyu.zhu@intel.com>
wainersm added a commit to wainersm/kata-containers that referenced this issue May 29, 2024
This test fails with qemu-coco-dev configuration and guest-pull image pull.

Issue: kata-containers#9668
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
ryansavino added a commit to AdithyaKrishnan/kata-containers that referenced this issue Jun 1, 2024
This test is failing due to the initContainers not being properly
handled with the guest image pulling.
Issue tracked here: kata-containers#9668
Skipping for now.

Signed-Off-By: Ryan Savino <ryan.savino@amd.com>
ryansavino added a commit to AdithyaKrishnan/kata-containers that referenced this issue Jun 1, 2024
This test is failing due to the initContainers not being properly
handled with the guest image pulling.
Issue tracked here: kata-containers#9668
Skipping for now.

Signed-Off-By: Ryan Savino <ryan.savino@amd.com>
ryansavino added a commit to AdithyaKrishnan/kata-containers that referenced this issue Jun 1, 2024
This test is failing due to the initContainers not being properly
handled with the guest image pulling.
Issue tracked here: kata-containers#9668
Skipping for now.

Signed-Off-By: Ryan Savino <ryan.savino@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/guest-pull bug Incorrect behaviour needs-review Needs to be assessed by the team.
Projects
Issue backlog
  
In progress
Status: No status
Development

No branches or pull requests

1 participant