podman kube play support for data volume container podman run --volume-from

[vikas-goel]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind feature

Description

We have a use-case of data volume container where such containers are created but never started. The volumes of these containers are then bind mounted to an application container using --volumes-from option. Docker Compose supports this deployment. In order to move to Podman kube, we need this use-case supported.

Does podman kube play support it?

[rhatdan]

Would an init container work for this use case? I think they would get started, but you could simply have them exec /bin/true.

[rhatdan]

But wouldn't it be better to just define the volumes in the kube yaml?

[vikas-goel]

The data volume container images do not contain any runtime environment. So, exec to sh/true will fail.

If we were to specific the volumes in the YML, then those data volume containers need to be created, inspect their volumes and then specify in the YML.

Whereas, with Docker Compose I can specify the application and data volume containers all in the same YML with service dependencies setup in there itself. The Docker Compose can just create (not start) the data volume containers and bind mount all the volumes to the application container.

I am looking for equivalent in podman kube.

[rhatdan]

Podman kube uses kubernetes.yaml so the equivalent would be how kubernetes handles volumes. There is nothing specific about podman use of kubernetes.yaml.

With k8s volumes there is no need to specify a container with a volume. But maybe I don't under stand, are you saying that a container image comes with data stored in its image that other containers they use?

Could you point at an example of a "volume container"?

[vikas-goel]

I understand your point of k8 yaml and volumes. My use case is more of general available podman run --volumes-from that we are able to manage using the Compose yaml. In order to move away from Compose to 'podman kube', we will need this capability.

There are a variety of use cases for the data volume container image. For example, one use case is to bundle the orchestration scripts for life cycle management of the containers. The other being bundling some (on-demand, out-of-band) plugins such as some library files to extend the capability of an application without having to recreating the application image for every such use-case.

[rhatdan]

Well we can not directly change the kubernetes.YAML file in a way that would break k8s. We could add ANNOTATIONS, which we have added in the past and then make podman kube play work differently when it sees these annotations.
But that would probably mean you Kubernetes YAML files would only work with Podman.

[rhatdan]

Look for annotations mentions in the podman kube play man page.

[vikas-goel]

Use of annotation and restriction to Podman environment are absolutely fine. To reiterate the use-case,

1. A data volume container image, that may consist of script, .so libraries etc., is built from SCRATCH. There is no runtime environment inside the image. The container image may have or more volumes. A container of this image is created but never run. The volumes of the image are setup (copy-up) when the container is created.
2. An application container image that has application bundled in it.
3. A Podman kube YML file that has definition of both the application container and data volume container. When the YML is used to run podman kube play, all the volumes of the data volume container are attached to the application container in read-only (or rw) mode as specified.

[rhatdan]

I wonder if an image volume would help with this. Currently Podman supports image volumes, which is somewhat similar.

[vikas-goel]

Is there a sample podman kube YML file for the image volumes that satisfies the use case?

[vikas-goel]

Is there an update?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@ygalblum @umohnani8 WDYT?

[vikas-goel]

Been a while. Checking if there is any update.

[rhatdan]

Best way to move something forward is to open a PR.

[vikas-goel]

Sure. Any pointers where to start with?

[rhatdan]

Take a look at cmd/podman/create and see how we pass in --volume-from to start and then port this to cmd/podman/kube.

Does Kubernetes have some concept of this?
@ygalblum @umohnani8 WDYT?

[ygalblum]

I'm not sure if I understand the requirement. So, I'll first repeat what I understood.

The requirement is to mount a container image as a volume. Is this correct?

If yes, then the way to achieve this today is by creating a volume:

podman volume create --driver image --opt image=<Container Image>

Please note that volume create expects the image to be available locally (i.e. will not pull it).

In addition, since the question is about kube play, I've noticed that while there is an annotation for setting the driver volume.podman.io/driver, there wasn't one for setting the image. So, I've opened a PR for it #21342.

Once it's merged and released, you will be able to create image based volumes using PVCs. For example:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: fedora
  annotations:
    volume.podman.io/driver: image
    volume.podman.io/image: registry.fedoraproject.org/fedora:39
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 8Gi

Please let me know if this addresses the issue, or if I misunderstood the request altogether.

[ygalblum]

@rhatdan as for your question about K8S, I don't know that it is supported for pods. I do know that in KubeVirt users can use container images for storing the qcow2 files.

[vikas-goel]

Hi @ygalblum ,
This is about podman container create --volumes-from option.
https://docs.podman.io/en/stable/markdown/podman-create.1.html#volumes-from-container-options
https://github.com/containers/podman/blob/main/pkg/specgen/specgen.go#L260
https://github.com/containers/podman/blob/main/cmd/podman/common/create.go#L824
https://github.com/containers/podman/blob/main/pkg/domain/entities/pods.go#L146
https://github.com/containers/podman/blob/main/pkg/domain/entities/pods.go#L299

[vikas-goel]

I checked the image volume option. It is not an alternative of --volumes-from in the current form for

• it mounts everything that's present in the image. The use-case is to mount only the VOLUMES declared in the image.
• unlike --volumes-from, mounting this volume in a container requires destination when starting the container. Whereas, the -volumes-from option mounts the volumes to the declared VOLUMES location.

[vikas-goel]

@rhatdan ,
Does the following approach make sense to support volumes-from?

Under volumes, set persistentVolumeClaim claimName to container:<ctr-name>
Under volumeMounts, leave mountPath unspecified

For example,

apiVersion: v1
kind: Pod
metadata:
  name: volumes-from-test
  annotations:
    io.podman.annotations.init.container.type: "idle"
spec:
  initContainers:
  - name: myctr1
    image: localhost/dv-ctr:latest
  containers:
  - name: myctr2
    image: registry.access.redhat.com/ubi8/ubi:8.9
    volumeMounts:
    - name: from-myctr1-pvc
      
volumes:
  - name: from-myctr1-pvc
    persistentVolumeClaim:
      claimName: "container:myctr1"

The respective code change would be as below.

In abi.playKubePod()
https://github.com/containers/podman/blob/main/pkg/domain/infra/abi/play.go#L456

-> kube.InitializeVolumes() returns a list of volumes-from container names (#L613) https://github.com/containers/podman/blob/main/pkg/specgen/generate/kube/volume.go#L288

-> playKubePod() sets a new VolumesFrom in the kube.CtrSpecGenOptions struct (#L789,#L865)

-> kube.ToSpecGen() in turn sets the VolumesFrom in the specgen.SpecGenerator struct (#L811,#L892) https://github.com/containers/podman/blob/main/pkg/specgen/generate/kube/kube.go#L184

-> generate.MakeContainer() handles the VolumesFrom option (#L827,#L908)

[umohnani8]

@vikas-goel the example you have from #16819 (comment), does that work in a k8s cluster? With adding supporting for this behavior we don't want to end up breaking portability to a k8s cluster.

[vikas-goel]

@vikas-goel the example you have from #16819 (comment), does that work in a k8s cluster? With adding supporting for this behavior we don't want to end up breaking portability to a k8s cluster.

The way I look at it is the k8s cluster yaml works in podman. The podman kube yaml, regardless of the change in the example, may or may not work. That is, the same desired effect may not be seen as the podman specific annotations and customization will be disregarded by k8s.

[ygalblum]

@umohnani8 @vikas-goel First, clearly this YAML will not work on a K8S cluster because the feature of instructing K8S to mount volumes defined on one container on a different one does not exits. Furthermore, the schema of the value container:<Name> is illegal since having the character : in the name is not allowed in K8S.
Having said that, I think what's important here is that the change will not (and IIUC it doesn't) change the key schema.