Image volumes and container volumes

This would map closely to Docker's native volumes support, and allow people to build and version pre-baked data as containers. Maybe read-only? Haven't thought that far...

I guess so? But why not just use a git repo?

On Thu, Aug 7, 2014 at 10:11 PM, Tim Hockin notifications@github.com
wrote:

This would map closely to Docker's native volumes support, and allow
people to build and version pre-baked data as containers. Maybe read-only?
Haven't thought that far...

—
Reply to this email directly or view it on GitHub
#831.

More plugins more better? I wanted to put it out there, since we do sort
of diverge from Docker's native volumes support. Clearly not urgent :)

On Thu, Aug 7, 2014 at 10:18 PM, brendandburns notifications@github.com
wrote:

I guess so? But why not just use a git repo?

On Thu, Aug 7, 2014 at 10:11 PM, Tim Hockin notifications@github.com
wrote:

This would map closely to Docker's native volumes support, and allow
people to build and version pre-baked data as containers. Maybe
read-only?
Haven't thought that far...

Reply to this email directly or view it on GitHub
#831.

Reply to this email directly or view it on GitHub
#831 (comment)
.

More potential uses of this:

Deployment of scripts/programs for lifecycle hooks: One of the main points of the lifecycle hooks (PreStart and PostStop event hooks #140) is to decouple applications from the execution environment (Kubernetes in this case). If the hook scripts/programs must be deployed as part of the application container, that compromises this objective.
Dynamic package composition more generally: This would be more similar to our internal package model, where we can independently manage the base filesystem, language runtime, application, utility programs for debugging, etc.
Configuration deployment
Input data deployment

A git repo could be used for some of these cases, but for others it would be less than ideal.

If the base image of a docker file (e.g. FROM fedora) is a linux distro,
then isn't it going to be annoying to have a bunch of Linux Standard Base
type of files in what is really supposed to be a data-only packge?

On the other hand, if it is creating using tar -c . | docker import - myimage, then what is the advantage is of a docker image over a tar file?

On Tue, Sep 30, 2014 at 9:00 PM, bgrant0607 notifications@github.com
wrote:

More potential uses of this:

Deployment of scripts/programs for lifecycle hooks: One of the main
points of the lifecycle hooks (PreStart and PostStop event hooks #140
PreStart and PostStop event hooks #140) is to
decouple applications from the execution environment (Kubernetes in this
case). If the hook scripts/programs must be deployed as part of the
application container, that compromises this objective.

Dynamic package composition more generally: This would be more
similar to our internal package model, where we can independently manage
the base filesystem, language runtime, application, utility programs for
debugging, etc.

Configuration deployment

Input data deployment

A git repo could be used for some of these cases, but for others it would
be less than ideal.

—
Reply to this email directly or view it on GitHub
#831 (comment)
.

How about the VOLUMES directives of a container? Any container can declare
itself to be exposing any number of volumes. Maybe the functionality to
expose is not the whole container, but just the volumes from that container?

On the other hand, people can create data containers "FROM scratch", and
who are we to say it's annoying?

On Wed, Oct 1, 2014 at 7:20 AM, erictune notifications@github.com wrote:

If the base image of a docker file (e.g. FROM fedora) is a linux distro,
then isn't it going to be annoying to have a bunch of Linux Standard Base
type of files in what is really supposed to be a data-only packge?

On the other hand, if it is creating using tar -c . | docker import - myimage, then what is the advantage is of a docker image over a tar file?

On Tue, Sep 30, 2014 at 9:00 PM, bgrant0607 notifications@github.com
wrote:

More potential uses of this:

Deployment of scripts/programs for lifecycle hooks: One of the main
points of the lifecycle hooks (PreStart and PostStop event hooks #140
PreStart and PostStop event hooks #140) is to
decouple applications from the execution environment (Kubernetes in this
case). If the hook scripts/programs must be deployed as part of the
application container, that compromises this objective.

Dynamic package composition more generally: This would be more
similar to our internal package model, where we can independently manage
the base filesystem, language runtime, application, utility programs for
debugging, etc.

Configuration deployment

Input data deployment

A git repo could be used for some of these cases, but for others it
would
be less than ideal.

Reply to this email directly or view it on GitHub
<
https://github.com/GoogleCloudPlatform/kubernetes/issues/831#issuecomment-57416301>

.

Reply to this email directly or view it on GitHub
#831 (comment)
.

@thockin

@thockin I like the initial idea of having new volume type to support docker's data volume container, which matches common package, shared package etc. concept internally. I also can see the potential use cases listed by @bgrant0607. But please don't go down to the road like Docker had today: declare a container as data volume, in which case, we are going to introduce another level complexity of dependencies to containers within a pod, or even dependencies between pods if a pod only has data volume container, etc. I think your initial idea of having a volume type which actually refers as a docker volume container or other read-only volume is a better approach for a long run.

@thockin

The interesting thing about docker volumes is that a container does not
have to be RUNNING for the volumes to exist. It's a weird model, but I
think it could work.

I don't think we know what people really want in this space yet, though.

On Wed, Oct 1, 2014 at 10:57 AM, Dawn Chen notifications@github.com wrote:

@thockin https://github.com/thockin I like the initial idea of having
new volume type to support docker's data volume container, which matches
common package, shared package etc. concept internally. I also can see the
potential use cases listed by @bgrant0607 https://github.com/bgrant0607.
But please don't go down to the road like Docker had today: declare a
container as data volume, in which case, we are going to introduce another
level complexity of dependencies to containers within a pod, or even
dependencies between pods if a pod only has data volume container, etc. I
think your initial idea of having a volume type which actually refers as a
docker volume container or other read-only volume is a better approach for
a long run.

Reply to this email directly or view it on GitHub
#831 (comment)
.

It appears the same net effect as this issue can be achieved without introducing a new volume type using two containers in a pod and some shell wrapped around the underling container command lines (see #1589 (comment)).

How to decide on container-as-a-volume vs. command-line-based sequencing?

more portability between Kubernetes and non-Kubernetes docker use cases with container-as-volume.
easier for user to discover container-as-volume concept and identify that it is the right solution?
having fewer and less general mechanisms for setting up "packages" may lend itself to more tightly integrated build/deploy systems. But, maybe that is not a goal for Kubernetes.
either solution can integrate with data durability, I think.
liveness checking is more complex with command-line-based sequencing, since the pod to be liveness checked goes through a waiting phase and then a running phase.

There are a lot of ways to go about it and a new volume type is in my opinion not needed. I tried to standardize the way we structure data and make different volume providers possible. These range from Host Volumes, Data Volume Containers, Side Containers with additional logic to Volume as a Service, which is where k8s could integrate greatly. The start is already available via git as volume. I think the native Volumes in Docker are enough, but just lack a standard. The more detailed ideas are available at moby/moby#9277.

I think the question is whether the data volume container should be represented as a container or as a volume. I prefer to think of them as volumes and find passive containers to be non-intuitive for users, problematic for management systems to deal with, and the source of weird corner-case behaviors.

In the spirit of k8s composability though, the desire is to have one container for nginx that can be independently updated from a second container storing your static files. they are combined together at runtime via k8s pod semantics. This is what the issue is about.

I have a similar requirement in my project - I'm building a platform that exposes a number of REST services for language processing. This includes tools like speech recognition which are structured as an "engine" (a few MB in size, the code that implements the service) and a separate "model" (1-5GB in size, for each specific language/domain/etc.). The two can evolve independently, and we might have three engine versions and 20 different models active at any given time. Currently we have 60 different images, and when a new engine is released we have to re-build 20 images of many GB each to embed all the different models.

Ideally we'd be able to build three images for the engines and 20 images for the models, and mount the model into the engine container via the pod spec. This would make updating the engine a much lighter weight process as the nodes only have to pull one new engine image (a few MB) rather than 20 new engine+model images (potentially up to 100GB).

Merge pull request kubernetes#831 from bertinatto/gce-azure-csi

i want mount docker registry auth and certs to official registry images , i'd like put the auth and certs in another image , i don't need prepare auth and certs at local host path

i want mount docker registry auth and certs to official registry images , i'd like put the auth and certs in another image , i don't need prepare auth and certs at local host path

Secrets in general are better suited to be put into Kubernetes secrets then images, IMO. Easier to encrypt/audit. You can treat them like volumes. Could you try that and see if it will do what you want?

I already commented here: #6120

But to summarize, I managed to share the container filesystem using shareProcessNamespace like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: datasharing
spec:
  selector:
    matchLabels:
      app: datasharing
  template:
    metadata:
      labels:
        app: datasharing
    spec:
      shareProcessNamespace: true
      containers:
        - name: data
          image: ubuntu
          imagePullPolicy: IfNotPresent
          command: ["/bin/bash", "-c"]
          args: ["trap 'echo signal;exit 0' SIGINT; sleep infinity"]
          resources:
            limits:
              memory: "128Mi"
              cpu: "500m"
          lifecycle:
            postStart:
              exec:
                command:
                  - /bin/sh
                  - -c
                  - mkdir -p /DATA && touch /DATA/empty_test_file
        - name: nginx
          image: nginx:stable
          imagePullPolicy: IfNotPresent
          resources:
            limits:
              memory: "128Mi"
              cpu: "500m"
          ports:
            - containerPort: 8080
          env:
            - name: GET_DATA_PROC_DIR
              value: 'find /proc -maxdepth 1 -type d -regex "/proc/[0-9]*" | head -2 | tail -1'
          lifecycle:
            postStart:
              exec:
                command:
                  - /bin/sh
                  - -c
                  - ln -s $(eval $GET_DATA_PROC_DIR)/root/DATA /mnt/DATA

If you can put a native executable into the data container, then it's less tricky to just symlink from there via emptyDir intermediary. Perhaps, also less opportunity for not having the symlink already in place when the service starts.

shareProcessNamespace: true
securityContext:
  runAsUser: 0 # /proc/<pid>/root resolve fails if not root
volumes:
- name: share
  emptyDir: {}
containers:
- name: data
  image: data:1 # FROM busybox
  volumeMounts:
  - name: share
    mountPath: /share
  command:
  - "/bin/sh"
  - "-c"
  - "ln -s /proc/$$$$/root/model /share/; sleep infinity"
- name: service
  image: service:1
  volumeMounts:
  - name: share
    mountPath: /share
    readOnly: true
  command:
  - "/bin/sh"
  - "-c"
  - "while test ! -L /share/model; do sleep 1; done; exec -a service /bin/service --data=/share/model"

@arkadijs

@arkadijs thanks for validating the symbolic link approach that model cars have been introduce to KServe (see also https://docs.google.com/document/d/1Bs4fnP8rhPMaoPoLSYVvuRq-z9vkGPQ0rKbmfH4I7js/edit?usp=sharing for the design document from 10/2023)

See also the proposal kserve/kserve#3646 to avoid race conditions decoupled from the main container (by using K8s sidecar containers (initContainers with restartPolicy set to Always) and a startupProbe)

@rhuss

Hi @rhuss , in your linked document you state that it is only required that the containers run with the same UID, but in my tests I was able to open /proc/$pid/root of one container from another only when running the containers as root. Otherwise I get permission errors.

Do you have any test case that you could share by any chance?

Thanks in advance

@rhuss

Hi @rhuss , in your linked document you state that it is only required that the containers run with the same UID, but in my tests I was able to open /proc/$pid/root of one container from another only when running the containers as root. Otherwise I get permission errors.

Sorry for the late reply. Unfortunately, I have no clear indication of which UID combination is allowed on which installation, but it definitely depends on your Kubernetes setup. E.g. on Minikube it does not work with UID 0, but only when both containers run with the same UID not equal to 0. No idea why this is the case. On OpenShift it works with the standard setup as OpenShift chooses arbitrary UIDs that is the same for all containers of a Pod.

It would be really cool to find out, how to influence the filesystem permission handling on the various Kubernetes setups.

Can we close this in favor of kubernetes/enhancements#4639?

It's been 10 years. So, sure.

It's been 10 years. So, sure.

I mostly wonder if I missed any context here - something that the new KEP will not cover. If you see something I missed - please comment. Thx

Happy to close it in favor of a real KEP :)

It's been 10 years. So, sure.

I mostly wonder if I missed any context here - something that the new KEP will not cover. If you see something I missed - please comment. Thx

Agreed, we're right now collecting feedback for a possible beta graduation of the feature. Any input is welcome 👍

Navigation Menu

Repository navigation

Image volumes and container volumes #831

167 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Image volumes and container volumes #831

Description

Activity

brendandburns commented on Aug 8, 2014

thockin commented on Aug 8, 2014

bgrant0607 commented on Oct 1, 2014

erictune commented on Oct 1, 2014

thockin commented on Oct 1, 2014

dchen1107 commented on Oct 1, 2014

thockin commented on Oct 1, 2014

erictune commented on Oct 6, 2014

stp-ip commented on Nov 21, 2014

bgrant0607 commented on Nov 21, 2014

167 remaining items

ianroberts commented on May 15, 2021

weironz commented on Feb 11, 2023

kfox1111 commented on Feb 13, 2023

toms-place commented on Jul 6, 2023

arkadijs commented on Apr 16, 2024

rhuss commented on Apr 29, 2024

ralgozino commented on Jun 11, 2024

rhuss commented on Jul 22, 2024

SergeyKanzhelev commented on Sep 13, 2024

dvusboy commented on Sep 13, 2024

SergeyKanzhelev commented on Sep 13, 2024

thockin commented on Sep 16, 2024

saschagrunert commented on Sep 17, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions