Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

volume hostpath with subpath #14836

Open
panho66 opened this issue Jul 29, 2018 · 16 comments

Comments

@panho66
Copy link

@panho66 panho66 commented Jul 29, 2018

Rancher versions:
rancher/server or rancher/rancher: 2.0.6

Infrastructure Stack versions:
kubernetes (if applicable): v.1.10.5

Create a volume in a pod
image

According to document, https://kubernetes.io/docs/concepts/storage/volumes/#using-subpath

a Pod uses subPath to create a directory pod1 within the hostPath volume /var/log/pods, using the pod name from the Downward API. The host directory /var/log/pods/pod1 is mounted at /logs in the container.

Expecting directory /iag/unix create in node host and mount to pod as /unix

But got
image

view yaml
image

Not sure what I missed. No directory created in node host.

@loganhz

This comment has been minimized.

Copy link
Member

@loganhz loganhz commented Jul 29, 2018

It's a known k8s issue. It will work if you leave subpath empty
Please check kubernetes/kubernetes#61456

@janeczku

This comment has been minimized.

Copy link
Contributor

@janeczku janeczku commented Mar 13, 2019

This is still an issue with Rancher installed K8s 1.11 and 1.12.
The issue for hostPath+subPath with containerized Kubelet has been fixed upstream since v1.11: kubernetes/kubernetes#63143

It looks like there is some work todo on RKE side (like mounting the host rootfs to kubelet /rootfs mountpoint).

@janeczku

This comment has been minimized.

Copy link
Contributor

@janeczku janeczku commented Mar 13, 2019

Steps to reproduce:

  1. Create a cluster in Rancher using K8s v1.12.x
  2. Deploy a workload that mounts a hostPath volume using a subPath (see manifest below)
  3. Exec to the pod and create a new file under the mount point (e.g. uniquefile)
  4. SSH to the host where the Pod is running.
  5. Verify that the file was created under the expected hostPath/subPath in the rootfs

Result:

worker1:/# stat /site-data/nginx
stat: cannot stat '/site-data/nginx': No such file or directory
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        volumeMounts:
        - mountPath: /volume/nginx
          name: site-data
          subPath: nginx
      volumes:
      - name: site-data
        hostPath:
          path: /site-data
          type: DirectoryOrCreate
@JorinL

This comment has been minimized.

Copy link

@JorinL JorinL commented Mar 21, 2019

Ran into that problem today and can also reproduce it with mosquitto and a sub path in the claimed volume. Using v1.13.4-rancher1-1 and rancher-agent:v2.1.7

@janeczku

This comment has been minimized.

Copy link
Contributor

@janeczku janeczku commented Mar 28, 2019

/cc @loganhz

@loganhz

This comment has been minimized.

Copy link
Member

@loganhz loganhz commented Mar 28, 2019

@gitlawr

This comment has been minimized.

Copy link
Member

@gitlawr gitlawr commented May 10, 2019

Here's the workaround:

  1. Edit cluster, Edit as YAML
  2. Add the following flags for kubelet:
services:
  kubelet:
    extra_args:
      containerized: "true"
    extra_binds: 
      - "/:/rootfs:rshared"
  1. Click save and wait till the cluster is updated.

Notes:
The community is planning to deprecate the "--containerized" for kubelet(kubernetes/kubernetes#74148).
But the flag is essential for the capability as there is no alternative at the moment.

@mrtndwrd

This comment has been minimized.

Copy link

@mrtndwrd mrtndwrd commented May 23, 2019

@gitlawr does that mean you're mount-binding the whole root system of your Kubernetes host to the kubelet container? That can't be good for security, right?

@JorinL

This comment has been minimized.

Copy link

@JorinL JorinL commented May 23, 2019

My Workaround is, to just create for every subpath I would have used, a persistent volume which I can claim.
So I have exactly the path I wanted to use but without using the field subpath at all.

@cjellick

This comment has been minimized.

Copy link
Member

@cjellick cjellick commented Aug 12, 2019

Looks like an rke fix?

@cjellick

This comment has been minimized.

Copy link
Member

@cjellick cjellick commented Aug 12, 2019

I realize @galal-hussein is out this week, but assigning to him at least temporarily

@deniseschannon deniseschannon modified the milestones: v2.3, v2.3.x Aug 16, 2019
@varac

This comment has been minimized.

Copy link

@varac varac commented Aug 27, 2019

From #14836 (comment):

The community is planning to deprecate the "--containerized" for kubelet(kubernetes/kubernetes#74148).

This already got merged, so a containerized kubelet is already deprecated.

But the flag is essential for the capability as there is no alternative at the moment.

What's the plan for rancher/rke regarding hyperkube ? When does the containerized kubelet gets replaced ? This issue is really annoying because a lot of upstream charts break on a hostpath provider due to their use of subpath.

@Leen15

This comment has been minimized.

Copy link

@Leen15 Leen15 commented Aug 30, 2019

Any news here?

@varac

This comment has been minimized.

Copy link

@varac varac commented Sep 16, 2019

I created a feature request to deprecate the containerized kubelet.
Meanwhile I have two questions, maybe somebody from rancher could comment on:

  • What's the recommended workaround for this issue ? Is it bind-mounting the whole host rootdir into the kubelet container like suggested by @gitlawr in above comment ?
  • If so, are there any security implications with this, or anything that could reduce potential risks ?
@varac

This comment has been minimized.

Copy link

@varac varac commented Sep 16, 2019

Is it bind-mounting the whole host rootdir into the kubelet container like suggested by @gitlawr in above comment ?

To be more precise, which host directories need to get additionally bind-mounted into the pod in order to make subpath work ?

@bruno-lopes

This comment has been minimized.

Copy link

@bruno-lopes bruno-lopes commented Oct 11, 2019

Any news on this??? The ability to mount an specific file is very important to us.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.