-
Notifications
You must be signed in to change notification settings - Fork 4.9k
-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
how to persist data for postgres and mounted hostpath volume #8564
Comments
Virtualbox on macOS should automatically mount /Users, I verified that it still works as intended. Any changes to /Users in the VM should get persisted to the host machine. |
@bahtou do you still have this issue ? with latest version of minikube ? do you mind re-verifying or provide me exact steps so I could reproduce this ? (maybe the commands that you run that shows the data is not presistant) if we can have a reproduce-able way we could and we should make an integration test for this, so we test this on every PR. |
@sharifelgamal I agree with your comment. But for some reason postgres data does not persist to host. And I'm not sure if this is a permissions issue or minikube. |
@bahtou thank you, do you mind also try if you explicitly mount the folder you want either using --mount-string option to the start or using minikube mount command and see if you still have that problem? minikube start --help | grep mount-string
|
minikube version
start minikubeminikube start --driver=virtualbox --cpus=2 --memory=5120 --kubernetes-version=v1.18.3 --container-runtime=docker --mount=true --mount-string=/Users/<>/minikube/pgdata:/data
// output
😄 [test-host] minikube v1.11.0 on Darwin 10.15.4
✨ Using the virtualbox driver based on user configuration
👍 Starting control plane node test-host in cluster test-host
🔥 Creating virtualbox VM (CPUs=2, Memory=5120MB, Disk=20000MB) ...
🐳 Preparing Kubernetes v1.18.3 on Docker 19.03.8 ...
🔎 Verifying Kubernetes components...
📁 Creating mount /Users/<>/minikube/pgdata:/data ...
🌟 Enabled addons: default-storageclass, storage-provisioner
🏄 Done! kubectl is now configured to use "test-host" From the above mount we have connected host <--> vm via the create a file on your local host names ---
apiVersion: v1
kind: Pod
metadata:
name: pg-pod
labels:
name: postgres
spec:
containers:
- name: postgres
image: postgres:12.3
imagePullPolicy: IfNotPresent
ports:
- name: pg-port
containerPort: 5432
env:
- name: POSTGRES_PASSWORD
value: admin
- name: PGDATA
value: /data/k8s
volumeMounts:
- name: pg-vol
mountPath: /var/lib/postgresql/data
securityContext:
runAsUser: 0
runAsGroup: 0
volumes:
- name: pg-vol
hostPath:
path: /data
# path: /Users/<>/minikube/pgdata
restartPolicy: Never
securityContext:
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000 Then run kubectl apply -f postgres-pod.yaml Once the pod is up and running go into it and verify that the data is persisted at $ kubectl exec -it pg-pod -- /bin/bash
root@pg-pod:/# ls /data/k8s
// you should get a list of postgres files Now check that the files are showing up in the minikube vm $ minikube ssh
$ ls -al /data
// nothing shows up in the vm Go ahead and uncomment the second path to see the same results. |
@bahtou I think there was a typo in your yaml I was able to have a successful data by changin
to
I think the problem was you are mounting the folder into the wrong path into your pod see details:
|
@bahtou if you please confirm that was the typo we could close this issue ? btw thank you for the great step by step repoducable detail, we could make an integraiton test out of this ! |
@medyagh yes sorry typo on my part. You are correct and the data does persist to the minikube vm. What is missing is the persistence to the host machine: host <--> vm <--> container. The above persists only to the vm. I'm not able to view the files on the host machine even though it has been mounted. |
@bahtou thank you for the clearification ! this indeed does look like a bug ! and we should fix it soon !
and then inside minikube ssh
outside minikube:
is empty !!! this is a bug ! and we need to fix this ! thank you brining this to our attention @bahtou I wonder if "minikube mount" command would fix your problem? |
I open another terminal and run this: minikube mount /Users/<>/minikube/pgdata:/data
// output
📁 Mounting host path /Users/<>/minikube/pgdata into VM as /data ...
▪ Mount type: <no value>
▪ User ID: docker
▪ Group ID: docker
▪ Version: 9p2000.L
▪ Message Size: 262144
▪ Permissions: 755 (-rwxr-xr-x)
▪ Options: map[]
▪ Bind Address: 192.168.99.1:62273
🚀 Userspace file server: ufs starting
✅ Successfully mounted /Users/<>/minikube/pgdata to /data
📌 NOTE: This process must stay alive for the mount to be accessible ... In another terminal run the manifest: kubectl apply -f postgres-pod.yaml
kubectl get pod
NAME READY STATUS RESTARTS AGE
pg-pod 0/1 Error 0 40s
kubectl logs pg-pod
mkdir: cannot create directory ‘/data’: Permission denied @medyagh This could also be a postgres permissions issue? Something to keep in mind. |
Also, just to confirm. By running this: minikube mount /Users/<>/minikube/pgdata:/data
// output
📁 Mounting host path /Users/<>/minikube/pgdata into VM as /data ...
▪ Mount type: <no value>
▪ User ID: docker
▪ Group ID: docker
▪ Version: 9p2000.L
▪ Message Size: 262144
▪ Permissions: 755 (-rwxr-xr-x)
▪ Options: map[]
▪ Bind Address: 192.168.99.1:62273
🚀 Userspace file server: ufs starting
✅ Successfully mounted /Users/<>/minikube/pgdata to /data
📌 NOTE: This process must stay alive for the mount to be accessible ... and then going into another terminal and $ minikube ssh
$ echo 'something' > /data/myfile.txt Checking locally I see the file persist here |
@bahtou I think you are right ! this could be a permission you might need to either make a service account for the pod that has access to the hostpath I verfied without postgress the mount works fine
|
we are doing a terrible job in minikube in providing a good tutorial for storage provsioner, |
@medyagh actually, I was writing a tutorial on using postgres + kubernetes when I encountered this issue with minikube. I began with PersistentVolume and hostPath, and then just did a simple pod with hostPath which lead to me posting here. Wasn't sure if it was a postgres permissions or pod/container or minikube. Unfortunately, setting |
@bahtou does the comment in this issue help https://github.com/kubernetes/minikube/issues/7828#issuecomment-661831907\ |
cc: @priyawadhwa |
@bahtou have you tried recently we had an update on storage provisioner ? do you mind sharing with us if you gave up on minikube or found a solution ? |
@medyagh I'll take a look tonight. |
Have this issue too when running something like: And it solved my issue, I can see my file changes in both minikube vm and my macos. However, I'm facing new issue with the --mount flag. Using helm with jenkins with pvc |
works in the VM. |
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-contributor-experience at kubernetes/community. |
Stale issues rot after 30d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-contributor-experience at kubernetes/community. |
Hello, Any news on this issue? I have been facing it using Mac OS and MiniKube with hyperkit driver. Either I get a permission error or by manipulating the PGDATA variable I can make the pod run but files are simply not persisted on the host path. The only thing I need is to share a folder from the minikube vm with the host machine (minikube mount) and then use that sane folder as mount of /var/lib/postgresql/data on postgresql pod so that pod whiites on that vm folder and this is reflected on the host folder, keeping the data ammong minikube runs. I have tried every possible thing I could think of, but still no results. |
I guess I am facing this same issue with virtualbox as driver. The image I am trying to deploy sonatype:nxus3 operates with uid and gid 200.
drwxrwxr-x 3 1000 1000 4096 storage
😄 [test-mount] minikube v1.24.0 on Ubuntu 20.04
✨ Using the virtualbox driver based on user configuration
👍 Starting control plane node test-mount in cluster test-mount
🔥 Creating virtualbox VM (CPUs=2, Memory=6000MB, Disk=15360MB) ...
🐳 Preparing Kubernetes v1.22.3 on Docker 20.10.8 ...
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
📁 Creating mount /tmp/storage/:/storage ...
🔎 Verifying Kubernetes components...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟 Enabled addons: default-storageclass, storage-provisioner
🏄 Done! kubectl is now configured to use "test-mount" cluster and "default" namespace by default
drwxr-xr-x 2 0 0 4096 data
drwxrwxr-x 1 1000 1000 4096 storage
/storage:
drwxrwxr-x 1 1000 1000 4096 nexus
uid=1000(docker) gid=1000(docker) groups=1000(docker),10(wheel),1011(buildkit),1016(podman),1017(vboxsf)
namespace/nexus created
persistentvolume/nexus-data-pv created
persistentvolume/nexus-storage-pv created
persistentvolumeclaim/nexus-data-pvc created
persistentvolumeclaim/nexus-storage-pvc created
deployment.apps/nexus created
service/nexus-service created
NAME READY STATUS RESTARTS AGE
pod/nexus-66f4ffdb9f-zxqhb 1/1 Running 0 106s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/nexus-service NodePort 10.110.81.115 <none> 8081:32000/TCP 106s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/nexus 1/1 1 1 106s
NAME DESIRED CURRENT READY AGE
replicaset.apps/nexus-66f4ffdb9f 1 1 1 106s
Defaulted container "nexus" out of: nexus, fix-data-pvc (init), fix-storage-pvc (init)
drwxr-xr-x 15 200 200 4096 nexus-data
Defaulted container "nexus" out of: nexus, fix-data-pvc (init), fix-storage-pvc (init)
total 4
drwxrwxr-x 1 1000 1000 4096 nexus-storage I also tried to use fsGroup, but didn't work either. When the application tries to write data on |
Hi all, any news about this issue? Postgres Container Log: If I configure the - mountPath: "/var/lib/postgresql" instead of - mountPath: "/var/lib/postgresql/data", the permissions error goes away, but the host .../data folder is created empty, and the real data is inside the container. So, after reboot the data is lost. I'm also using minikube mount /Users/[mylocalpath]/data/:/data , and it's visible using "minikube ssh" command, and also it's enabled in Docker "Shared folders" config (/Users). What I need is to store my Postgres PV in a local Mac folder and keep it persistent even after a minikube stop/start (or notebook reboots) BTW, I'm mounting another folder with the same approach as pv + pvc, etc, for Geoserver app, and it works as expected keeping the data after a minikube stop/start, and the data is written in the correct local MAC path. I think that the Postgres container cannot write in the local folder even with chmod 777, caused by it's using the postgres user from the container. Any idea in how to solve the PV problem? Thank you |
The files stored under The same goes for the hostpath-provisioner, if using PV. Currently there is some confusion about /tmp mountpoints vs. storage. |
ohhh yes!, you are right, thanks to your comment I finally did it! My notes to help others:
pvapiVersion: v1
pvcapiVersion: v1
deploymentapiVersion: apps/v1 Hope it helps other people :) |
I would like to add some more info I dug up about this issue.
You can see that only
So Minikube makes mounted folder available to all users. When I then run PostgreSQL pod I get this logs:
From the other side, PostreSQL tries to make the data folder available to I would say this is not a bug of I guess there we have two ways. First, setup PostgreSQL in some way that it doesn't change permissions of data folder. It's bad practice from security point but could be useful for development purpose (just like If you want to reproduce my experiments, I'll attach manifests.
|
Ok, I give up. I've tried a lot of different approaches. And the only thing I found is that minikube doesn't allow to change permissions of volumes mounted by it. |
Steps to reproduce the issue:
minikube start --driver=virtualbox --network-plugin=cni --memory=5120 --kubernetes-version=v1.18.3 --container-runtime=docker
kubectl apply -f postgres-pod.yaml
I am trying to persist the data to the host machine and not the minikube vm. The above manifest works and no errors are reported but the data files are not persisted to the host machine, but they are in the minikube vm.
I have been fidgeting with the security context and the above configuration doesn't throw ‘permission denied’ errors. How to pass through the data files onto the host machine?
host machine <---> minikube vm <---> pod/container
My understanding is that
/Users
should reflect in both minikube vm and host machine [https://minikube.sigs.k8s.io/docs/handbook/mount/]The text was updated successfully, but these errors were encountered: