NFS example does not work on google container engine. #24687
Comments
|
What version of NFS are you using? |
|
Can you also include your exports from the nfs server |
|
@erinboyd I don't know. What ever version is running in the google image from the example. gcr.io/google_containers/volume-nfs I can't find the source of google docker images. This is a vanilla setup taken straight out of the documentation it should "just work". |
|
According to the README it is exporting /mnt/data as /. |
|
@erinboyd I have tried a couple of other docker images in place of the gcr.io/google_containers/volume-nfs without any success. Do you have or know of a working nfs-server docker implementation? This is the last piece keeping me from moving my setup from colocation to google cloud. |
|
+1 |
|
here is my output: |
|
@arenoir I have solved this problem, see if my circumstance also applies for you: I found that there are some problems within the image The docker file says that it copies |
|
@xidui, thanks for the info... I was not able to get the gcr.io/google_containers/volume-nfs to image to work. I ended up using the image jsafrane/nfsexporter. All is well that ends well. |
mml
added
kind/documentation
|
@arenoir it looks like you got things working, but I believe we should still correct our docs, at least. |
|
@pwittrock ^^^ |
bgrant0607
added
priority/backlog
|
cc @kubernetes/examples |
|
The same error( in vagrant): And the nfs-server error: |
|
Note, the image is updated per #22665. The example works on GCE/GKE, AWS and Cinder. |
|
I am also getting this output, but the nfs-sample still works for me in the gce cloud installation
It might be interesting to know that reproducible it does NOT work in local k8s in docker contained installations based on debian. I am wondering if this might be a useful hint? Be it, that locally I am using HostPath pvs or connected to some network-manager I am running on my laptop. What hints me in the network direction rather than the HostPath is that when I start nfs-common service on my local machine I eventually get a full lockup of my box running the nfs example. Something about "cpu locked for more than 22s" |
|
cc @fabioy |
|
I got this example working, however during testing the busybox rc (replicas=2) I noticed that both busybox pods seemed to write to the file at the same time (first entry). Where all other cat file commands show only one host like the second one. Is this normal? |
|
@sijnc yes, that's how the example works. There are 2 replica writing to the nfs share. What you see from curl is the current snapshot of the file. |
|
I now got the example working, I still think my comment is not fully voided by this. I am running a k8s 1.4.2 local cluster via docker. This funnily selects quite old components, esp. the DNS component is outdated / not working and throwing a lot of messages. I did initially not see them as the cluster was behaving quite well. So really, DNS was the first service that fully depended on a perfectly working dns server.
so, with caveats, the example is working ... |
|
… On Thu, Jan 5, 2017 at 11:55 AM, Joshua Sindy ***@***.***> wrote:
@jingxu97 <https://github.com/jingxu97> know where I can track down the
Dockerfile for gcr.io/google_containers/volume-nfs:0.8
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#24687 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ASSNxa9m5o4JwnPUeOB7fV47xFVFapXgks5rPUqzgaJpZM4IN-T7>
.
--
- Jing
|
|
@jingxu97 Thanks a lot for poiting those lines out, just tought of letting people know that it doesnt work on GCE kubernetes 1.4.6 only on 1.4.7, and when listing this nfs disk (df -h) on the pod it shows the size of the original disk and not the size set on the nfs-pvc. |
|
only to inform that i was able to use the nfs example in GKE node version 1.4.8 but only when using Node image container-vm. If i tried to use with node image gci it doesn't work, giving this information in event history of the pod |
|
From your log, I think you are trying to use NFSv3 which is currently not
supported. Please use "/" instead of "/exports" for the mount path so that
you can use NFSv4. Please let me know if you have any issue with that.
Thanks!
…On Wed, Jan 25, 2017 at 3:38 AM, Andre Freitas ***@***.***> wrote:
only to inform that i was able to use the nfs example in GKE node version
1.4.8 but only when using Node image container-vm. If i tried to use with
node image gci it doesn't work, giving this information in event history of
the pod
18m 5m 7 {kubelet gke-qamar-n1-standard2-55a0bb05-wcqw} Warning FailedMount Unable to mount volumes for pod "frontoffice-rc-39.0-u7vp7_default(ebd46ce9-e253-11e6-b119-42010a84013c)": timeout expired waiting for volumes to attach/mount for pod "frontoffice-rc-39.0-u7vp7"/"default". list of unattached/unmounted volumes=[nfsvol]
18m 5m 7 {kubelet gke-qamar-n1-standard2-55a0bb05-wcqw} Warning FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "frontoffice-rc-39.0-u7vp7"/"default". list of unattached/unmounted volumes=[nfsvol]
20m 4m 15 {kubelet gke-qamar-n1-standard2-55a0bb05-wcqw} Warning FailedMount MountVolume.SetUp failed for volume "kubernetes.io/nfs/ebd46ce9-e253-11e6-b119-42010a84013c-client-nfs-pv" (spec.Name: "client-nfs-pv") pod "ebd46ce9-e253-11e6-b119-42010a84013c" (UID: "ebd46ce9-e253-11e6-b119-42010a84013c") with: mount failed: exit status 32
Mounting command: /home/kubernetes/bin/mounter
Mounting arguments: 10.100.245.245:/exports /var/lib/kubelet/pods/ebd46ce9-e253-11e6-b119-42010a84013c/volumes/kubernetes.io~nfs/client-nfs-pv nfs []
Output: Running mount using a rkt fly container
run: group "rkt" not found, will use default gid when rendering images
mount.nfs: rpc.statd is not running but is required for remote locking.
mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
mount.nfs: an incorrect mount option was specified
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#24687 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ASSNxVQ2aoCXt9yrnQFGz4p58yFW_ZQDks5rVzQpgaJpZM4IN-T7>
.
--
- Jing
|
|
@jingxu97, using path '/' gives a different error: For the client container, the error is still the same: |
|
@avra911 the following are the changes I made to make the test /examples/nfs work Edit examples/volumes/nfs/nfs-pv.yaml Edit examples/volumes/nfs/nfs-server-rc.yaml Your error message has this Temporary failure in name resolution. Are you using IP in volume spec for the client pod? You could also share us your yaml file so we can help take a look. |
|
Hello, I will try today once again exactly the files from the example. If it works, I will double check my files and come back here for more help. @jingxu97 , quick question, does it matter how the cluster is created, because I am starting with EDITED: The files from the example works out of the box, without any modifications to the image or path. Tested on 1.5.2 (server and client) on a cluster created with Thank you! |
|
Also having issues. I have made the suggested edits above, but the busybox pod can not mount the pvc: This is on minikube 0.16, kube 1.5.2 Also - is there a tracking bug for |
|
@avra911 sorry that I missed your message. It does not matter how the cluster is created, I think. |
|
All users, now NFSv3 is also supported on GKE, please give it a try and let us know if there is any problem. Thanks! |
|
We moved the examples to their own repo (https://github.com/kubernetes/examples) for further maintenance. However such popular examples to host their own repo for maintenance, and/or convert into Helm charts. It also looks like this issue is now fixed, @jingxu97 should we close now? |
|
@jingxu97 we just migrated to the new cos due to the deprecation of container-vm and I can confirm that NFS is still not working with us. We keep getting the |
|
@kwiesmueller Could you please provide more details about your NFS setup so that we can help figure out what the problem is. Thanks! |
|
@jingxu97 sure thing! The Volume for the Server is a Google PD. The Clients are using the Server like this: Oh and the NFS Server IP is fixed in the Service: Oh and there are no errors on the server side and I can not find any in Google Logs as well... |
|
Could you try to use gcr.io/google_containers/volume-nfs:0.8 as the NFS server container image? And also use Path: '/' in client volumes. This makes sure to use NFSv4. |
|
Will try right tomorrow, thanks! |
|
Getting this on the NFS Pod: The client Pod now only gives a timeout, nfs error: |
|
NFS Depl: Client: |
|
Your server points to `- mountPath: /exports` while apache tries to reach `nfs: path: /`; should be the same path, this is one thing I see right now
…---- On Thu, 06 Jul 2017 15:44:48 +0300 Kevin Wiesmüller <notifications@github.com> wrote ----
NFS Depl:
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: fileserver namespace: {{ "NAMESPACE" | env }} labels: app: fileserver spec: replicas: 1 selector: matchLabels: app: fileserver-worker template: metadata: labels: app: fileserver-worker spec: containers: - name: fileserver-server image: 'gcr.io/google_containers/volume-nfs:0.8' imagePullPolicy: IfNotPresent ports: - name: nfs containerPort: 2049 - name: mountd containerPort: 20048 - name: rpcbind containerPort: 111 securityContext: privileged: true resources: limits: cpu: 200m memory: 100Mi requests: cpu: 10m memory: 10Mi volumeMounts: - mountPath: /exports name: files-store livenessProbe: failureThreshold: 3 initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 tcpSocket: port: 2049 timeoutSeconds: 2 readinessProbe: failureThreshold: 1 initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 tcpSocket: port: 2049 timeoutSeconds: 2 volumes: - name: files-store gcePersistentDisk: fsType: "ext4" pdName: "{{ "NFS_PD_NAME" | env }}" affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: cloud.google.com/gke-preemptible operator: DoesNotExist
Client:
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: app namespace: {{ "NAMESPACE" | env }} labels: app: app spec: replicas: 2 revisionHistoryLimit: 1 selector: matchLabels: app: app-worker template: metadata: labels: app: app-worker spec: containers: - name: app-apache image: 'eu.gcr.io/project/app:{{ "VERSION" | env }}' imagePullPolicy: IfNotPresent # command: ["tail", "-f", "/var/log/dpkg.log"] ports: - name: gt containerPort: 8080 - name: api containerPort: 8085 resources: limits: # cpu: 4 memory: 2Gi requests: cpu: 0.1 memory: 0.5Gi volumeMounts: - mountPath: /mnt name: file-store volumes: - name: file-store nfs: server: {{ "NFS_SERVER_IP" | env }} path: / affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: cloud.google.com/gke-preemptible operator: DoesNotExist
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
Never mind... Running now... |
|
Thanks @jingxu97. I got it working with your client-side method instead of the PVC from example, which never matched the PV. Do you know if there is a way to provide input on k8s direction to help prioritize? I'd like to see k8s provide a purely dynamic, including elastic scaling, with no single points of failure for a horizontally clustered database. This NFS solution, with a static IP in the client YAML to a single point of failure, the node running nfs-server, appears to be a a work-around until that can be achieved. Setting the NFS patch solution aside, I love k8s and am very hopeful its direction will address these use cases. |
|
@erik777 You mentioned PVC never matches PV. Do you know the reason for it? I am not sure I understand your second question. Could you please give me more details? Thanks! |
|
@jinkxu97 I could not figure out the reason. It does not produce an error. Is there a way to diagnose the matching logic of PVCs on GKE? I think I found the answer to #2, how to contribute to direction: https://github.com/kubernetes/community Heck, I even found Priority column in this spreadsheet. lol |
|
The NFS example should be updated now to work on the latest version of K8s. Can this be closed? |
|
Yes, it is fixed with kubernetes/examples#30. Close this issue |
and others
The nfs example at /examples/nfs does not work on google container engine.
The nfs-server runs but the nfs-busybox won't mount the PersistentVolumeClaim.
The busybox pod errors out trying to mount the persistent volume claim.
The nfs-pv uses the nfs-server service ip. Both the persistent volume and the persistent volume claim are bound.
I did notice the nfs server logged a warning.
I have tried exposing additional ports 111 tcp/udp 2049/udp but that had no effect.
Please help.
The text was updated successfully, but these errors were encountered: