-
Notifications
You must be signed in to change notification settings - Fork 38.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
volumes/nfs example: service name instead hardcoded IP #44528
Comments
Old issue here: #8735 . The message in the README is about as old. I would be interested in this too :) |
CC @kubernetes/sig-storage-feature-requests |
Issues go stale after 90d of inactivity. Prevent issues from auto-closing with an If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or |
Stale issues rot after 30d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or |
Ran into this, myself, while attempting to configure my storageclasses to speak to a heketi glusterfs pod by service name. When I rebooted my cluster, the cluster IP changed, which broke my glusterfs storage solution. |
I believe the issue is the node's resolv.conf needs to be configured to point to Kubernetes' dns service. This is a host configuration that needs to be done per deployment. I believe we do it automatically for GCE/GKE but I'm unsure about other environments. cc @jingxu97 |
So how would I setup the resolve.conf files on each node to resolve to
kube-dns? The service for it is setup as a ClusterIP service, and in a CNI
environment like flannel, that becomes non accessible from outside of a
container.
Can I edit the service to set it up as a NodePort and then direct
resolve.conf to every node in the cluster?
…On Feb 4, 2018 9:41 AM, "Michelle Au" ***@***.***> wrote:
I believe the issue is the node's resolv.conf needs to be configured to
point to Kubernetes' dns service.
This is a host configuration that needs to be done per deployment. I
believe we do it automatically for GCE/GKE but I'm unsure about other
environments. cc @jingxu97 <https://github.com/jingxu97>
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#44528 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAkVBXltTyCzpb8DYn0gh9DheDhoy9xJks5tRevQgaJpZM4M-X4Q>
.
|
Hm maybe @kubernetes/sig-network-misc knows something that can be done here. The problem is volume mounts are done by kubelet, so the nfs server IP/hostname needs to be accessible to kubelet's network. |
Which on a baremetal deployment should be the node's/host's network, correct? |
Correct. So if your nfs server is being provided by a Pod, then you need the node/host network to be able to access the pod's network, which like you pointed out, could be tricky depending on how you've configured your network. |
Okay, so I see two solutions to this problem, then:
Of the two, the first seems like the more generic solution for getting name resolution working across the cluster, but may have unintended side-effects if things are setup to expect it as a We may want to update the public facing docs to mention that |
Did you have a chance to experiment a bit ? For solution 2), Nodeport define ports in the 30000-32767 range, so it seems also necessary to modify nfs PV default ports ? (2049 & 111) I have a baremetal setup with flannel, and editing resolv.conf doesn't work, because like you pointed out, my nodes don't have access to container's network. |
Unfortunately, I can't /edit/ the restUrl for my storageclass because "updating parameters is illegal"... I may end up losing data by deleting the storage class and recreating it with the right url. |
Fortunately my analysis was wrong -- didn't lose any data at all, thankfully. So what I've done is update my Services to be NodePorts, exposed them on port 32708, and set my resturls in the storageclasses to
But things work at least. |
@jtgans Sorry that I might be missing the point, why rebooting cluster changes the cluster IP? Did the NFS service got deleted and recreated? If so, what about giving the NFS service a fixed IP in manifest and keeping it as type=ClusterIP? |
I actually didn't realize that ClusterIP services could specify their IPs in the definition. I'll give this a try today. Pretty sure I didn't recreate the service post reboot, but it's been a while since I restarted the cluster. |
use full service name, it works fine:
|
Rotten issues close after 30d of inactivity. Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
@mtricolici Tried. Didn't work. |
@mtricolici @digglife 'build' should be replaced by the namespace. |
I have also tried using the service name, and that did not work. Can you clarify the kubectl command that gets the "full" service name that resolves? |
Take that back. I just got it to work using "{service-name}.{namespace}.svc.cluster.local". I did not realize that svc.cluster.local was always the same. |
I just tried e.g. service name I got
They are still no fix or clean workaround available? |
I'm also having this issue on EKS. |
Also having this issue on Digital Ocean. |
/reopen |
@rjohnson3: You can't reopen an issue/PR unless you authored it or you are a collaborator. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Also having this issue on GKE |
Same issue on OpenStack. |
Same issue on NFS |
I'm also having this issue on AKS. |
@will-beta: You can't reopen an issue/PR unless you authored it or you are a collaborator. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Same issue on CDK. |
From what I just learned this is the issue only on non-GKE Kubernetes. Can't wait for upstream fix.so we can get proper service DNS name resolution on all providers. Any progress perhaps? |
Worth noting that the example highlighted here is now at: https://github.com/kubernetes/examples/tree/master/staging/volumes/nfs |
Yeah, regardless, the question still applies as there was no upstream solution at the time of the docs writing. What does GKE does that allow for this different behavior from upstream Kubernetes? |
has there been any work on this ? |
Did anyone found a solution for this? It really surprises me that this (imo big) issue hasn't been resolved in 3 years? Any suggestions on how to workaround this? |
I'm not sure but could be using the ExternalName service be a viable solution? For example, using NFS server service DNS name in the externalName property? I was under impression that this particular object was created to solve these issues. I didn't try it yet but would welcome feedback from those who did, regardless of the outcome. |
FWIW I have worked around this by using the nfs-server-provisioner helm chart, persistent volume claims, and moving on with my life. I will say, something has changed about helm's website and now there seem to be two (identical?) options for this, which is a bit weird. https://artifacthub.io/packages/helm/kvaps/nfs-server-provisioner Hope that helps! It has worked well enough for me! It would definitely be nice to have a fix though! A coworker dug into the source and suspected the bug was here, in case it is any help: https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/nfs/nfs.go#L256 |
The issue is in some environments, kubelet's host network does not have access to the cluster dns. Using https://github.com/kubernetes-csi/csi-driver-nfs should resolve this because it runs as a Pod so has access to cluster services. |
This does not appear to us as far as I understand it, when I connect via shell I can ping the nfs server directly via |
As @msau42 mentioned, I solved this issue using the https://github.com/kubernetes-csi/csi-driver-nfs |
Kubernetes version (use
kubectl version
):Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.1", GitCommit:"82450d03cb057bab0950214ef122b67c83fb11df", GitTreeState:"clean", BuildDate:"2016-12-14T00:57:05Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.6", GitCommit:"114f8911f9597be669a747ab72787e0bd74c9359", GitTreeState:"clean", BuildDate:"2017-03-28T13:36:31Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
Environment:
What happened:
I've tried this great nfs example here
I try to use service name instead of hardcoded IP in PersistentVolume configuration and it is not work with "Failed to resolve server nfs-server.default.svc.cluster.local: Temporary failure in name resolution" in pod status. At the same time I can ping this FQDN (and just host nfs-server) from other containers.
I saw annotation in README.md:
So, what is your plans for this feature?
The text was updated successfully, but these errors were encountered: