New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DNS Inconsistencies #216

Closed
number101010 opened this Issue Jul 4, 2017 · 8 comments

Comments

Projects
None yet
2 participants
@number101010

number101010 commented Jul 4, 2017

Seeing the below results while using telepresence on OSX 10.11.6

username@work k8s-prometheus-autoscaler (master) $ telepresence --method=vpn-tcp --run-shell
Starting proxy...
@minikube|username@work k8s-prometheus-autoscaler $ nslookup prometheus
Server:		<home dns server>
Address:	<home dns server>#53

Name:	prometheus.default.svc.cluster.local
Address: 10.0.0.192

@minikube|username@work k8s-prometheus-autoscaler $ nslookup prometheus.default.svc.cluster.local
Server:		<home dns server>
Address:	<home dns server>#53

Non-authoritative answer:
Name:	prometheus.default.svc.cluster.local
Address: 10.0.0.192

@minikube|username@work k8s-prometheus-autoscaler $ curl http://prometheus:9090
curl: (6) Could not resolve host: prometheus

@minikube|username@work k8s-prometheus-autoscaler $ curl http://prometheus.default.svc.cluster.local:9090
<a href="/graph">Found</a>.

@minikube|username@work k8s-prometheus-autoscaler $ curl http://10.0.0.192:9090
<a href="/graph">Found</a>.

Some things I noticed while debugging:

  • nslookup lists my home dns server name and yet still returns valid in cluster (minikube) results
  • nslookup returns valid values for prometheus and prometheus.default.svc.cluster.local
  • curl only can resolve the fully qualified name. this is consistent with results seen while debugging an application through telepresence and using dscacheutil through telepresence
  • Neither my machine nor the container's /etc/hosts file make any reference to prometheus

This behavior is the opposite of what's discussed in the documentation here: http://www.telepresence.io/reference/proxying.htm

Maybe a difference between how osx and linux resolve names?

@itamarst

This comment has been minimized.

Contributor

itamarst commented Jul 5, 2017

On OS X .local domains do work with vpn-tcp mode, yes (docs mention that not supporting .local is Linux-specific).

The question is why curl breaks...

  1. Could you attach a telepresence.log from the session above or a similar session?
  2. Does curl http://prometheus.default:9090 work?

Meanwhile I'll try to reproduce that on mac we have here.

@itamarst

This comment has been minimized.

Contributor

itamarst commented Jul 5, 2017

OK, I am able to reproduce locally. So this suggests current test suite isn't sufficient, since it doesn't catch this. Possibly it should use curl too.

@itamarst

This comment has been minimized.

Contributor

itamarst commented Jul 5, 2017

OK, figured it out I think: currently the DNS proxy server includes the full domain name in response, rather than original domain name, and some DNS resolvers seem to not like that. Will write reproducer test and then bug fix next.

@itamarst itamarst referenced this issue Jul 5, 2017

Merged

Curl dns broken #218

@itamarst itamarst closed this in #218 Jul 5, 2017

itamarst added a commit that referenced this issue Jul 5, 2017

Merge pull request #218 from datawire/curl-dns-broken
Unbreak Curl DNS on macOS.

Fixes #216.
@itamarst

This comment has been minimized.

Contributor

itamarst commented Jul 5, 2017

Telepresence v0.57 should be available in half an hour: can you try out and see if it fixes your problem?

@number101010

This comment has been minimized.

number101010 commented Jul 6, 2017

Wow man. Thanks very much for the quick fix. Worked like a charm.

I've converted my current k8s project over to using telepresence for debugging. Loving it.

@itamarst

This comment has been minimized.

Contributor

itamarst commented Jul 6, 2017

Glad it's working! Could you tell us a bit more about your use cases for Telepresence?

@number101010

This comment has been minimized.

number101010 commented Jul 7, 2017

Sure. I recently got involved in go development through docker/kubernetes. I initially put this together to dev:

https://github.com/number101010/go-minikube-vscode-dev

It pushes your code and dlv into a minikube cluster and then vscode connects to the remote debugging port. Unfortunately, it's slow, clunky and often hangs removing the old pod. Then I found telepresence and put this together:

https://github.com/number101010/go-kubernetes-vscode-dev

Much faster but comes with all the caveats you're aware of. I need to find a way to use chroot or something along with $TELEPRESENCE_ROOT to make debugging that much more seamless. Definitely a template I want to continue improving.

Then my current real project is:

https://github.com/number101010/kubernetes-prometheus-scaler

It scales deployments based on the results of prometheus queries.

@itamarst

This comment has been minimized.

Contributor

itamarst commented Jul 10, 2017

That sounds interesting! Would you be willing to talk to us on the phone/hangouts or something to get a better sense of your use cases?

Separately, if you email me your address and t-shirt size we'd be happy to send you a t-shirt as a thank you for the bug report and debugging help - itamar@datawire.io.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment