docs(proposal) DNS using Envoy's DNSFilter #1488

lahabana · 2021-01-28T14:46:06Z

No description provided.

nickolaev

I think it is a good start. Please be careful with how this will operate on Universal and not breaking the "normal" resolving.

docs/proposals/dns-using-envoy.md

nickolaev · 2021-01-28T18:22:20Z

docs/proposals/dns-using-envoy.md

+
+* The top-level domain should be configurable (defaults to `mesh`)
+* Drop in replacement to Kuma DNS (Default configuration should match Kuma's current configuration)
+* ability to set upstream dns resolvers to have a single resolver in the configuration


How is this going to work in the Universal deployments? Today with Transparent Proxying, we do edit /etc/resolv.conf, which is tricky, yet we preserve the original servers.

I think you'd do the same (update the /etc/resolv.conf).

For example in our universal deployment we set an iptable rule to direct to whatever address you'd like .e.g: -A CNI-o-631b6b8422a8498934cee4 -d 192.168.0.53/32 -p udp -m udp --dport 53 -j REDIRECT --to-ports <passthrough-outbound-port> and the /etc/resolv.conf to this as well.

nickolaev · 2021-01-28T18:24:22Z

docs/proposals/dns-using-envoy.md

+  domain: "kuma" # [Optional defaults to "mesh"] the TLD to use for services exposed with the DNS
+  upstreamResolvers: # [Optional] A list of resolvers to use when we resolving names not in the TLD
+    - "8.8.8.8"
+    - "4.4.4.4"


Ahem, in K8s we'll have to add here also whatever ends up in /etc/resolv.conf. We do not want to break the normal K8s service resolving, i.e. resolving to Cluster IPs

I think you can do either:

leave that empty and rely on the rest of the /etc/resolv.conf

just rely on this and have envoy in the /etc/resolv.conf (You could be hardcore and redirect all :53 to envoy but that'd be the nuclear option :)).

I was also expecting that our Envoy will just do the resolving of Kuma domains.

Do you need this option @lahabana for your deployment?
If not, I'd consider not including this for now

No we don't I'll remove it then.

docs/proposals/dns-using-envoy.md

nickolaev · 2021-01-28T18:28:00Z

docs/proposals/dns-using-envoy.md

+- The default configuration will make DNS names similar to what is present by default in Kuma
+- Users pick the address and port at which the DNS server runs, thus being able to run both setups concurrently
+
+In a future version it will be possible to have a default DNS policy to replace Kuma DNS altogether.


I like that. We may make this in-Envoy DNS as a 'beta' feature until proven stable and then deprecate the 'old' way.

jakubdyszkiewicz

Nice, thank you for this proposal.

The only thing that I'm missing is how are we planning to deploy this on Kubernetes?
Right now we are just changing configuration of CoreDNS for the whole cluster which points DNS to the control plane. In this case I think we need to modify /etc/resolve.conf on every machine when Kuma is injected? Is it possible?

jakubdyszkiewicz · 2021-01-29T11:52:54Z

docs/proposals/dns-using-envoy.md

+  upstreamResolvers: # [Optional] A list of resolvers to use when we resolving names not in the TLD
+    - "8.8.8.8"
+    - "4.4.4.4"
+  address: "169.254.53.53" # [Optional defaults to 0.0.0.0] the address on which the UDP listener should bind in envoy


I think we should always bind this listener to Dataplane#networking.address

It's possible I just thought I'd keep it flexible but we can always add it later if it's needed.

jakubdyszkiewicz · 2021-01-29T11:54:00Z

docs/proposals/dns-using-envoy.md

+    - "8.8.8.8"
+    - "4.4.4.4"
+  address: "169.254.53.53" # [Optional defaults to 0.0.0.0] the address on which the UDP listener should bind in envoy
+  port: "53" # [Optional defaults to 53 or redirectOutbound if transparent proxying is set] the port on which the UDP listener should bind in envoy


or redirectOutbound if transparent proxying is set

How would this work? RedirectOutoubnd port will be already taken, you cannot bind 2 listeners to it as far as I know.

Also if we are picking port, there probably should be an option to override this for one dataplane. Just like we do with prometheus endpoint.

Because redirectOutbound is currently only TCP we can add UDP without problem (DNSFilter only works on UDP DNS).

Also if we are picking port, there probably should be an option to override this for one dataplane. Just like we do with prometheus endpoint.

I guess that would be an extra config in the dataplane spec?

Because redirectOutbound is currently only TCP we can add UDP without problem (DNSFilter only works on UDP DNS).

What is the problem that prevents us from binding to any other port? Is it about iptables? I thought we are touching only TCP traffic with iptables.

I guess that would be an extra config in the dataplane spec?

correct

What is the problem that prevents us from binding to any other port? Is it about iptables? I thought we are touching only TCP traffic with iptables.

If by any you mean any arbitrary port, nothing prevents you from doing this. Would you want to always default to 53 regardless or transparent proxying or not?

jakubdyszkiewicz · 2021-01-29T11:57:42Z

docs/proposals/dns-using-envoy.md

+    - "4.4.4.4"
+  address: "169.254.53.53" # [Optional defaults to 0.0.0.0] the address on which the UDP listener should bind in envoy
+  port: "53" # [Optional defaults to 53 or redirectOutbound if transparent proxying is set] the port on which the UDP listener should bind in envoy
+  tags: ["kuma.io/protocol", "kuma.io/service"] # [Optional defaults to ["kuma.io/service"]] A list of tags to use to build domnains they are joined with "." for example: "http.my-service.mesh" would be a final name.


How about instead of tags + domain we have one field called template that would look like this

template: "{{ kuma.io/protocol}}.{{kuma.io/service}}.mesh"

This way you can have more powerful mechanism of defining DNS name.

You could just replace it with string.Replace or execute full blown gotemplate which would give even more options like

template: "{{ kuma.io/service | replace '_' '.' }}.mesh" What od you think?

I think this would be very cool. The only thing we lose with this is the ability for DNSFilter to ignore all requests not within a TLD, though I'm not sure it changes much TBH

jakubdyszkiewicz · 2021-01-29T12:02:30Z

docs/proposals/dns-using-envoy.md

+  domain: "kuma" # [Optional defaults to "mesh"] the TLD to use for services exposed with the DNS
+  upstreamResolvers: # [Optional] A list of resolvers to use when we resolving names not in the TLD
+    - "8.8.8.8"
+    - "4.4.4.4"


I was also expecting that our Envoy will just do the resolving of Kuma domains.

Do you need this option @lahabana for your deployment?
If not, I'd consider not including this for now

lahabana · 2021-01-29T13:39:00Z

The only thing that I'm missing is how are we planning to deploy this on Kubernetes?
Right now we are just changing configuration of CoreDNS for the whole cluster which points DNS to the control plane. In this case I think we need to modify /etc/resolve.conf on every machine when Kuma is injected? Is it possible?

So here you are touching my lack of familiarity of Kuma on k8s.
Here are some possible solutions:

Add a PodDNSConfig which would enable you to set a different DNS than coreDNS.
You could also inject this in the Admission webhook like we override probes currently.
The one issue with this is that /etc/resolv.conf only allows port 53.
Do everything with iptables and have envoy "replace" coreDNS in the init-container. Istio does this code.

Now this makes me think again about upstreamResolvers, maybe it makes sense from a user point of view to define the DNS for the mesh in the service mesh configuration instead of a per deployment platform configuration (.e.g you'd need to configure DNS in K8s or in whatever Universal is for you).
WDYT?

jakubdyszkiewicz · 2021-02-01T08:56:06Z

If it's possible to override DNSConfig in such a way to redirect only .mesh domain to Envoy then I think I'd go this way.
I think the less "intrusive" our implementation is the better.

@nickolaev thoughts?

lahabana · 2021-02-02T08:40:30Z

The way I see this possible is by appending Envoy as a resolver to the nameservers seems like a DNSConfig with a DNSPolicy and Envoy DNS always listening on the same IP:53 would do the trick.
We should probably do this setting in the pod_controller no?
I can update the doc with this if @nickolaev is ok with it.

nickolaev · 2021-02-02T09:07:47Z

The way I see this possible is by appending Envoy as a resolver to the nameservers seems like a DNSConfig with a DNSPolicy and Envoy DNS always listening on the same IP:53 would do the trick.
We should probably do this setting in the pod_controller no?
I can update the doc with this if @nickolaev is ok with it.

Let's put iptables aside for now. Please proceed with the DNSConfing in the proposal. I guess we are mostly aligned on everything else?

lahabana · 2021-02-02T17:03:43Z

Agreed I'll update the doc to make it ready for a final review with the details around CoreDNS

Signed-off-by: Charly Molter <charly@koyeb.com>

lahabana · 2021-02-04T12:54:05Z

@nickolaev @jakubdyszkiewicz updated with the latest conversations. If we're all ok with this I'll start the work on this probably tomorrow and early next week.

nickolaev · 2021-02-04T15:38:54Z

@nickolaev @jakubdyszkiewicz updated with the latest conversations. If we're all ok with this I'll start the work on this probably tomorrow and early next week.

Perfect, I think you can start with the implementation. I am sure there will be corner cases we'll find down the road. This is fine, as I don't expect we can foresee everything without touching the code.
Great stuff!

lahabana requested a review from a team as a code owner January 28, 2021 14:46

lahabana mentioned this pull request Jan 28, 2021

DNS for overlapping kuma.io/service in different meshes #1450

Closed

nickolaev reviewed Jan 28, 2021

View reviewed changes

jakubdyszkiewicz reviewed Jan 29, 2021

View reviewed changes

Charly Molter added 2 commits February 4, 2021 13:33

docs(proposal) DNS using Envoy's DNSFilter

753bdc8

Signed-off-by: Charly Molter <charly@koyeb.com>

Review comments

38b1e66

Signed-off-by: Charly Molter <charly@koyeb.com>

lahabana force-pushed the docs/envoy-dns branch from 3771d70 to 0ca95bf Compare February 4, 2021 12:52

Review comments

28cb8d3

Signed-off-by: Charly Molter <charly@koyeb.com>

lahabana force-pushed the docs/envoy-dns branch from 0ca95bf to 28cb8d3 Compare February 4, 2021 12:53

nickolaev approved these changes Feb 4, 2021

View reviewed changes

jakubdyszkiewicz approved these changes Feb 5, 2021

View reviewed changes

nickolaev merged commit d894008 into kumahq:master Feb 5, 2021

lahabana deleted the docs/envoy-dns branch March 29, 2024 12:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs(proposal) DNS using Envoy's DNSFilter #1488

docs(proposal) DNS using Envoy's DNSFilter #1488

lahabana commented Jan 28, 2021

nickolaev left a comment

nickolaev Jan 28, 2021

lahabana Jan 28, 2021

nickolaev Jan 28, 2021

lahabana Jan 28, 2021

jakubdyszkiewicz Jan 29, 2021

lahabana Jan 29, 2021

nickolaev Jan 28, 2021

jakubdyszkiewicz left a comment

jakubdyszkiewicz Jan 29, 2021

lahabana Jan 29, 2021

jakubdyszkiewicz Jan 29, 2021

jakubdyszkiewicz Jan 29, 2021

lahabana Jan 29, 2021

lahabana Jan 29, 2021

jakubdyszkiewicz Feb 1, 2021

lahabana Feb 1, 2021

jakubdyszkiewicz Jan 29, 2021

lahabana Jan 29, 2021

jakubdyszkiewicz Jan 29, 2021

lahabana commented Jan 29, 2021

jakubdyszkiewicz commented Feb 1, 2021

lahabana commented Feb 2, 2021

nickolaev commented Feb 2, 2021

lahabana commented Feb 2, 2021

lahabana commented Feb 4, 2021

nickolaev commented Feb 4, 2021

docs(proposal) DNS using Envoy's DNSFilter #1488

docs(proposal) DNS using Envoy's DNSFilter #1488

Conversation

lahabana commented Jan 28, 2021

nickolaev left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jakubdyszkiewicz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lahabana commented Jan 29, 2021

jakubdyszkiewicz commented Feb 1, 2021

lahabana commented Feb 2, 2021

nickolaev commented Feb 2, 2021

lahabana commented Feb 2, 2021

lahabana commented Feb 4, 2021

nickolaev commented Feb 4, 2021