pkg/srv, embed, etcdmain: Support multiple clusters in the same DNS domain #8690

tavish-stripe · 2017-10-12T20:52:46Z

This PR adds the ability to configure the subdomain used for DNS cluster discovery. Previously, the domain was hardcoded to _etcd-server._tcp.FOO and _etcd-server-ssl._tcp.FOO. This did not work with our DNS setup, so I've made this patch to allow configuring the subdomain portion of the SRV queries.

We've been running this in production for a few months.

codecov-io · 2017-10-12T21:48:56Z

Codecov Report

Merging #8690 into master will decrease coverage by 0.03%.
The diff coverage is 19.04%.

@@            Coverage Diff             @@
##           master    #8690      +/-   ##
==========================================
- Coverage   75.87%   75.84%   -0.04%     
==========================================
  Files         363      363              
  Lines       30161    30168       +7     
==========================================
- Hits        22884    22880       -4     
- Misses       5670     5689      +19     
+ Partials     1607     1599       -8

Impacted Files	Coverage Δ
embed/config.go	`59.85% <0%> (-3.47%)`	⬇️
etcdmain/config.go	`81.5% <100%> (+0.1%)`	⬆️
pkg/srv/srv.go	`79.03% <66.66%> (+5.79%)`	⬆️
etcdserver/api/v3rpc/lease.go	`78.37% <0%> (-8.11%)`	⬇️
clientv3/leasing/util.go	`91.66% <0%> (-6.67%)`	⬇️
clientv3/namespace/watch.go	`93.93% <0%> (-6.07%)`	⬇️
etcdctl/ctlv3/command/lease_command.go	`65.34% <0%> (-5.95%)`	⬇️
etcdserver/api/v3rpc/watch.go	`80.68% <0%> (-4.73%)`	⬇️
clientv3/concurrency/session.go	`88.63% <0%> (-4.55%)`	⬇️
clientv3/leasing/txn.go	`88.09% <0%> (-3.18%)`	⬇️
... and 15 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 37f577c...81c9f78. Read the comment docs.

xiang90 · 2017-12-22T02:29:07Z

/cc @hexfusion can you take a look of this PR?

hexfusion · 2017-12-22T02:32:58Z

@xiang90, 👍

tavish-stripe · 2017-12-22T17:50:00Z

@xiang90 / @hexfusion Thanks for taking a look. Let me know how I can help. If this overall approach looks good I'm happy to rebase on master to resolve the conflict.

xiang90 · 2017-12-23T00:13:44Z

etcdmain/config.go

@@ -152,6 +152,8 @@ func newConfig() *config {

 	fs.StringVar(&cfg.Dproxy, "discovery-proxy", cfg.Dproxy, "HTTP proxy to use for traffic to discovery service.")
 	fs.StringVar(&cfg.DNSCluster, "discovery-srv", cfg.DNSCluster, "DNS domain used to bootstrap initial cluster.")
+	fs.StringVar(&cfg.DNSClusterServiceNameSSL, "discovery-srv-name-ssl", cfg.DNSClusterServiceNameSSL, "Service name to query when using DNS discovery for SSL")


Can we reduce the configurable flag down to one? The goal is to be able to have N etcd discovery under one domain, not to make the ssl suffix configurable?

Yes we already know if etcd is using TLS at this point right, how can we get this information from etcd? Can you name your etcd service _etcd-server-ssl then in the case of cluster 'a' cumulative SRV would be _etcd-server-ssl-a._tcp.service.consul. using --discovery-srv-name a?

xiang90 · 2017-12-23T00:14:15Z

@tavish-stripe I am OK with the motivation.

hexfusion · 2017-12-23T15:56:44Z

@tavish-stripe thank you for your contribution! Could you do me a favor and create an issue for this PR and briefly outline the limitations you faced with your DNS setup and how this PR resolved it? You could copy and paste a lot of what you have above. Would be good to track the issue while we work through this. I hope to have sometime to review after the holiday weekend. Thanks again, Happy Holidays!

hexfusion · 2018-01-02T02:54:39Z

Previously, the domain was hardcoded to _etcd-server._tcp.FOO and _etcd-server-ssl._tcp.FOO. This did not work with our DNS setup

@tavish-stripe, just checking in. I am trying to better understand the issues you faced with your DNS setup. For example, why does this type of setup not work for you? It would help me a lot to get some context to why this change is necessary.

cluster 1

-–discovery-srv=foo.local

;SRV Records
_etcd-server-ssl._tcp.foo.local. 300 IN SRV 0 0 2380 node1.foo.local.
_etcd-server-ssl._tcp.foo.local. 300 IN SRV 0 0 2380 node2.foo.local.
_etcd-server-ssl._tcp.foo.local. 300 IN SRV 0 0 2380 node3.foo.local.

_etcd-client-ssl._tcp.foo.local. 300 IN SRV 0 0 2379 node1.foo.local.
_etcd-client-ssl._tcp.foo.local. 300 IN SRV 0 0 2379 node2.foo.local.
_etcd-client-ssl._tcp.foo.local. 300 IN SRV 0 0 2379 node3.foo.local.

cluster 2

-–discovery-srv=sub.foo.local

;SRV Records
_etcd-server-ssl._tcp.sub.foo.local. 300 IN SRV 0 0 2380 node4.sub.foo.local.
_etcd-server-ssl._tcp.sub.foo.local. 300 IN SRV 0 0 2380 node5.sub.foo.local.
_etcd-server-ssl._tcp.sub.foo.local. 300 IN SRV 0 0 2380 node6.sub.foo.local.

_etcd-client-ssl._tcp.sub.foo.local. 300 IN SRV 0 0 2379 node4.sub.foo.local.
_etcd-client-ssl._tcp.sub.foo.local. 300 IN SRV 0 0 2379 node5.sub.foo.local.
_etcd-client-ssl._tcp.sub.foo.local. 300 IN SRV 0 0 2379 node6.sub.foo.local.

hexfusion · 2018-01-08T22:51:55Z

ping @tavish-stripe :)

tavish-stripe · 2018-01-09T03:09:13Z

@hexfusion thanks for the ping!

With our internal service configuration setup everything is under the same foo.local domain and we can't tack on an extra subdomain on a per-cluster basis, so being able to configure the subdomain purpose is useful for us.

hexfusion · 2018-01-10T15:59:16Z

@tavish-stripe thank you for the reply, so if I understand you correctly your saying that your DNS setup somehow is not configurable to allow sub domains for SRV records. So that all SRV records are defined as the root domain?

So as a work around you are passing the SRV name to etcd to circumvent this? I think that is a crafty solution :) but at that point we are hacking the DNS results which are supposed to be defining the discovery. So this would be a hybrid DNS discovery mechanism one that uses parts of the SRV records and manually overriding others. Please feel free to chime in if my assessment is wrong.

Before we go down this route I would like to propose an alternative discovery mechanism based on TXT records and see if it would be useful for others. I believe it could also solve your problem? See #9128

/cc @gyuho @xiang90

tavish-stripe · 2018-01-18T16:24:24Z

@hexfusion sorry for the delay.

Since we're using Consul for service discovery internally, it's very easy for us to register each etcd cluster in our consul cluster as a separate service. This allows us to use Consul's automatically generated SRV records for each etcd service in the cluster. As a result, we get SRV services registered at _etcd-cluster-1._tcp.service.consul, _etcd-cluster-2._tcp.service.consul, etc. Consul does not, as far as I know, allow you to tack on subdomains per service.

Using SRV discovery (with our own provided service names) allows us to easily use Consul DNS to bootstrap our etcd clusters without having to configure DNS specially for each cluster. The TXT solution you suggested does not allow us to leverage Consul's built-in functionality.

but at that point we are hacking the DNS results which are supposed to be defining the discovery. So this would be a hybrid DNS discovery mechanism one that uses parts of the SRV records and manually overriding others.

It might be helpful to clarify our use case. The solution I'm proposing uses all of the SRV responses and does not override any of them. Here's an example query and response generated by consul. There are two clusters here advertising a client service each (etcd-client-a and etcd-client-b).

$ dig srv _etcd-client-a._tcp.service.consul

;; ANSWER SECTION:
_etcd-client-a._tcp.service.consul. 5 IN  SRV 1 1 2379 etcda-00000000000000000.node.qa-northwest.consul.
_etcd-client-a._tcp.service.consul. 5 IN  SRV 1 1 2379 etcda-11111111111111111.node.qa-northwest.consul.
_etcd-client-a._tcp.service.consul. 5 IN  SRV 1 1 2379 etcda-22222222222222222.node.qa-northwest.consul.

;; ADDITIONAL SECTION:
etcda-00000000000000000.node.qa-northwest.consul. 5 IN A 10.0.0.1
etcda-11111111111111111.node.qa-northwest.consul. 5 IN A 10.0.0.2
etcda-22222222222222222.node.qa-northwest.consul. 5 IN A 10.0.0.3


$ dig srv _etcd-client-b._tcp.service.consul

;; ANSWER SECTION:
_etcd-client-b._tcp.service.consul. 5 IN  SRV 1 1 2379 etcdb-00000000000000000.node.qa-northwest.consul.
_etcd-client-b._tcp.service.consul. 5 IN  SRV 1 1 2379 etcdb-11111111111111111.node.qa-northwest.consul.
_etcd-client-b._tcp.service.consul. 5 IN  SRV 1 1 2379 etcdb-22222222222222222.node.qa-northwest.consul.

;; ADDITIONAL SECTION:
etcdb-00000000000000000.node.qa-northwest.consul. 5 IN A 10.1.0.1
etcdb-11111111111111111.node.qa-northwest.consul. 5 IN A 10.1.0.2
etcdb-22222222222222222.node.qa-northwest.consul. 5 IN A 10.1.0.3

Does that help clarify the use case?

hexfusion · 2018-01-18T18:29:30Z

@tavish-stripe this is very helpful will review shortly. Thanks!

hexfusion · 2018-01-20T13:38:29Z

Since we're using Consul for service discovery internally, it's very easy for us to register each etcd cluster in our consul cluster as a separate service. This allows us to use Consul's automatically generated SRV records for each etcd service in the cluster. As a result, we get SRV services registered at _etcd-cluster-1._tcp.service.consul, _etcd-cluster-2._tcp.service.consul, etc. Consul does not, as far as I know, allow you to tack on subdomains per service.

@tavish-stripe

First lets start with consol, it appears that support for multiple domains is a long standing issue. Support multiple domains per datacenter hashicorp/consul#290. Currently the recommended solution by Armon is to have a separate consul per domain. But as this is a one to many relationship of consul/datacenter to etcd cluster, your usage is really the only option for now.
Second it does seems that using your proposed instance naming is supported by https://tools.ietf.org/html/rfc6763 which is good.

The problem from my experience is is that SRV discovery causes a lot of confusion for many folks. So to @xiang90's point we want to keep this to a single flag for simplicity. We would also need to add documentation and tests.

--discovery-srv-name

will comment on the code now a bit.

hexfusion · 2018-01-20T14:00:46Z

etcdmain/config.go

@@ -152,6 +152,8 @@ func newConfig() *config {

 	fs.StringVar(&cfg.Dproxy, "discovery-proxy", cfg.Dproxy, "HTTP proxy to use for traffic to discovery service.")
 	fs.StringVar(&cfg.DNSCluster, "discovery-srv", cfg.DNSCluster, "DNS domain used to bootstrap initial cluster.")
+	fs.StringVar(&cfg.DNSClusterServiceNameSSL, "discovery-srv-name-ssl", cfg.DNSClusterServiceNameSSL, "Service name to query when using DNS discovery for SSL")


Yes we already know if etcd is using TLS at this point right, how can we get this information from etcd? Can you name your etcd service _etcd-server-ssl then in the case of cluster 'a' cumulative SRV would be _etcd-server-ssl-a._tcp.service.consul. using --discovery-srv-name a?

hexfusion · 2018-01-20T14:15:15Z

embed/config.go

+	var clusterStrs []string
+	var cerr error
+	var serviceName string
+	var scheme string


var ( clusterStrs []string cerr error serviceName string scheme string )?

hexfusion · 2018-01-20T14:21:48Z

@tavish-stripe this is a good idea please rework your logic based on a single flag. Looking forward to getting this into core! Let us know if you have any questions.

tavish-stripe · 2018-01-22T18:56:47Z

@hexfusion that seems like a reasonable solution. One alternative would be to provide the name and have -ssl get added in the TLS case. e.g. -discovery-srv-name=foobar would give _foobar._tcp.service.consul and _foobar-ssl._tcp.service.consul. This feels more flexible and readable to me, but either way solves the problem. What do you think? Happy to rework the patch either way.

hexfusion · 2018-01-22T19:01:56Z

@tavish-stripe I feel the definition should be explicit but to your point flexibility is a consideration.

defer to @xiang90

hexfusion · 2018-01-22T19:24:35Z

Another thing is that if we keep this _etcd-server-ssl-${name} this might make it less confusing as this is the previous standard and we are only appending name. Also a query of clusters might be easier to predefine/grep.

tavish-stripe · 2018-01-22T19:38:36Z

Yeah, that seems like a reasonable argument. I'll wait for @xiang90 's opinion and then rework the patch.

…iscovery region.

hexfusion · 2018-01-24T21:29:36Z

@tavish-stripe since @xiang90 has not added a comment I will assume he has no strong feelings. I would like to keep the existing format etcd-server, etcd-server-ssl +-{name}. Thank you and I look forward to our next review.

xiang90 · 2018-01-24T21:34:13Z

sorry for the delay. i do not have strong feeling. just want to keep the flag down to one, and make it as clear as possible. we probably need to document it, and provide an example in the configuration doc as well.

tavish-stripe · 2018-01-24T22:10:40Z

@hexfusion @xiang90 I've reworked the patch to use the suffix. Please take a look and let me know if anything needs changing.

we probably need to document it, and provide an example in the configuration doc as well.

Do I need to include that as part of this PR in the Documentation/ folder, or is this something that should be handled as a separate PR?

hexfusion · 2018-01-24T22:19:39Z

Do I need to include that as part of this PR in the Documentation/ folder, or is this something that should be handled as a separate PR?

@tavish-stripe please include in this PR as a single commit.

xiang90 · 2018-01-24T22:23:28Z

@tavish-stripe

probably the documentation should be added here https://github.com/coreos/etcd/blob/master/Documentation/op-guide/clustering.md#dns-discovery

tavish-stripe · 2018-01-24T22:31:40Z

@hexfusion @xiang90 I made a first cut of documentation for the new flag. Please let me know if there's anything I should change.

xiang90 · 2018-01-24T22:33:22Z

@tavish-stripe This PR looks good to me in general. I would rely on @hexfusion and @gyuho or @spzala to give a final LGTM before merging.

Thanks.

hexfusion · 2018-01-24T23:20:53Z

Documentation/op-guide/configuration.md

+ Suffix to the DNS srv name queried when bootstrapping using DNS.
+ default: ""
+ env variable: ETCD_DISCOVERY_SRV
+


@tavish-stripe ETCD_DISCOVERY_SRV_NAME ?

hexfusion · 2018-01-24T23:24:06Z

@tavish-stripe one little knit, otherwise looks great!

…service name in DNS discovery.

hexfusion

lgtm

gyuho

lgtm. Thanks @tavish-stripe @hexfusion

hexfusion · 2018-01-25T15:42:18Z

👍

xiang90 added the area/performance label Dec 22, 2017

xiang90 reviewed Dec 23, 2017

View reviewed changes

hexfusion mentioned this pull request Jan 10, 2018

RFC : TXT DNS discovery #9128

Closed

hexfusion reviewed Jan 20, 2018

View reviewed changes

tavish-stripe force-pushed the tavish/svc-flag branch 4 times, most recently from d388145 to f44123b Compare January 24, 2018 19:49

pkg/srv, embed, etcdmain: Support multiple clusters in the same DNS d…

b664b91

…iscovery region.

tavish-stripe force-pushed the tavish/svc-flag branch from f44123b to b664b91 Compare January 24, 2018 20:15

gyuho added the kind/need-release-note-v3.4 label Jan 24, 2018

hexfusion reviewed Jan 24, 2018

View reviewed changes

Documentation: document the -discovery-srv-name flag for configuring …

81c9f78

…service name in DNS discovery.

tavish-stripe force-pushed the tavish/svc-flag branch from ca99cb2 to 81c9f78 Compare January 25, 2018 01:56

hexfusion approved these changes Jan 25, 2018

View reviewed changes

gyuho approved these changes Jan 25, 2018

View reviewed changes

hexfusion merged commit 7dd9c30 into etcd-io:master Jan 25, 2018

hexfusion mentioned this pull request Jan 30, 2018

etcdmain: add discovery-srv-name #9252

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pkg/srv, embed, etcdmain: Support multiple clusters in the same DNS domain #8690

pkg/srv, embed, etcdmain: Support multiple clusters in the same DNS domain #8690

tavish-stripe commented Oct 12, 2017 •

edited

Loading

codecov-io commented Oct 12, 2017 •

edited

Loading

xiang90 commented Dec 22, 2017

hexfusion commented Dec 22, 2017

tavish-stripe commented Dec 22, 2017

xiang90 Dec 23, 2017

hexfusion Jan 20, 2018

xiang90 commented Dec 23, 2017

hexfusion commented Dec 23, 2017 •

edited

Loading

hexfusion commented Jan 2, 2018 •

edited

Loading

hexfusion commented Jan 8, 2018

tavish-stripe commented Jan 9, 2018

hexfusion commented Jan 10, 2018

tavish-stripe commented Jan 18, 2018

hexfusion commented Jan 18, 2018

hexfusion commented Jan 20, 2018

hexfusion Jan 20, 2018

hexfusion Jan 20, 2018

hexfusion commented Jan 20, 2018

tavish-stripe commented Jan 22, 2018

hexfusion commented Jan 22, 2018

hexfusion commented Jan 22, 2018

tavish-stripe commented Jan 22, 2018

hexfusion commented Jan 24, 2018

xiang90 commented Jan 24, 2018

tavish-stripe commented Jan 24, 2018

hexfusion commented Jan 24, 2018

xiang90 commented Jan 24, 2018

tavish-stripe commented Jan 24, 2018

xiang90 commented Jan 24, 2018

hexfusion Jan 24, 2018

hexfusion commented Jan 24, 2018

hexfusion left a comment

gyuho left a comment

hexfusion commented Jan 25, 2018

pkg/srv, embed, etcdmain: Support multiple clusters in the same DNS domain #8690

pkg/srv, embed, etcdmain: Support multiple clusters in the same DNS domain #8690

Conversation

tavish-stripe commented Oct 12, 2017 • edited Loading

codecov-io commented Oct 12, 2017 • edited Loading

Codecov Report

xiang90 commented Dec 22, 2017

hexfusion commented Dec 22, 2017

tavish-stripe commented Dec 22, 2017

xiang90 Dec 23, 2017

Choose a reason for hiding this comment

hexfusion Jan 20, 2018

Choose a reason for hiding this comment

xiang90 commented Dec 23, 2017

hexfusion commented Dec 23, 2017 • edited Loading

hexfusion commented Jan 2, 2018 • edited Loading

cluster 1

cluster 2

hexfusion commented Jan 8, 2018

tavish-stripe commented Jan 9, 2018

hexfusion commented Jan 10, 2018

tavish-stripe commented Jan 18, 2018

hexfusion commented Jan 18, 2018

hexfusion commented Jan 20, 2018

hexfusion Jan 20, 2018

Choose a reason for hiding this comment

hexfusion Jan 20, 2018

Choose a reason for hiding this comment

hexfusion commented Jan 20, 2018

tavish-stripe commented Jan 22, 2018

hexfusion commented Jan 22, 2018

hexfusion commented Jan 22, 2018

tavish-stripe commented Jan 22, 2018

hexfusion commented Jan 24, 2018

xiang90 commented Jan 24, 2018

tavish-stripe commented Jan 24, 2018

hexfusion commented Jan 24, 2018

xiang90 commented Jan 24, 2018

tavish-stripe commented Jan 24, 2018

xiang90 commented Jan 24, 2018

hexfusion Jan 24, 2018

Choose a reason for hiding this comment

hexfusion commented Jan 24, 2018

hexfusion left a comment

Choose a reason for hiding this comment

gyuho left a comment

Choose a reason for hiding this comment

hexfusion commented Jan 25, 2018

tavish-stripe commented Oct 12, 2017 •

edited

Loading

codecov-io commented Oct 12, 2017 •

edited

Loading

hexfusion commented Dec 23, 2017 •

edited

Loading

hexfusion commented Jan 2, 2018 •

edited

Loading