Allow to use own certificate for metrics endpoint #10504

legionus · 2019-02-27T10:33:00Z

As a admin I want to give the monitoring system access only to metrics without access to the rest of the API.

codecov-io · 2019-02-27T12:38:28Z

Codecov Report

Merging #10504 into master will decrease coverage by 0.04%.
The diff coverage is 65.21%.

@@            Coverage Diff             @@
##           master   #10504      +/-   ##
==========================================
- Coverage   71.56%   71.52%   -0.05%     
==========================================
  Files         393      393              
  Lines       36503    36525      +22     
==========================================
  Hits        26123    26123              
- Misses       8546     8567      +21     
- Partials     1834     1835       +1

Impacted Files	Coverage Δ
embed/config.go	`56.37% <ø> (ø)`	⬆️
etcdmain/config.go	`83.03% <100%> (+0.54%)`	⬆️
etcdmain/grpc_proxy.go	`62.18% <44.44%> (-0.43%)`	⬇️
embed/etcd.go	`72.76% <57.14%> (+1.75%)`	⬆️
proxy/grpcproxy/register.go	`69.44% <0%> (-13.89%)`	⬇️
pkg/adt/interval_tree.go	`83.78% <0%> (-7.51%)`	⬇️
pkg/netutil/netutil.go	`63.11% <0%> (-6.56%)`	⬇️
etcdserver/v2_server.go	`80.76% <0%> (-3.85%)`	⬇️
pkg/testutil/recorder.go	`77.77% <0%> (-3.71%)`	⬇️
clientv3/leasing/kv.go	`89.03% <0%> (-1.33%)`	⬇️
... and 16 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 17de9bd...1da4d75. Read the comment docs.

pweil- · 2019-02-27T12:41:51Z

/cc @brancz @jim-minter

brancz · 2019-02-27T12:55:15Z

I lack expertise in the etcd code base to review this, but 👍 on the feature in general!

jingyih

Thanks for your PR!

jingyih · 2019-02-28T06:51:42Z

tests/e2e/metrics_test.go

@@ -57,3 +61,44 @@ func metricsTest(cx ctlCtx) {
 		cx.t.Fatalf("failed get with curl (%v)", err)
 	}
 }
+
+func metricsTestCertAuth(cx ctlCtx) {


Is it possible to unify this function with metricsTest?

@jingyih Everything is possible. Now the test looks better?

jingyih · 2019-02-28T07:10:51Z

embed/etcd.go

@@ -763,6 +763,9 @@ func (e *Etcd) serveMetrics() (err error) {

 		for _, murl := range e.cfg.ListenMetricsUrls {
 			tlsInfo := &e.cfg.ClientTLSInfo
+			if !e.cfg.MetricsTLSInfo.Empty() {
+				tlsInfo = &e.cfg.MetricsTLSInfo


If we override tlsInfo, should we add a warning to log?

When MetricsTLSInfo is provided, client who has access to other APIs will not be able to access '/metrics' and '/health'?

When MetricsTLSInfo is provided, client who has access to other APIs will not be able to access '/metrics' and '/health'?

Yes. Theoretically, we can make both work (add them to tls.Config.RootCAs and to tls.Config.ClientCAs), but it will be much more difficult solution because it requires to extend transport.TLSInfo.

If we override tlsInfo, should we add a warning to log?

This is not a hidden behavior (the admin must pass the options explicitly). I am not sure what message do I need to write to log ?
You specified other certificates for metrics. Access by client certificates will not be possible ?

This is not a hidden behavior (the admin must pass the options explicitly). I am not sure what message do I need to write to log ?

I was thinking about the scenario where etcd clients with clientTLS (not metricsTLS) might try to access the metrics and health endpoint. Adding something like 'ignoring client certificates since metrics certificates given' would help them understand why the connection is rejected. I am not an expert in this area, please feel free to let me know if you think this is reasonable.

@jingyih That's reasonable. I added the message.

For better or worse, warning messages are often noted by administrators and automated systems which analyze log output for anomalies. The proposed metrics TLS configuration when used in conjunction with client TLS configuration is a valid state. So in that regard it's not clear to me a preemptive warning makes sense. It seems possible that when using both TLS configurations (again, a valid state), this warning might be interpreted as a false negative alert of some kind.

So, I would suggest we consider changing this to INFO level or omitting it entirely.

jingyih · 2019-02-28T18:59:43Z

/cc @wenjiaswe

legionus · 2019-03-04T16:52:39Z

@jingyih looks good now ?

jingyih · 2019-03-06T03:54:03Z

@legionus Sorry for the delayed response. I will take a look later this week (hopefully on Wednesday).

jingyih · 2019-03-13T01:50:40Z

tests/e2e/v2_curl_test.go

@@ -155,6 +156,8 @@ func cURLPrefixArgs(clus *etcdProcessCluster, method string, req cURLReq) []stri
 				cmdArgs = append(cmdArgs, "--cacert", caPath, "--cert", certPath3, "--key", privateKeyPath3)
 			}
 		}
+	} else if req.useCertAuth {


I am having a hard time following the logic on how the command is generated. Does the if (line 145) make sense to you? For example, I tried to test TestV3MetricsSecure, but the actual curl command in the test was curl -L http://localhost:20000/metrics

This particular test I mentioned above was not affected by the changes in your PR. But I want to make sure the tests are correct and reader-friendly.

I am having a hard time following the logic on how the command is generated.

@jingyih Me too. I had to add one more block to be able to use another certificate. It seems to me wrong that they are hardcoded in this function. On the other hand, they are pregenerated and their purpose is predefined. So maybe this makes sense.

Does the if (line 145) make sense to you?

No, it does not make sense to me. This code seems broken. It looks like someone tried to add the logic for secure connections, but failed.

For example, I tried to test TestV3MetricsSecure, but the actual curl command in the test was curl -L http://localhost:20000/metrics.

This is because this test does not test a secure connection :) The TestV3MetricsSecure and TestV3MetricsInsecure create custom cfg, but don't use it:

etcd/tests/e2e/metrics_test.go

Lines 25 to 37 in 024f3df

func TestV3MetricsSecure(t *testing.T) {

cfg := configTLS

cfg.clusterSize = 1

cfg.metricsURLScheme = "https"

testCtl(t, metricsTest)

}

func TestV3MetricsInsecure(t *testing.T) {

cfg := configTLS

cfg.clusterSize = 1

cfg.metricsURLScheme = "http"

testCtl(t, metricsTest)

}

To use custom config they must use withCfg(). It means that these tests use the default config:

etcd/tests/e2e/ctl_v3_test.go

Lines 133 to 141 in 024f3df

func testCtl(t *testing.T, testFunc func(ctlCtx), opts ...ctlOption) {

defer testutil.AfterTest(t)

ret := ctlCtx{

t: t,

cfg: configAutoTLS,

dialTimeout: 7 * time.Second,

}

ret.applyOpts(opts)

As you can see there is no metricsURLScheme definition here:

etcd/tests/e2e/cluster_test.go

Lines 42 to 47 in 024f3df

configAutoTLS = etcdProcessClusterConfig{

clusterSize: 3,

isPeerTLS: true,

isPeerAutoTLS: true,

initialToken: "new",

}

It means the cx.cfg.metricsURLScheme will be empty as well:

etcd/tests/e2e/metrics_test.go

Line 43 in 024f3df

if err := cURLGet(cx.epc, cURLReq{endpoint: "/metrics", expected: `etcd_debugging_mvcc_keys_total 1`, metricsURLScheme: cx.cfg.metricsURLScheme}); err != nil {

jingyih · 2019-03-15T07:58:19Z

Your PR looks good to me. One option is that we get it merged first, and fix the tests in a separate PR.

@hexfusion Any comment?

legionus · 2019-04-08T06:52:05Z

@jingyih @hexfusion ping

hexfusion · 2019-04-09T13:46:31Z

@legionus thanks for your patience. I will try to dig into this soon.

xiang90 · 2019-05-01T23:35:56Z

/cc @hexfusion can you please take a look?

stale · 2020-04-06T19:55:45Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

brancz · 2020-04-07T05:59:30Z

I think this is still something that’s very desirable to have. Anyone continuing the work? :)

stale · 2020-07-06T06:19:13Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

brancz · 2020-07-08T07:26:49Z

This would continue to be a valuable addition I think.

ironcladlou · 2020-07-13T13:48:43Z

@legionus do you intend to complete this?

legionus · 2020-07-13T13:57:38Z

@legionus do you intend to complete this?

@ironcladlou I have already changed the scope of work, but I can complete this PR if someone finds this PR useful.
What do I need to do ? rebase & resolve conflicts ?

ironcladlou · 2020-07-13T14:04:27Z

@legionus

@ironcladlou I have already changed the scope of work, but I can complete this PR if someone finds this PR useful.
What do I need to do ? rebase & resolve conflicts ?

Can you explain what you mean re: the scope change?

I'm trying to decide how to proceed with #11993 and want to make sure my solution integrates with this work if the intent is to proceed as-is.

I can't say whether there's demand for this one, as I haven't been involved.

legionus · 2020-07-13T14:14:11Z

Can you explain what you mean re: the scope change?

@ironcladlou I no longer develop openshift and etcd. Now I am involved in the development of completely different projects.
But I still remember golang :)

ironcladlou · 2020-07-13T14:31:23Z

@legionus

@ironcladlou I no longer develop openshift and etcd. Now I am involved in the development of completely different projects.
But I still remember golang :)

Ah ha! Thanks!

Okay, so as far as I can tell from the history of this PR all that's left is rebasing, but I can't speak to whether the demand for the change still exists. @brancz seems to say yes, but I'm not personally aware of anyone with use cases for it, so I'm neutral.

Who can make the decision about whether this should proceed or not?

tangcong · 2020-07-13T15:16:27Z

i think this pr is useful, @jingyih also agrees with it. can you rebase & resolve conflicts ? thanks. @legionus

Signed-off-by: Gladkov Alexey <agladkov@redhat.com>

…ndpoints Signed-off-by: Gladkov Alexey <agladkov@redhat.com>

legionus · 2020-07-13T16:45:19Z

i think this pr is useful, @jingyih also agrees with it. can you rebase & resolve conflicts ? thanks.

@tangcong, @ironcladlou Yes, I can. Done.

tangcong · 2020-07-14T00:03:23Z

etcdmain/grpc_proxy.go

@@ -211,6 +223,12 @@ func startGRPCProxy(cmd *cobra.Command, args []string) {
 	go func() { errc <- srvhttp.Serve(httpl) }()
 	go func() { errc <- m.Serve() }()
 	if len(grpcProxyMetricsListenAddr) > 0 {
+		if grpcProxyMetricsListenCert != "" && grpcProxyMetricsListenKey != "" {
+			tlsinfo = newTLS(grpcProxyMetricsListenCA, grpcProxyMetricsListenCert, grpcProxyMetricsListenKey)


pr #12114 adds health handler for grpcproxy self, if grpcProxyMetricsListenAddr uses own certificate for metrics endpoint, we should also update healthcheck client certs, otherwise, it is failed to access '/proxy/health' handle.

@ironcladlou said For the health listener, honor global the global TLS configuration by default (for compatibility with today's behavior), but provide a new flag to allow control over health endpoint client certificate auth (e.g. --health-client-cert-auth=false). in issue #11993.
i agrees with it. can we add a flag to access /health,/proxy/health without cert auth?

pr #12114 adds health handler for grpcproxy self, if grpcProxyMetricsListenAddr uses own certificate for metrics endpoint, we should also update healthcheck client certs, otherwise, it is failed to access '/proxy/health' handle.

I already mentioned this before. Therefore, when using separate certificates for metrics, a warning is displayed.

i agrees with it. can we add a flag to access /health,/proxy/health without cert auth?

Do you want the server to serve some endpoints without certificates? Theoretically, this is possible, but will require rewriting the server. To do this, you will need to make authorization optional and do it not at the TLS level. This is out of scope of this proposal.

ok. thanks. I will try to add --health-client-cert-auth flag in another PR. thanks.

@tangcong Can this PR be merged?

lgtm. can you add changelog and doc in another pr? thanks. @hexfusion can you take a final look? thanks.

@tangcong sorry for delay yes I will take a look.

hexfusion · 2020-08-11T13:42:53Z

/assign

legionus · 2020-09-08T11:40:50Z

@hexfusion @tangcong, @ironcladlou Hi! Is there any news?

stale · 2021-01-03T16:22:21Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

legionus force-pushed the metrics-certs branch from e3c0bb2 to 112b962 Compare February 27, 2019 11:21

jingyih reviewed Feb 28, 2019

View reviewed changes

legionus force-pushed the metrics-certs branch from 112b962 to fa73854 Compare February 28, 2019 09:10

jingyih self-assigned this Mar 6, 2019

jingyih reviewed Mar 13, 2019

View reviewed changes

jingyih mentioned this pull request Mar 16, 2019

Explore possibility to enable serving etcd health check on a different port so we can have different tls requirement for etcd server and etcd health #10545

Closed

stale bot added the stale label Apr 6, 2020

stale bot removed the stale label Apr 7, 2020

stale bot added the stale label Jul 6, 2020

stale bot removed the stale label Jul 8, 2020

ironcladlou mentioned this pull request Jul 13, 2020

The health endpoint should have its own listener TLS configuration #11993

Closed

legionus added 5 commits July 13, 2020 17:21

embed: configure TLSInfo

093db65

Signed-off-by: Gladkov Alexey <agladkov@redhat.com>

etcdmain: add options to configure metrics cert

575a000

Signed-off-by: Gladkov Alexey <agladkov@redhat.com>

integration/fixtures: add metrics cert

f27a3f5

Signed-off-by: Gladkov Alexey <agladkov@redhat.com>

tests/e2e: add test for certificate protected metrics

670a78a

Signed-off-by: Gladkov Alexey <agladkov@redhat.com>

embed: add a warning to log that we changed certificate for metrics e…

d76775e

…ndpoints Signed-off-by: Gladkov Alexey <agladkov@redhat.com>

legionus force-pushed the metrics-certs branch from 1da4d75 to d76775e Compare July 13, 2020 16:24

tangcong reviewed Jul 14, 2020

View reviewed changes

hexfusion self-assigned this Aug 11, 2020

stale bot added the stale label Jan 3, 2021

stale bot closed this Jan 24, 2021

ironcladlou mentioned this pull request Oct 21, 2021

Minimum viable HA etcd API and implementation openshift/hypershift#580

Merged

	func TestV3MetricsSecure(t *testing.T) {
	cfg := configTLS
	cfg.clusterSize = 1
	cfg.metricsURLScheme = "https"
	testCtl(t, metricsTest)
	}

	func TestV3MetricsInsecure(t *testing.T) {
	cfg := configTLS
	cfg.clusterSize = 1
	cfg.metricsURLScheme = "http"
	testCtl(t, metricsTest)
	}

	func testCtl(t *testing.T, testFunc func(ctlCtx), opts ...ctlOption) {
	defer testutil.AfterTest(t)

	ret := ctlCtx{
	t: t,
	cfg: configAutoTLS,
	dialTimeout: 7 * time.Second,
	}
	ret.applyOpts(opts)

	configAutoTLS = etcdProcessClusterConfig{
	clusterSize: 3,
	isPeerTLS: true,
	isPeerAutoTLS: true,
	initialToken: "new",
	}

Allow to use own certificate for metrics endpoint #10504

Allow to use own certificate for metrics endpoint #10504

Conversation

legionus commented Feb 27, 2019

codecov-io commented Feb 27, 2019 • edited

Codecov Report

pweil- commented Feb 27, 2019

brancz commented Feb 27, 2019

jingyih left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jingyih commented Feb 28, 2019

legionus commented Mar 4, 2019

jingyih commented Mar 6, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jingyih commented Mar 15, 2019 • edited

legionus commented Apr 8, 2019

hexfusion commented Apr 9, 2019

xiang90 commented May 1, 2019

stale bot commented Apr 6, 2020

brancz commented Apr 7, 2020

stale bot commented Jul 6, 2020

brancz commented Jul 8, 2020

ironcladlou commented Jul 13, 2020

legionus commented Jul 13, 2020

ironcladlou commented Jul 13, 2020

legionus commented Jul 13, 2020

ironcladlou commented Jul 13, 2020

tangcong commented Jul 13, 2020

legionus commented Jul 13, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hexfusion commented Aug 11, 2020

legionus commented Sep 8, 2020

stale bot commented Jan 3, 2021

codecov-io commented Feb 27, 2019 •

edited

jingyih commented Mar 15, 2019 •

edited