Add TLS and basic auth #4211

saswatamcode · 2021-05-10T02:54:33Z

I added CHANGELOG entry for this change.
Change is not relevant to the end user.

Changes

This PR adds in TLS and basic auth to Thanos HTTP APIs using exporter-toolkit. Configuration follows a similar pattern to Prometheus, where a file containing TLS configuration is provided using http-tls-config flag.
Addresses issue #4168

Verification

Tested locally.

bwplotka

Thanks! Some thoughts. Also we might want some docs 🙈

bwplotka · 2021-05-10T11:36:32Z

cmd/thanos/config.go

@@ -54,6 +55,10 @@ func (hc *httpConfig) registerFlag(cmd extkingpin.FlagClause) *httpConfig {
 	cmd.Flag("http-grace-period",
 		"Time to wait after an interrupt received for HTTP Server.").
 		Default("2m").SetValue(&hc.gracePeriod)
+	cmd.Flag(
+		"http-tls-config",
+		"[EXPERIMENTAL] Path to configuration file that can enable TLS or authentication.",


Suggested change

"[EXPERIMENTAL] Path to configuration file that can enable TLS or authentication.",

"[EXPERIMENTAL] Path to the configuration file that can enable TLS or authentication for all HTTP endpoints.",

bwplotka · 2021-05-10T11:37:13Z

cmd/thanos/config.go

@@ -54,6 +55,10 @@ func (hc *httpConfig) registerFlag(cmd extkingpin.FlagClause) *httpConfig {
 	cmd.Flag("http-grace-period",
 		"Time to wait after an interrupt received for HTTP Server.").
 		Default("2m").SetValue(&hc.gracePeriod)
+	cmd.Flag(


What about doing pathorcontent flag? 🤔

Sure. Will add that in!

Currently, exporter-toolkit doesn't take in content(byte[]) of the TLS config file as an argument for its server, it only takes in the path(string) as highlighted here. So using pathorcontent isn't currently possible I think, since this flag fetches bytes. @bwplotka

bwplotka · 2021-05-10T11:37:33Z

pkg/extkingpin/flags.go

 	httpBindAddr = cmd.Flag("http-address", "Listen host:port for HTTP endpoints.").Default("0.0.0.0:10902").String()
 	httpGracePeriod = ModelDuration(cmd.Flag("http-grace-period", "Time to wait after an interrupt received for HTTP Server.").Default("2m")) // by default it's the same as query.timeout.
-
-	return httpBindAddr, httpGracePeriod
+	httpTLSConfig = cmd.Flag(


bwplotka · 2021-05-10T11:39:18Z

pkg/server/http/http.go

+	err := toolkit_web.Validate(s.opts.tlsConfigPath)
+	if err != nil {
+		level.Error(s.logger).Log("msg", "server could not be started", "err", err)
+		return errors.Wrap(err, "server could not be started")
+	}


Suggested change

err := toolkit_web.Validate(s.opts.tlsConfigPath)

if err != nil {

level.Error(s.logger).Log("msg", "server could not be started", "err", err)

return errors.Wrap(err, "server could not be started")

}

if err := toolkit_web.Validate(s.opts.tlsConfigPath); err != nil {

return errors.Wrap(err, "server could not be started")

}

Let's make sure to avoid handling the same error twice.

bwplotka

Amazing! LGTM, just one nit.

bwplotka · 2021-05-11T12:22:40Z

pkg/extkingpin/flags.go

-
-	return httpBindAddr, httpGracePeriod
+	httpTLSConfig = cmd.Flag(
+		"http-tls-config",


One nit, I heard from @roidelapluie that Prometheus flag is called web.config.file and not tls-config.file for a good reason: It's not only TLS. I think it's good idea to change it to http.config maybe? Also we can expand (contribute) to exporter toolking to support passing parsed configuration too if we want pathorcontent flag (up to us)

http.config sounds nice, will change it to that. I'll try to add the passing configuration functionality to exporter-toolkit, but once merged they would probably need to release that. Once it's released with configuration, we can switch over to pathorcontent flag(will make it a separate PR). WDYT? @bwplotka

From what I understand you do not need changes in the downstream exporter toolkit to achieve what you want.

You could probably do all of this in thanos. --http-config={file} or --http-config=/path:{file} could be supported without upstream changes.

You could probably do all of this in thanos. --http-config={file} or --http-config=/path:{file} could be supported without upstream changes.

Currently we are doing this, i.e, taking a file path string. But if we try to use the pathorcontent flag that would return the content of the file in byte[] which we can't pass to any of the toolkit methods. Wouldn't that require some changes? @roidelapluie

Maybe I do not understand the change. Are you trying to use basic auth on certain paths only? Or am I misunderstanding?

I'm trying to add TLS and basic auth in all HTTP endpoints. This PR currently adds a flag(http-tls-config which I'll change tohttp.config), which like Prometheus takes in the path of the file containing TLS/auth config. If we use the pathorcontent method for this flag, user can pass path of the config file or directly pass the content of the file. Either way pathorcontent will return the content of the config file. But then we can't pass it to exporter-toolkit since the methods only take in path. So, I think the change needs to be made in toolkit...

There is a reason why we take a path. The file is read on every request to make sure that the TLS certificate and the users are up to date.

Oh, okay. Then I guess using pathorcontent isn't suitable and the normal Flag method would be better. Also, no change is needed in exporter or Thanos then. Thanks a lot for explaining! 🙂

Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com>

bwplotka

Epic work! LGTM 💪🏽

cc @kakkoyun and anyone else

I will merge in few hours, if no objections

kakkoyun

👍

saswatamcode force-pushed the add-tls branch from 7b1f9b7 to 044170f Compare May 10, 2021 03:31

bwplotka reviewed May 10, 2021

View reviewed changes

saswatamcode requested a review from bwplotka May 10, 2021 14:47

bwplotka previously approved these changes May 11, 2021

View reviewed changes

saswatamcode dismissed bwplotka’s stale review via 83f3620 May 11, 2021 14:23

saswatamcode requested a review from bwplotka May 11, 2021 14:31

saswatamcode force-pushed the add-tls branch from 83f3620 to 5edfbe2 Compare May 14, 2021 15:03

saswatamcode added 4 commits May 17, 2021 21:15

Add TLS and basic auth

dd225e8

Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com>

Add changelog and fix docs

3d5bfcb

Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com>

Add docs

2c51600

Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com>

Rename to http.config

eb9c4b9

Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com>

saswatamcode force-pushed the add-tls branch from 5edfbe2 to eb9c4b9 Compare May 17, 2021 15:45

bwplotka approved these changes May 17, 2021

View reviewed changes

kakkoyun approved these changes May 17, 2021

View reviewed changes

bwplotka merged commit f2564a7 into thanos-io:main May 17, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add TLS and basic auth #4211

Add TLS and basic auth #4211

saswatamcode commented May 10, 2021 •

edited

bwplotka left a comment

bwplotka May 10, 2021

bwplotka May 10, 2021

saswatamcode May 10, 2021

saswatamcode May 10, 2021

bwplotka May 10, 2021

bwplotka May 10, 2021

bwplotka May 10, 2021

bwplotka left a comment

bwplotka May 11, 2021

saswatamcode May 11, 2021

roidelapluie May 11, 2021 •

edited

saswatamcode May 11, 2021 •

edited

roidelapluie May 11, 2021

saswatamcode May 11, 2021 •

edited

roidelapluie May 11, 2021

saswatamcode May 11, 2021

bwplotka left a comment

kakkoyun left a comment

	"[EXPERIMENTAL] Path to configuration file that can enable TLS or authentication.",
	"[EXPERIMENTAL] Path to the configuration file that can enable TLS or authentication for all HTTP endpoints.",

Add TLS and basic auth #4211

Add TLS and basic auth #4211

Conversation

saswatamcode commented May 10, 2021 • edited

Changes

Verification

bwplotka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bwplotka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

roidelapluie May 11, 2021 • edited

Choose a reason for hiding this comment

saswatamcode May 11, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

saswatamcode May 11, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bwplotka left a comment

Choose a reason for hiding this comment

kakkoyun left a comment

Choose a reason for hiding this comment

saswatamcode commented May 10, 2021 •

edited

roidelapluie May 11, 2021 •

edited

saswatamcode May 11, 2021 •

edited

saswatamcode May 11, 2021 •

edited