feat sicedar: added readiness prober #1395

FUSAKLA · 2019-08-10T15:35:43Z

Signed-off-by: Martin Chodur m.chodur@seznam.cz

CHANGELOG entry if change is relevant to the end user.

Changes

Next PR in the from the series of adding readiness and liveness probes.

Added readiness and liveness prober to the thanos sidecar.
It follows the logic of setting the promUp metric so when it successfully fetches config of the Prometheus it becomes ready.

The question is should it take into account status of the GRPC interface also?

Verification

started sidecar without running the Prometheus instance results in

$ curl http://0.0.0.0:10902/-/ready
thanos sidecar is not ready. Reason: request flags against http://localhost:9090/api/v1/status/config: Get http://localhost:9090/api/v1/status/config: dial tcp 127.0.0.1:9090: connect: connection refused

started the prometheus sidecar becomes ready

$ curl http://0.0.0.0:10902/-/ready
thanos sidecar is ready

stopped the prometheus instance again sidecar goes again to not ready state after max 30s which is the sidecar check interval

$ curl http://0.0.0.0:10902/-/ready
thanos sidecar is not ready. Reason: request flags against http://localhost:9090/api/v1/status/config: Get http://localhost:9090/api/v1/status/config: dial tcp 127.0.0.1:9090: connect: connection refused

Signed-off-by: Martin Chodur <m.chodur@seznam.cz>

povilasv

cmd/thanos/sidecar.go

bwplotka

Some suggestions (: Thanks overall!

bwplotka · 2019-08-13T14:14:25Z

cmd/thanos/sidecar.go

-func registerSidecar(m map[string]setupFunc, app *kingpin.Application, name string) {
-	cmd := app.Command(name, "sidecar for Prometheus server")
+func registerSidecar(m map[string]setupFunc, app *kingpin.Application) {
+	comp := component.Sidecar


I would maybe inline those, would be more verbose.

I inlined it in the registerSidecar function, not sure if you want to inline it everywhere even in the runSidecar ?

cmd/thanos/sidecar.go

Signed-off-by: Martin Chodur <m.chodur@seznam.cz>

FUSAKLA · 2019-08-14T04:53:29Z

@povilasv thanks! there are some new changes just so you don't approve something you didn't see so maybe check that out :)

@bwplotka Thanks for the suggestions! I tried some refactor of the sidecar liveness so PTAL.
Also I applied your other suggestions on the compactor also since the changes are the same as in the previous PR.

FUSAKLA · 2019-08-14T05:38:45Z

those failing test are weird, not sure if related to my changes?

cmd/thanos/sidecar.go

bwplotka

I think it generally make sense.. I am bit worried that those values for sinceLastSuccessfulHeartbeatLimit might be generally nice.. but might not match some unique use cases user may have. Also this means any flake on connection to Prometheus (single heartbeat failed) might mean unavailability. I am a bit skeptical, I would mark it as non ready after 3 heartbeats or something (?)

cmd/thanos/sidecar.go

pkg/prober/prober.go

bwplotka · 2019-08-16T08:56:32Z

Let's chat offline @FUSAKLA but overall problem with those probes is that we need to be careful. @povilasv mentioned that all of this can be configured anyone on Kubernetes side, but that's not that simple. Problems:

You can control those things.. timeouts, thresholds etc in Kubernetes, but maybe not in thousands of other solutions/load balancers
You can control those things... But only per container. This means that if you have 4 different cases in the application (like sidecar) when you make something not ready or not healthy.... You cannot do it yolo and ask someone to configure something on top. As they can only adjust all cases in same time there is No granularity.

bwplotka · 2019-08-16T08:57:07Z

Regarding healthyness: https://twitter.com/miekg/status/1152911058627153920

We are I think on the same page that we should do it ONLY set healthyness to NOT healthy if we are sure that restart will help.

FUSAKLA · 2019-08-16T17:29:19Z

Yeah, so should we get back to the original state of just modifying the readiness based on the halthchecks?

I wanted originally to avoid touching the liveness probe because it felt bit harsh but I understood that this was what you suggested. But I probably did not clearly understood what you meant, sorry for the confusement 😕

Signed-off-by: Martin Chodur <m.chodur@seznam.cz>

FUSAKLA · 2019-08-20T13:24:43Z

Along with the offline agreement, the readiness probe is only set after the initial Prometheus external labels fetch from the Prometheus. Once they are fetched it stays ready all the time.

If we hit any case in the future when more complex readiness handling will be needed it can be added then.

bwplotka

This looks good 👍

One important thing before merge. We never set prober to be healthy. And in prober constructor we set healthyness to be initialError. Maybe we should be healthy by default in NewProber?

FUSAKLA · 2019-08-27T17:14:35Z

@bwplotka this is done in the defaultHTTPListener constructor where the mux is started actually if that is ok?

thanos/cmd/thanos/main.go

Lines 362 to 369 in 0a30a2e

    
           g.Add(func() error { 
        
           	level.Info(logger).Log("msg", "listening for metrics", "address", httpBindAddr) 
        
           	readinessProber.SetHealthy() 
        
           	return errors.Wrap(http.Serve(l, mux), "serve metrics") 
        
           }, func(err error) { 
        
           	readinessProber.SetNotHealthy(err) 
        
           	runutil.CloseWithLogOnErr(logger, l, "metric listener") 
        
           })

Signed-off-by: Martin Chodur <m.chodur@seznam.cz>

FUSAKLA · 2019-08-27T17:17:49Z

But I found one bug with opposite condition on error logging in the Prober, fixed in 60897a8 🙄

bwplotka

LGTM

bwplotka · 2019-08-27T20:18:49Z

pkg/prober/prober.go

@@ -37,6 +37,19 @@ type Prober struct {
 func NewProber(component component.Component, logger log.Logger, reg prometheus.Registerer) *Prober {
 	initialErr := fmt.Errorf(initialErrorFmt, component)

+	// From Kubernetes documentation https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/ :


Minor nit but this should part of struct TBH (for GoDocs)

good point, is it ok this way?

Signed-off-by: Martin Chodur <m.chodur@seznam.cz>

bwplotka

LGTM!

bwplotka · 2019-08-28T08:57:24Z

Thanks @FUSAKLA !

FUSAKLA · 2019-08-28T09:46:15Z

Great, thank you all for the comments and going through the discussions around the readiness!

There is only few components left 😄
If you have the time and will discuss another one, then I made next PR for the store #1460

* feat sicedar: added readiness prober Signed-off-by: Martin Chodur <m.chodur@seznam.cz>

feat sicedar: added readiness prober

3b2d411

Signed-off-by: Martin Chodur <m.chodur@seznam.cz>

FUSAKLA force-pushed the fus-sidecar-prober branch from e460685 to 3b2d411 Compare August 10, 2019 15:37

povilasv approved these changes Aug 13, 2019

View reviewed changes

cmd/thanos/sidecar.go Outdated Show resolved Hide resolved

bwplotka reviewed Aug 13, 2019

View reviewed changes

FUSAKLA added 2 commits August 14, 2019 06:40

cr: inline the component usage

9193da3

Signed-off-by: Martin Chodur <m.chodur@seznam.cz>

cr: refactor sidecar prober logic

cdc30f4

Signed-off-by: Martin Chodur <m.chodur@seznam.cz>

FUSAKLA force-pushed the fus-sidecar-prober branch from 3eaf82a to cdc30f4 Compare August 14, 2019 04:41

bwplotka reviewed Aug 15, 2019

View reviewed changes

cmd/thanos/sidecar.go Outdated Show resolved Hide resolved

bwplotka reviewed Aug 15, 2019

View reviewed changes

cmd/thanos/sidecar.go Outdated Show resolved Hide resolved

pkg/prober/prober.go Outdated Show resolved Hide resolved

CR: revert to original PR logic wothout touching sidecar liveness

153cd3f

Signed-off-by: Martin Chodur <m.chodur@seznam.cz>

FUSAKLA force-pushed the fus-sidecar-prober branch from 326450a to 153cd3f Compare August 19, 2019 07:42

CR: removed readiness handling from periodical prometheus health checks

d351d03

Signed-off-by: Martin Chodur <m.chodur@seznam.cz>

FUSAKLA force-pushed the fus-sidecar-prober branch from 35633e7 to d351d03 Compare August 20, 2019 13:21

povilasv approved these changes Aug 21, 2019

View reviewed changes

bwplotka approved these changes Aug 27, 2019

View reviewed changes

fix prober: fixed false error logging

60897a8

Signed-off-by: Martin Chodur <m.chodur@seznam.cz>

bwplotka approved these changes Aug 27, 2019

View reviewed changes

bwplotka reviewed Aug 27, 2019

View reviewed changes

FUSAKLA force-pushed the fus-sidecar-prober branch from 79d3507 to 480fff5 Compare August 28, 2019 04:08

CR: fix prober comment to work with godoc

210075a

Signed-off-by: Martin Chodur <m.chodur@seznam.cz>

FUSAKLA force-pushed the fus-sidecar-prober branch from 480fff5 to 210075a Compare August 28, 2019 04:28

bwplotka approved these changes Aug 28, 2019

View reviewed changes

bwplotka merged commit c83332f into thanos-io:master Aug 28, 2019

kakkoyun mentioned this pull request Sep 17, 2019

feat store: added readiness and livenes prober #1460

Merged

1 task

wbh1 pushed a commit to wbh1/thanos that referenced this pull request Sep 17, 2019

feat sicedar: added readiness prober (thanos-io#1395)

d52d15a

* feat sicedar: added readiness prober Signed-off-by: Martin Chodur <m.chodur@seznam.cz>

FUSAKLA mentioned this pull request Sep 23, 2019

Add missing /-/healthy endpoint #644

Closed

povilasv mentioned this pull request Oct 24, 2019

Ready state handling of sidecar #1677

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat sicedar: added readiness prober #1395

feat sicedar: added readiness prober #1395

FUSAKLA commented Aug 10, 2019 •

edited

Loading

povilasv left a comment

bwplotka left a comment

bwplotka Aug 13, 2019

FUSAKLA Aug 14, 2019

FUSAKLA commented Aug 14, 2019

FUSAKLA commented Aug 14, 2019

bwplotka left a comment

bwplotka commented Aug 16, 2019

bwplotka commented Aug 16, 2019 •

edited

Loading

FUSAKLA commented Aug 16, 2019

FUSAKLA commented Aug 20, 2019

bwplotka left a comment

FUSAKLA commented Aug 27, 2019

FUSAKLA commented Aug 27, 2019

bwplotka left a comment

bwplotka Aug 27, 2019

FUSAKLA Aug 28, 2019

bwplotka left a comment

bwplotka commented Aug 28, 2019

FUSAKLA commented Aug 28, 2019

feat sicedar: added readiness prober #1395

feat sicedar: added readiness prober #1395

Conversation

FUSAKLA commented Aug 10, 2019 • edited Loading

Changes

Verification

povilasv left a comment

Choose a reason for hiding this comment

bwplotka left a comment

Choose a reason for hiding this comment

bwplotka Aug 13, 2019

Choose a reason for hiding this comment

FUSAKLA Aug 14, 2019

Choose a reason for hiding this comment

FUSAKLA commented Aug 14, 2019

FUSAKLA commented Aug 14, 2019

bwplotka left a comment

Choose a reason for hiding this comment

bwplotka commented Aug 16, 2019

bwplotka commented Aug 16, 2019 • edited Loading

FUSAKLA commented Aug 16, 2019

FUSAKLA commented Aug 20, 2019

bwplotka left a comment

Choose a reason for hiding this comment

FUSAKLA commented Aug 27, 2019

FUSAKLA commented Aug 27, 2019

bwplotka left a comment

Choose a reason for hiding this comment

bwplotka Aug 27, 2019

Choose a reason for hiding this comment

FUSAKLA Aug 28, 2019

Choose a reason for hiding this comment

bwplotka left a comment

Choose a reason for hiding this comment

bwplotka commented Aug 28, 2019

FUSAKLA commented Aug 28, 2019

FUSAKLA commented Aug 10, 2019 •

edited

Loading

bwplotka commented Aug 16, 2019 •

edited

Loading