Added support for tail to query frontend #2032

tivvit · 2020-05-02T00:44:38Z

What this PR does / why we need it:

It is adding support for tailing to query frontend (basically WebSocket proxy)

Which issue(s) this PR fixes:
First I wanted to create issue for this. Then I checked in the code and saw that it is not implemented and decided to try to implement it.

Special notes for your reviewer:

I am not sure how to handle client certificates, any ideas?
Is WsTailProxy in the correct place?

Checklist

Documentation added
Tests updated

I think it is expected behavior - no need to add documentation
Not sure how to test this

codecov-io · 2020-05-02T00:57:01Z

Codecov Report

Merging #2032 into master will decrease coverage by 0.03%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master    #2032      +/-   ##
==========================================
- Coverage   64.46%   64.43%   -0.04%     
==========================================
  Files         131      131              
  Lines       10038    10046       +8     
==========================================
+ Hits         6471     6473       +2     
- Misses       3084     3091       +7     
+ Partials      483      482       -1

Impacted Files	Coverage Δ
pkg/loki/modules.go	`15.25% <0.00%> (-0.54%)`	⬇️
pkg/promtail/targets/tailer.go	`78.40% <0.00%> (+2.27%)`	⬆️

cyriltovena · 2020-05-05T20:22:35Z

We skip the frontend on our side for tail request, but I'm not against this, although this means support will be limited.

WDYT @slim-bean ? If we're good I'll review this !

tivvit · 2020-05-06T19:08:14Z

Hi @cyriltovena thanks for your reply. I think it is good to have direct support for this in the query frontend. It simplifies the setup when you are not using a proxy. I get that you can route the traffic based on path directly to the querier or to query frontend.

Yes the support is limited, but the features we are losing are related to auth. And when you are using auth you already have to use some auth proxy, therefore, it is not an issue anymore. Am I correct? But it definitely should be described in the docs somehow.

tivvit · 2020-05-26T11:14:05Z

@cyriltovena did you have time to look at it?

cyriltovena · 2020-05-26T13:47:13Z

I'm waiting for feedback from someone else, I'm not against it at all !

/cc @slim-bean @owen-d @sandeepsukhani

pkg/loki/modules.go

cyriltovena · 2020-06-01T15:37:00Z

I'll bring this up in the next meeting see if what the team thinks about it.

periklis · 2020-06-17T15:55:40Z

Good intent here! Thank you for contributing this. It goes a step closer to have simpler Loki deployments. E.g. on kubernetes, where you need to maintain three distinct services one for writes to the distributor, one for reads to the frontend and one for tails to the queriers. However, I ack to maintainers' comments, that a simple reverse proxy would fix the need to have the tail endpoint on the proxy. Thanks again for bringing this up and all the effort!

cyriltovena · 2020-06-19T12:01:02Z

We don’t see any issues with this if you replace your implementation with this https://golang.org/pkg/net/http/httputil/#ReverseProxy since go1.12 it supports websockets.

🙏

owen-d · 2020-06-19T12:27:57Z

Agree with @cyriltovena. I think we'll be able to get this merged.

stale · 2020-07-19T12:37:50Z

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

tivvit · 2020-07-19T22:18:14Z

Not stale. I will prepare it with the use of the Reverseproxy from httputil

…on, added tailproxy config

codecov-commenter · 2020-07-21T22:49:48Z

Codecov Report

Merging #2032 into master will decrease coverage by 0.07%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master    #2032      +/-   ##
==========================================
- Coverage   61.67%   61.60%   -0.08%     
==========================================
  Files         160      160              
  Lines       13569    13582      +13     
==========================================
- Hits         8369     8367       -2     
- Misses       4577     4592      +15     
  Partials      623      623

Impacted Files	Coverage Δ
pkg/loki/loki.go	`0.00% <ø> (ø)`
pkg/loki/modules.go	`8.59% <0.00%> (-0.54%)`	⬇️
pkg/querier/queryrange/downstreamer.go	`95.87% <0.00%> (-2.07%)`	⬇️
pkg/logql/evaluator.go	`92.47% <0.00%> (-0.41%)`	⬇️
pkg/promtail/targets/file/tailer.go	`78.40% <0.00%> (+2.27%)`	⬆️

tivvit · 2020-07-21T22:51:25Z

I am not sure if the tailproxy config is in correct location.

Can somebody please guide me how to add section to the flags? Because now it is located in the main section.

slim-bean · 2020-07-22T10:10:14Z

Thanks for making these changes, will take a look today or tomorrow!

pkg/loki/modules.go

cyriltovena

LGTM

sandeepsukhani · 2020-07-23T08:32:35Z

pkg/loki/modules.go

@@ -308,7 +320,7 @@ func (t *Loki) initQueryFrontend() (_ services.Service, err error) {
 	t.server.HTTP.Handle("/api/prom/label/{name}/values", frontendHandler)
 	t.server.HTTP.Handle("/api/prom/series", frontendHandler)
 	// fallback route
-	t.server.HTTP.PathPrefix("/").Handler(frontendHandler)
+	t.server.HTTP.PathPrefix("/").Handler(tailProxy)


The default value for tailproxy.downstream-url is empty string and not everyone would want to set it but here we use it anyways. I think we should use it only when a value is set for it.

oh great point I will fix it

great point thanks, I will fix it

what do you think about this solution ef61f93?

owen-d

This is looking good! A little nitpick here, then LGTM.

owen-d · 2020-07-23T13:16:49Z

pkg/loki/loki.go

@@ -55,6 +56,7 @@ type Config struct {
 	TableManager     chunk.TableManagerConfig    `yaml:"table_manager,omitempty"`
 	Worker           frontend.WorkerConfig       `yaml:"frontend_worker,omitempty"`
 	Frontend         frontend.Config             `yaml:"frontend,omitempty"`
+	TailProxy        tailproxy.Config            `yaml:"tail_proxy,omitempty"`


In addition to what @sandeepsukhani mentioned, I think this should be part of the frontend config rather than it's own top level value because it's only used in the frontend.

We may need to extend our frontend config definition to be something like

type Config struct { frontend.Config `yaml:",inline"` tail_proxy_url string `yaml:"tail_proxy_url"` }

That is a good idea. I did it like this because I knew it does not belong to Cortex.

This is better I will prepare it

edit: i love the podracer gif :)

@owen-d what do think about it now? I used embedded struct as you proposed and IMHO it is better.

cyriltovena

Lgtm

* don't exec create keyspace if it exists Signed-off-by: Wing924 <weihe924stephen@gmail.com> * fix Signed-off-by: Wing924 <weihe924stephen@gmail.com> * fix Signed-off-by: Wing924 <weihe924stephen@gmail.com>

tivvit added 2 commits May 2, 2020 02:33

Added support for tail to query frontend

d02f8c7

cleanup WsTailProxy

a12473f

sandeepsukhani self-requested a review May 26, 2020 15:21

sandeepsukhani reviewed May 27, 2020

View reviewed changes

pkg/loki/modules.go Outdated Show resolved Hide resolved

pkg/loki/modules.go Outdated Show resolved Hide resolved

stale bot added the stale A stale issue or PR that will automatically be closed. label Jul 19, 2020

stale bot removed the stale A stale issue or PR that will automatically be closed. label Jul 19, 2020

sandeepsukhani added the keepalive An issue or PR that will be kept alive and never marked as stale. label Jul 20, 2020

merge master

418b5b2

pull-request-size bot added the size/L label Jul 21, 2020

used httputil.NewSingleHostReverseProxy insted of custom implementati…

0a209d4

…on, added tailproxy config

pull-request-size bot added size/M and removed size/L labels Jul 21, 2020

proxy all unknown requests

e690786

cyriltovena reviewed Jul 22, 2020

View reviewed changes

pkg/loki/modules.go Outdated Show resolved Hide resolved

cyriltovena approved these changes Jul 22, 2020

View reviewed changes

rename proxy to tailProxy

1a87f60

sandeepsukhani reviewed Jul 23, 2020

View reviewed changes

default handler, if tailproxy url is set up default proxy is tailproxy

ef61f93

owen-d reviewed Jul 23, 2020

View reviewed changes

tivvit added 2 commits July 23, 2020 17:34

embedded lokifrontend conf which adds tail_proxy_url

bb23ee3

fixed tail_proxy_url flag

8ce9891

cyriltovena approved these changes Jul 24, 2020

View reviewed changes

cyriltovena merged commit 2e1fd42 into grafana:master Jul 24, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added support for tail to query frontend #2032

Added support for tail to query frontend #2032

tivvit commented May 2, 2020

codecov-io commented May 2, 2020

cyriltovena commented May 5, 2020

tivvit commented May 6, 2020

tivvit commented May 26, 2020

cyriltovena commented May 26, 2020 •

edited

Loading

cyriltovena commented Jun 1, 2020

periklis commented Jun 17, 2020

cyriltovena commented Jun 19, 2020

owen-d commented Jun 19, 2020

stale bot commented Jul 19, 2020

tivvit commented Jul 19, 2020 •

edited

Loading

codecov-commenter commented Jul 21, 2020 •

edited

Loading

tivvit commented Jul 21, 2020

slim-bean commented Jul 22, 2020

cyriltovena left a comment

sandeepsukhani Jul 23, 2020

tivvit Jul 23, 2020

tivvit Jul 23, 2020

tivvit Jul 23, 2020

owen-d left a comment

owen-d Jul 23, 2020 •

edited

Loading

tivvit Jul 23, 2020

owen-d Jul 23, 2020

tivvit Jul 23, 2020

cyriltovena left a comment

Added support for tail to query frontend #2032

Added support for tail to query frontend #2032

Conversation

tivvit commented May 2, 2020

codecov-io commented May 2, 2020

Codecov Report

cyriltovena commented May 5, 2020

tivvit commented May 6, 2020

tivvit commented May 26, 2020

cyriltovena commented May 26, 2020 • edited Loading

cyriltovena commented Jun 1, 2020

periklis commented Jun 17, 2020

cyriltovena commented Jun 19, 2020

owen-d commented Jun 19, 2020

stale bot commented Jul 19, 2020

tivvit commented Jul 19, 2020 • edited Loading

codecov-commenter commented Jul 21, 2020 • edited Loading

Codecov Report

tivvit commented Jul 21, 2020

slim-bean commented Jul 22, 2020

cyriltovena left a comment

Choose a reason for hiding this comment

sandeepsukhani Jul 23, 2020

Choose a reason for hiding this comment

tivvit Jul 23, 2020

Choose a reason for hiding this comment

tivvit Jul 23, 2020

Choose a reason for hiding this comment

tivvit Jul 23, 2020

Choose a reason for hiding this comment

owen-d left a comment

Choose a reason for hiding this comment

owen-d Jul 23, 2020 • edited Loading

Choose a reason for hiding this comment

tivvit Jul 23, 2020

Choose a reason for hiding this comment

owen-d Jul 23, 2020

Choose a reason for hiding this comment

tivvit Jul 23, 2020

Choose a reason for hiding this comment

cyriltovena left a comment

Choose a reason for hiding this comment

cyriltovena commented May 26, 2020 •

edited

Loading

tivvit commented Jul 19, 2020 •

edited

Loading

codecov-commenter commented Jul 21, 2020 •

edited

Loading

owen-d Jul 23, 2020 •

edited

Loading