Exemplar: Can not set partial response.

[hanjm]

Thanos, Prometheus and Golang version used:

Object Storage Provider:
COS
What happened:
query_exemplar response error:
error: "retrieving exemplars: proxy Exemplars: receiving exemplars from exemplars client &{0xc000b2a000}: rpc error: code = Unimplemented desc = unknown service thanos.Exemplars"

What you expected to happen:
partial response with warning.
How to reproduce it (as minimally and precisely as possible):
Thanos Query 0.23 beta + a old version sidecar not support exemplar
Full logs to relevant components:

Anything else we need to know:

seems missing flag to control exemplar partial reponse

[hanjm]

if I add a flag and pass it to NewExemplarsHandler, https://github.com/thanos-io/thanos/blob/main/pkg/api/query/v1.go#L798, thanos-query run OOM when query. may be too many exemplars? could we set a number limit to it?

pprof heap is here

File: thanos
Type: inuse_space
Time: Sep 18, 2021 at 9:52pm (CST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top10
Showing nodes accounting for 461.11MB, 97.97% of 470.68MB total
Dropped 66 nodes (cum <= 2.35MB)
Showing top 10 nodes out of 37
      flat  flat%   sum%        cum   cum%
  256.06MB 54.40% 54.40%   256.06MB 54.40%  github.com/pkg/errors.callers
  140.52MB 29.85% 84.26%   142.02MB 30.17%  github.com/pkg/errors.(*withMessage).Error
   23.50MB  4.99% 89.25%   271.56MB 57.69%  github.com/pkg/errors.New (inline)
   16.95MB  3.60% 92.85%   288.50MB 61.30%  github.com/thanos-io/thanos/pkg/exemplars.(*exemplarsServer).Send
    7.98MB  1.70% 94.54%     7.98MB  1.70%  google.golang.org/grpc/internal/transport.newBufWriter
       6MB  1.27% 95.82%        6MB  1.27%  reflect.New
    3.61MB  0.77% 96.59%     3.61MB  0.77%  bufio.NewReaderSize
    3.50MB  0.74% 97.33%     4.04MB  0.86%  fmt.Sprintf
       2MB  0.42% 97.76%     3.50MB  0.74%  github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/tracing.(*opentracingClientReporter).PostCall
       1MB  0.21% 97.97%        3MB  0.64%  google.golang.org/grpc/internal/status.(*Error).Error
(pprof)
(pprof) traces github.com/pkg/errors.callers
File: thanos
Type: inuse_space
Time: Sep 18, 2021 at 9:52pm (CST)
-----------+-------------------------------------------------------
     bytes:  24B
  512.01kB   github.com/pkg/errors.callers
             github.com/pkg/errors.Wrapf
             github.com/thanos-io/thanos/pkg/exemplars.(*exemplarsStream).receive
             github.com/thanos-io/thanos/pkg/exemplars.(*Proxy).Exemplars.func1
             golang.org/x/sync/errgroup.(*Group).Go.func1
-----------+-------------------------------------------------------
     bytes:  24B
   20.50MB   github.com/pkg/errors.callers
             github.com/pkg/errors.New
             github.com/thanos-io/thanos/pkg/exemplars.(*exemplarsServer).Send
             github.com/thanos-io/thanos/pkg/exemplars.(*exemplarsStream).receive
             github.com/thanos-io/thanos/pkg/exemplars.(*Proxy).Exemplars.func1
             golang.org/x/sync/errgroup.(*Group).Go.func1
-----------+-------------------------------------------------------
     bytes:  256B
    7.50MB   github.com/pkg/errors.callers
             github.com/pkg/errors.Wrapf
             github.com/thanos-io/thanos/pkg/exemplars.(*exemplarsStream).receive
             github.com/thanos-io/thanos/pkg/exemplars.(*Proxy).Exemplars.func1
             golang.org/x/sync/errgroup.(*Group).Go.func1
-----------+-------------------------------------------------------
     bytes:  256B
  227.56MB   github.com/pkg/errors.callers
             github.com/pkg/errors.New
             github.com/thanos-io/thanos/pkg/exemplars.(*exemplarsServer).Send
             github.com/thanos-io/thanos/pkg/exemplars.(*exemplarsStream).receive
             github.com/thanos-io/thanos/pkg/exemplars.(*Proxy).Exemplars.func1
             golang.org/x/sync/errgroup.(*Group).Go.func1
-----------+-------------------------------------------------------

[yeya24]

So actually partial response in the Exemplars API works, right? @hanjm Can you help confirm this?

If the flag works, then let's rename the issue to discuss the exemplars limit.

[hanjm]

@yeya24 I found seems missing exemplar.partial-response flag, so i add exemplar.partial-response flag in hanjm@681608c, Partial response works in my brach.

[yeya24]

I see. @hanjm Looks like we have this config in the struct but forget to add a flag for it. Would you like to open a pr for it?

[hanjm]

I see. @hanjm Looks like we have this config in the struct but forget to add a flag for it. Would you like to open a pr for it?

ok.

[hanjm]

I am investigate it.
from pprof heap, it seems too many warning message consume the memory.

then i add a debug log to (*exemplarsServer).Send https://github.com/thanos-io/thanos/blob/main/pkg/exemplars/exemplars.go#L42

func (srv *exemplarsServer) Send(res *exemplarspb.ExemplarsResponse) error {
	if res.GetWarning() != "" {
		err := errors.New(res.GetWarning())
		log.Printf("err message size: errors:%d, srv.warnings:%d, res.GetWarning():%d",
			len(err.Error()),
			len(srv.warnings),
			len(res.GetWarning()))
		srv.warnings = append(srv.warnings, err)
		return nil
	}

it will print a lot of logs like

2021-09-19_12:19:54 2021/09/19 04:19:54 err message size: errors:130, srv.warnings:4277283, res.GetWarning():130

[hanjm]

then i print svr.warnings first ten message.

func (srv *exemplarsServer) Send(res *exemplarspb.ExemplarsResponse) error {
	if res.GetWarning() != "" {
		err := errors.New(res.GetWarning())
		if len(srv.warnings) == 100 {
			log.Printf("err message size: errors:%d, srv.warnings:%d, res.GetWarning():%d, srv.warnings:%+v",
				len(err.Error()),
				len(srv.warnings),
				len(res.GetWarning()),
				srv.warnings[:10],
			)
		}

		srv.warnings = append(srv.warnings, err)
		return nil
	}

it print a log like

2021/09/19 06:09:37 err message size: errors:130, srv.warnings:100, res.GetWarning():130, srv.warnings:[receiving exemplars from exemplars client &{0xc0000a0700}: rpc error: code = Unimplemented desc = unknown service thanos.Exemplars
github.com/thanos-io/thanos/pkg/exemplars.(*exemplarsServer).Send
        /Users/jimmiehan/github.com/hanjm/thanos/pkg/exemplars/exemplars.go:45
github.com/thanos-io/thanos/pkg/exemplars.(*exemplarsStream).receive
        /Users/jimmiehan/github.com/hanjm/thanos/pkg/exemplars/proxy.go:193
github.com/thanos-io/thanos/pkg/exemplars.(*Proxy).Exemplars.func1
        /Users/jimmiehan/github.com/hanjm/thanos/pkg/exemplars/proxy.go:133
golang.org/x/sync/errgroup.(*Group).Go.func1
        /Users/jimmiehan/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57
runtime.goexit
        /Users/jimmiehan/sdk/go1.16.3/src/runtime/asm_amd64.s:1371 receiving exemplars from exemplars client &{0xc0000a0700}: rpc error: code = Unimplemented desc = unknown service thanos.Exemplars
github.com/thanos-io/thanos/pkg/exemplars.(*exemplarsServer).Send
        /Users/jimmiehan/github.com/hanjm/thanos/pkg/exemplars/exemplars.go:45
github.com/thanos-io/thanos/pkg/exemplars.(*exemplarsStream).receive
        /Users/jimmiehan/github.com/hanjm/thanos/pkg/exemplars/proxy.go:193
github.com/thanos-io/thanos/pkg/exemplars.(*Proxy).Exemplars.func1
        /Users/jimmiehan/github.com/hanjm/thanos/pkg/exemplars/proxy.go:133
golang.org/x/sync/errgroup.(*Group).Go.func1
        /Users/jimmiehan/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57
runtime.goexit
        /Users/jimmiehan/sdk/go1.16.3/src/runtime/asm_amd64.s:1371 receiving exemplars from exemplars client &{0xc0000a0700}: rpc error: code = Unimplemented desc = unknown service thanos.Exemplars
github.com/thanos-io/thanos/pkg/exemplars.(*exemplarsServer).Send
        /Users/jimmiehan/github.com/hanjm/thanos/pkg/exemplars/exemplars.go:45
github.com/thanos-io/thanos/pkg/exemplars.(*exemplarsStream).receive
        /Users/jimmiehan/github.com/hanjm/thanos/pkg/exemplars/proxy.go:193
github.com/thanos-io/thanos/pkg/exemplars.(*Proxy).Exemplars.func1
        /Users/jimmiehan/github.com/hanjm/thanos/pkg/exemplars/proxy.go:133
golang.org/x/sync/errgroup.(*Group).Go.func1
        /Users/jimmiehan/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57
runtime.goexit

[hanjm]

Seems it is better if keep one warning in single gRPC bid-stream call. instead of multi warning . @yeya24 do you think so?

[MrYueQ]

Hi. The version of thanos has been rolled back to 0.22. In addition, in the current version, --query-frontend.downstream-url uses the load balancing mechanism, and the load balancing mechanism is weighted round-robin. No matter how the client's IP changes. The upstream of the backend will only be routed to the same IP.

thanos-query host resource

thanos-query-frontend host resource

[hanjm]

I found the root cause: If the store-api target not implement exemplar API, err is rpc error: code = Unimplemented desc = unknown service thanos.Exemplars. the grpc bid-steam loop will be infinite loop due to the wrong error handing.
grpc doc say https://github.com/grpc/grpc-go/blob/v1.40.0/stream.go#L123

	// RecvMsg blocks until it receives a message into m or the stream is
	// done. It returns io.EOF when the stream completes successfully. On
	// any other error, the stream is aborted and the error contains the RPC
	// status.

https://github.com/thanos-io/thanos/blob/main/pkg/exemplars/proxy.go#L195

for {
		exemplar, err := exemplars.Recv()
		if err == io.EOF {
			return nil
		}

		if err != nil {
			err = errors.Wrapf(err, "receiving exemplars from exemplars client %v", stream.client)

			if stream.request.PartialResponseStrategy == storepb.PartialResponseStrategy_ABORT {
				return err
			}

			if err := stream.server.Send(exemplarspb.NewWarningExemplarsResponse(err)); err != nil {
				return errors.Wrapf(err, "sending exemplars error to server %v", stream.server)
			}

			continue
		}

		if w := exemplar.GetWarning(); w != "" {
			if err := stream.server.Send(exemplarspb.NewWarningExemplarsResponse(errors.New(w))); err != nil {
				return errors.Wrapf(err, "sending exemplars warning to server %v", stream.server)
			}
			continue
		}

		select {
		case stream.channel <- exemplar.GetData():
		case <-ctx.Done():
			return ctx.Err()
		}
	}

[hanjm]

Fix at PR #4681 , should merge #4680 first.

[MrYueQ]

[hanjm]

Hi. The version of thanos has been rolled back to 0.22. In addition, in the current version, --query-frontend.downstream-url uses the load balancing mechanism, and the load balancing mechanism is weighted round-robin. No matter how the client's IP changes. The upstream of the backend will only be routed to the same IP.

@MrYueQ Seems not relevent? Please feel free to open another issue ~