Compactor: Add tracing support #4903

metonymic-smokey · 2021-11-25T15:53:19Z

Fixes #4832

I added CHANGELOG entry for this change.
Change is not relevant to the end user.

Changes

Added tracing support for the various compaction steps(similar to Querier).

Verification

Tested locally using Jaeger.

Signed-off-by: metonymic-smokey <ahuja.aditi@gmail.com>

matej-g

This is looking cool! I was wondering about one more thing, which I think could be particularly useful in the context of compaction and that is to also record the error in the span (we would have the error in the logs, but I think having them directly on span would be more handy).

We are using our convenience method DoInSpan here, but that does not capture an error if it occurs. Maybe we could have another method (e.g. DoInSpanWithErr) which could also set the error on span. Does that sound good?

pkg/compact/compact.go

Co-authored-by: Matej Gera <38492574+matej-g@users.noreply.github.com> Signed-off-by: metonymic-smokey <ahuja.aditi@gmail.com>

Signed-off-by: metonymic-smokey <ahuja.aditi@gmail.com>

matej-g

Great job @metonymic-smokey! Thanks for going above and beyond to address things. I'm wondering about one last thing, afterwards I believe this is ready.

Let's try to get a review from maintainer as well, perhaps @yeya24 as original issue reporter.

matej-g · 2021-11-30T11:34:10Z

pkg/tracing/tracing.go

+	span, newCtx := StartSpan(ctx, operationName, opts...)
+	defer span.Finish()
+	err := doFn(newCtx)
+	span.LogFields(opentracing_log.Error(err))


I'm not sure about the behavior if err is nil (whether it still logs error), but maybe we could check err != nil before we log error?

I think we can replace this with method from the ext package, see https://pkg.go.dev/github.com/opentracing/opentracing-go@v1.2.0/ext#LogError. This will set all necessary things on the span for us

Good point @matej-g. Thanks!
I've made the changes in a recent commit. Let me know if they're okay.

Signed-off-by: metonymic-smokey <ahuja.aditi@gmail.com>

matej-g

💯

matej-g · 2021-12-01T08:01:12Z

pkg/tracing/tracing.go

+	defer span.Finish()
+	err := doFn(newCtx)
+	if err != nil {
+		ext.LogError(span, err, opentracing_log.Error(err))


I believe we do not need to pass the last value, since the method will already include the field for error.

squat

Looks good overall! Some very small nits

squat · 2021-12-01T10:01:35Z

pkg/compact/compact.go

+			err = block.Download(ctx, cg.logger, cg.bkt, meta.ULID, bdir)
+			return err
+		}, opentracing.Tags{"block.id": meta.ULID})
+


tiny nit

Suggested change

squat · 2021-12-01T10:02:41Z

pkg/compact/compact.go

 			return false, ulid.ULID{}, retry(errors.Wrapf(err, "download block %s", meta.ULID))
 		}

 		// Ensure all input blocks are valid.
-		stats, err := block.GatherIndexHealthStats(cg.logger, filepath.Join(bdir, block.IndexFilename), meta.MinTime, meta.MaxTime)
+		var stats block.HealthStats
+		tracing.DoInSpanWithErr(ctx, "compaction_block_healthstats", func(ctx context.Context) error {


Since the Golang name is ...HealthStats, health and stats should be treated as separate words

Suggested change

tracing.DoInSpanWithErr(ctx, "compaction_block_healthstats", func(ctx context.Context) error {

tracing.DoInSpanWithErr(ctx, "compaction_block_health_stats", func(ctx context.Context) error {

squat · 2021-12-01T10:07:51Z

pkg/compact/compact.go

@@ -764,7 +766,11 @@ func (cg *Group) Compact(ctx context.Context, dir string, planner Planner, comp
 		return false, ulid.ULID{}, errors.Wrap(err, "create compaction group dir")
 	}

-	shouldRerun, compID, err := cg.compact(ctx, subDir, planner, comp)
+	var err error
+	tracing.DoInSpanWithErr(ctx, "group_compaction", func(ctx context.Context) error {


All other spans are named compaction_<x>, should we make this follow the same convention?

Signed-off-by: metonymic-smokey <ahuja.aditi@gmail.com>

squat · 2021-12-01T12:38:53Z

lgtm! restarting failed jobs and we can merge :))

metonymic-smokey · 2021-12-01T12:42:32Z

Thanks for the reviews @matej-g @squat !

metonymic-smokey and others added 16 commits November 18, 2021 18:36

first draft for tracing

9d3b32f

Signed-off-by: metonymic-smokey <ahuja.aditi@gmail.com>

add tracer to context

aa0ede1

Signed-off-by: metonymic-smokey <ahuja.aditi@gmail.com>

minor fixes

2f342ac

Signed-off-by: metonymic-smokey <ahuja.aditi@gmail.com>

create common block spans; log block errors

47c3171

Signed-off-by: metonymic-smokey <ahuja.aditi@gmail.com>

reverted to generic block spans

e1dd35a

Signed-off-by: metonymic-smokey <ahuja.aditi@gmail.com>

Merge branch 'thanos-io:main' into tracing

031aeaf

changed block ID tag; add span for block delete

8104e73

Signed-off-by: metonymic-smokey <ahuja.aditi@gmail.com>

minor fix

e329f85

Signed-off-by: metonymic-smokey <ahuja.aditi@gmail.com>

removed extra spans

ffd55fd

Signed-off-by: metonymic-smokey <ahuja.aditi@gmail.com>

Merge branch 'main' into tracing

480ffe7

addressed some review comments

a3a8f6d

Signed-off-by: metonymic-smokey <ahuja.aditi@gmail.com>

removed block spans

9305a89

Signed-off-by: metonymic-smokey <ahuja.aditi@gmail.com>

changed group key

2f50d22

Signed-off-by: metonymic-smokey <ahuja.aditi@gmail.com>

removed extra var declarations

6d4f0de

Signed-off-by: metonymic-smokey <ahuja.aditi@gmail.com>

Merge branch 'thanos-io:main' into tracing

7a5795e

removed comments

ae6b6d5

Signed-off-by: metonymic-smokey <ahuja.aditi@gmail.com>

metonymic-smokey force-pushed the tracing branch from 0aea0cb to ae6b6d5 Compare November 25, 2021 15:54

metonymic-smokey marked this pull request as draft November 25, 2021 15:58

added changelog entry

3783abd

Signed-off-by: metonymic-smokey <ahuja.aditi@gmail.com>

metonymic-smokey force-pushed the tracing branch from adda8bb to 3783abd Compare November 26, 2021 03:20

metonymic-smokey marked this pull request as ready for review November 26, 2021 05:40

matej-g reviewed Nov 26, 2021

View reviewed changes

pkg/compact/compact.go Outdated Show resolved Hide resolved

matej-g mentioned this pull request Nov 26, 2021

Add tracing support for compaction #4832

Closed

update healthcheck to healthstats.

f12a18e

Co-authored-by: Matej Gera <38492574+matej-g@users.noreply.github.com> Signed-off-by: metonymic-smokey <ahuja.aditi@gmail.com>

metonymic-smokey force-pushed the tracing branch from 3d0dd8f to f12a18e Compare November 27, 2021 02:49

Merge branch 'thanos-io:main' into tracing

7620389

metonymic-smokey force-pushed the tracing branch from e11d776 to 7620389 Compare November 29, 2021 05:15

metonymic-smokey added 2 commits November 29, 2021 15:23

draft: logging errors

a5ef615

Signed-off-by: metonymic-smokey <ahuja.aditi@gmail.com>

removed extra line

ced45c4

Signed-off-by: metonymic-smokey <ahuja.aditi@gmail.com>

matej-g reviewed Nov 30, 2021

View reviewed changes

used alternate function to log errors

9296528

Signed-off-by: metonymic-smokey <ahuja.aditi@gmail.com>

matej-g previously approved these changes Dec 1, 2021

View reviewed changes

squat previously approved these changes Dec 1, 2021

View reviewed changes

minor nits

01812fc

Signed-off-by: metonymic-smokey <ahuja.aditi@gmail.com>

metonymic-smokey dismissed stale reviews from squat and matej-g via 01812fc December 1, 2021 12:11

squat approved these changes Dec 1, 2021

View reviewed changes

squat enabled auto-merge (squash) December 1, 2021 12:39

squat merged commit d08a12a into thanos-io:main Dec 1, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compactor: Add tracing support #4903

Compactor: Add tracing support #4903

metonymic-smokey commented Nov 25, 2021 •

edited

matej-g left a comment

matej-g left a comment

matej-g Nov 30, 2021

metonymic-smokey Nov 30, 2021

matej-g left a comment

matej-g Dec 1, 2021

squat left a comment

squat Dec 1, 2021

squat Dec 1, 2021

squat Dec 1, 2021

squat commented Dec 1, 2021

metonymic-smokey commented Dec 1, 2021

	tracing.DoInSpanWithErr(ctx, "compaction_block_healthstats", func(ctx context.Context) error {
	tracing.DoInSpanWithErr(ctx, "compaction_block_health_stats", func(ctx context.Context) error {

Compactor: Add tracing support #4903

Compactor: Add tracing support #4903

Conversation

metonymic-smokey commented Nov 25, 2021 • edited

Changes

Verification

matej-g left a comment

Choose a reason for hiding this comment

matej-g left a comment

Choose a reason for hiding this comment

matej-g Nov 30, 2021

Choose a reason for hiding this comment

metonymic-smokey Nov 30, 2021

Choose a reason for hiding this comment

matej-g left a comment

Choose a reason for hiding this comment

matej-g Dec 1, 2021

Choose a reason for hiding this comment

squat left a comment

Choose a reason for hiding this comment

squat Dec 1, 2021

Choose a reason for hiding this comment

squat Dec 1, 2021

Choose a reason for hiding this comment

squat Dec 1, 2021

Choose a reason for hiding this comment

squat commented Dec 1, 2021

metonymic-smokey commented Dec 1, 2021

metonymic-smokey commented Nov 25, 2021 •

edited