[processor/tailsampling] add OTTL Condition policy #20890

jiekun · 2023-04-13T16:33:03Z

Description: We could solve #20294 by introducing OTTL to the tail sampling processor. By using the new policy, the processor can make sampling decision based on the OTTL statement.

Link to tracking Issue: #20294

Testing: See processor/tailsamplingprocessor/internal/sampling/ottl_test.go.

Documentation: See README.me

…ocessor

…policy; added ctx to evaluate interface method

processor/tailsamplingprocessor/internal/sampling/ottl.go

processor/tailsamplingprocessor/config.go

linux-foundation-easycla · 2023-04-19T13:37:45Z

The committers listed above are authorized under a signed CLA.

✅ login: jiekun / name: Jiekun (506de65, 0c34427, 268e6cc, 504080a, 3ab2f29, 063e3fc)

…ampling ottl policy

…eval logic into different function

…when eval error happened

jiekun · 2023-04-19T16:00:03Z

I will do the performance evaluation tomorrow.

TylerHelmuth

Please update the README and add a changelog entry

processor/tailsamplingprocessor/config.go

processor/tailsamplingprocessor/internal/sampling/ottl.go

linux-foundation-easycla · 2023-04-20T01:37:37Z

The committers listed above are authorized under a signed CLA.

✅ login: jiekun / name: Jiekun (506de65, 0c34427, 268e6cc, 504080a, 3ab2f29, 063e3fc, 9572e02, 54ad895, f97d1dd, 146865d, cbda9d5, c89970a)

…o ottl condition; added readme and changelog; removed unnecessary funcchain

processor/tailsamplingprocessor/internal/sampling/ottl.go

TylerHelmuth

This looks good from an OTTL perspective. @jpkrohling something to consider for the future is that this policy option has overlapping capabilities with many other policy options. IDK how much flexibility there is to change this processor's config, but it could be beneficial to deprecate the duplicate policies.

…r-ottl

jiekun · 2023-05-04T05:15:22Z

/ptal conflict resolved @jpkrohling

MovieStoreGuy · 2023-05-06T03:45:40Z

This looks good from an OTTL perspective. @jpkrohling something to consider for the future is that this policy option has overlapping capabilities with many other policy options. IDK how much flexibility there is to change this processor's config, but it could be beneficial to deprecate the duplicate policies.

@TylerHelmuth , did you want create a follow up issue for this? Or should this be decided before this PR is merged?

TylerHelmuth · 2023-05-06T14:04:58Z

@MovieStoreGuy definitely doesn't need to hold up this PR. Only needs to be an issue if @jpkrohling wants to take the processor that direction.

jiekun · 2023-05-10T16:46:37Z

Alright I think I could solve the conflict if we are going to do the merge. So we don't need to trigger CI flow again and again.

TylerHelmuth · 2023-05-10T17:42:28Z

@jiekun I'd like the code owner @jpkrohling to review before we merge bc this is a significant change. Feel free to hold off handling any merge conflicts until he has reviewed.

processor/tailsamplingprocessor/and_helper.go

jpkrohling · 2023-05-10T17:22:46Z

processor/tailsamplingprocessor/config_test.go

+						Type: OTTLCondition,
+						OTTLConditionCfg: OTTLConditionCfg{
+							ErrorMode:           ottl.IgnoreError,
+							SpanConditions:      []string{"attributes[\"test_attr_key_1\"] == \"test_attr_val_1\"", "attributes[\"test_attr_key_2\"] != \"test_attr_val_1\""},


Out of curiosity: would single quotes work to wrap the attribute names (such as test_attr_key_1)?

No, OTTL requires double quotes "". In situations like these you could create the strings using back ticks ` instead of ".

processor/tailsamplingprocessor/internal/sampling/ottl.go

jpkrohling · 2023-05-10T17:39:44Z

processor/tailsamplingprocessor/internal/sampling/ottl.go

+}
+
+func (ocf *ottlConditionFilter) Evaluate(ctx context.Context, _ pcommon.TraceID, trace *TraceData) (Decision, error) {
+	ocf.logger.Debug("Evaluating with OTTL conditions filter")


Is there a way to make this message more useful? Like, by adding the trace ID to it? If the purpose is to give the user feedback that this works, you may want a metric counter instead.

I prefer to add a traceID to it.

Actually the metric counter would be better. However, we have this debug log for all other tail sampling policies. I think it's bad when people see the log from other policies but empty output for ottl. They may wonder if it's not working.

And it's under Debug log level which may help a very little bit. I think we should either replace all of them with a metric, or keep the log and add more information to make it useful. WDYT?

opentelemetry-collector-contrib/processor/tailsamplingprocessor/internal/sampling/span_count_sampler.go

Line 41 in be4b4ef

c.logger.Debug("Evaluating spans counts in filter")

jpkrohling · 2023-05-10T17:48:55Z

processor/tailsamplingprocessor/internal/sampling/ottl.go

+
+				// Span evaluation
+				if ocf.sampleSpanExpr != nil {
+					ocf.logger.Debug("Evaluating spans with OTTL conditions filter")


Same as the previous comment. While this is useful for developing this component, extracting value from this log line is hard when you are operating the collector. Perhaps a counter would make sense here as well?

I removed 2 unnecessary logging and keep the entry one only.

jpkrohling · 2023-05-10T17:49:56Z

processor/tailsamplingprocessor/internal/sampling/ottl.go

+
+				// Span event evaluation
+				if ocf.sampleSpanEventExpr != nil {
+					ocf.logger.Debug("Evaluating span events with OTTL conditions filter")


jpkrohling · 2023-05-10T17:51:13Z

processor/tailsamplingprocessor/internal/sampling/ottl_test.go

+	}{
+		{
+			// policy
+			"OTTL conditions not set",


Should this be an error condition? At least one of the conditions should be specified, I suppose.

I referred to the impl of filterprocessor:
https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/be4b4efbb2dd32c9b1c7ded8720979fdeac33cd2/processor/filterprocessor/traces.go#LL45C3-L45C3

I think it's still good to stop user from using a blank condition(mistakenly). But I am not confident about the necessity to change it since we have validation at the begining of Evaluate:

if ocf.sampleSpanExpr == nil && ocf.sampleSpanEventExpr == nil { return NotSampled, nil }

Feel free to let me know and I am ready to change the init function to return an error when empty ottl condition is set.

Edit:
OK I prefer to check the condition during init. NewOTTLConditionFilter is updated with check:

if len(spanConditions) == 0 && len(spanEventConditions) == 0 { return nil, errors.New("expected at least one OTTL condition to filter on") }

Test case updated as well, which expect an error.

processor/tailsamplingprocessor/internal/sampling/ottl_test.go

jpkrohling · 2023-05-10T17:57:15Z

I'm fine merging this as is if the items I mentioned during my review are addressed in a follow-up PR.

…aceID; Added inverse match test case; unexposed errorMode field

…flict in go.mod

…ion config

jiekun · 2023-05-11T03:54:16Z

/ptal @jpkrohling Thanks for reviewing. I am lookging forward for more input since it's my first PR.

TylerHelmuth

@jiekun please resolve the conflicts then we'll be good to go.

jiekun · 2023-05-12T01:33:06Z

/ptal @TylerHelmuth. Thanks for reviewing. I love how the OTel community responds to newbies' PR. I'm really looking forward to my 2nd contribution.

) * [feature: OTTL sampling] Added OTTL config for tail-based sampling processor * [feature: OTTL sampling] Added OTTL impl struct * [feature: OTTL sampling] Change Query to Statement * [feature: OTTL sampling] Added bool evaluator for ottl tail sampling policy; added ctx to evaluate interface method * [feature: OTTL sampling] rename Eval response var * [feature: OTTL sampling] Added span and span event support for tail sampling ottl policy * [feature: OTTL sampling] Re-format the Evaluation method, move heavy eval logic into different function * [feature: OTTL sampling] Fixed typo * [feature: OTTL sampling] Simplified error log. return error decision when eval error happened * [feature: OTTL sampling] fixed typo and add debug log * [feature: OTTL sampling] Added config test for OTTL statement policy * [feature: OTTL sampling] Change after reivew: change ottl statement to ottl condition; added readme and changelog; removed unnecessary funcchain * [feature: OTTL sampling] make crosslink * [feature: OTTL sampling] commit tidy lint * Added license header to test file. * [feature: OTTL sampling] Change tail sampling processor sig to receive component otel settings instead of logger * [feature: OTTL sampling] Fixed lint issue * [feature: OTTL sampling] Fixed hardcoded errMode * [feature: unique trace path sampling] Review Changes: Logging with traceID; Added inverse match test case; unexposed errorMode field * [feature: OTTL tail sampling] Review Changes: Block empty OTTL condition config * [feature: ottl tail sampling policy] Sorting go.mod

jiekun added 5 commits April 13, 2023 09:40

[feature: OTTL sampling] Added OTTL config for tail-based sampling pr…

506de65

…ocessor

[feature: OTTL sampling] Added OTTL impl struct

0c34427

[feature: OTTL sampling] Change Query to Statement

268e6cc

[feature: OTTL sampling] Added bool evaluator for ottl tail sampling …

504080a

…policy; added ctx to evaluate interface method

[feature: OTTL sampling] rename Eval response var

3ab2f29

github-actions bot added the processor/tailsampling Tail sampling processor label Apr 13, 2023

github-actions bot requested a review from jpkrohling April 13, 2023 16:33

TylerHelmuth reviewed Apr 14, 2023

View reviewed changes

processor/tailsamplingprocessor/internal/sampling/ottl.go Outdated Show resolved Hide resolved

processor/tailsamplingprocessor/config.go Outdated Show resolved Hide resolved

[feature: OTTL sampling] Added span and span event support for tail s…

063e3fc

…ampling ottl policy

jiekun force-pushed the feature/tailsampling-processor-ottl branch from 93386ed to 063e3fc Compare April 19, 2023 13:43

jiekun added 5 commits April 19, 2023 23:39

[feature: OTTL sampling] Re-format the Evaluation method, move heavy …

9572e02

…eval logic into different function

[feature: OTTL sampling] Fixed typo

54ad895

[feature: OTTL sampling] Simplified error log. return error decision …

f97d1dd

…when eval error happened

[feature: OTTL sampling] fixed typo and add debug log

146865d

[feature: OTTL sampling] Added config test for OTTL statement policy

cbda9d5

jiekun marked this pull request as ready for review April 19, 2023 15:56

jiekun requested a review from a team as a code owner April 19, 2023 15:56

jiekun changed the title ~~WIP: feat: OTTL policy for tailsampling processor~~ feat: OTTL policy for tailsampling processor Apr 19, 2023

github-actions bot assigned MovieStoreGuy Apr 19, 2023

TylerHelmuth reviewed Apr 19, 2023

View reviewed changes

jiekun changed the title ~~feat: OTTL policy for tailsampling processor~~ [processor/tailsampling] add OTTL Condition policy Apr 20, 2023

[feature: OTTL sampling] Change after reivew: change ottl statement t…

c89970a

…o ottl condition; added readme and changelog; removed unnecessary funcchain

jiekun force-pushed the feature/tailsampling-processor-ottl branch from ce47862 to c89970a Compare April 20, 2023 03:27

jiekun added 3 commits April 20, 2023 11:32

[feature: OTTL sampling] make crosslink

ae8c3b1

[feature: OTTL sampling] commit tidy lint

14d757e

Added license header to test file.

7db819b

TylerHelmuth reviewed Apr 20, 2023

View reviewed changes

processor/tailsamplingprocessor/internal/sampling/ottl.go Outdated Show resolved Hide resolved

processor/tailsamplingprocessor/internal/sampling/ottl.go Outdated Show resolved Hide resolved

TylerHelmuth reviewed Apr 26, 2023

View reviewed changes

processor/tailsamplingprocessor/internal/sampling/ottl.go Outdated Show resolved Hide resolved

processor/tailsamplingprocessor/internal/sampling/ottl.go Outdated Show resolved Hide resolved

[feature: OTTL sampling] Fixed hardcoded errMode

85b02c0

jiekun requested a review from TylerHelmuth April 27, 2023 02:16

TylerHelmuth approved these changes Apr 27, 2023

View reviewed changes

jiekun added 2 commits April 28, 2023 09:59

Merge branch 'open-telemetry:main' into feature/tailsampling-processo…

8ae924b

…r-ottl

[feature: OTTL sampling] Merge upstream and solve conflict in go.mod

aa7c81f

jpkrohling reviewed May 10, 2023

View reviewed changes

jiekun added 4 commits May 11, 2023 11:25

[feature: unique trace path sampling] Review Changes: Logging with tr…

0b8775e

…aceID; Added inverse match test case; unexposed errorMode field

[feature: unique trace path sampling] Merge main branch and fixed con…

efab392

…flict in go.mod

[feature: OTTL tail sampling] Review Changes: Block empty OTTL condit…

bea6d6b

…ion config

[feature: OTTL tail sampling] Fixed conflicts

41a2af2

jiekun requested a review from jpkrohling May 11, 2023 03:54

jpkrohling approved these changes May 11, 2023

View reviewed changes

TylerHelmuth reviewed May 11, 2023

View reviewed changes

jiekun added 2 commits May 12, 2023 09:26

[feature: ottl tail sampling policy] solve conflict

b86f58c

[feature: ottl tail sampling policy] Sorting go.mod

1a42bce

TylerHelmuth merged commit 86b886d into open-telemetry:main May 12, 2023

github-actions bot added this to the next release milestone May 12, 2023

jiekun deleted the feature/tailsampling-processor-ottl branch May 12, 2023 06:37

ZenoCC-Peng mentioned this pull request Jun 30, 2023

[chore] Remove skip flaky test for tailsamplingprocessor #23852

Merged

jiekun mentioned this pull request Dec 13, 2023

REQUEST: New membership for @jiekun open-telemetry/community#1835

Closed

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[processor/tailsampling] add OTTL Condition policy #20890

[processor/tailsampling] add OTTL Condition policy #20890

jiekun commented Apr 13, 2023 •

edited

Loading

linux-foundation-easycla bot commented Apr 19, 2023 •

edited

Loading

jiekun commented Apr 19, 2023

TylerHelmuth left a comment

linux-foundation-easycla bot commented Apr 20, 2023 •

edited

Loading

TylerHelmuth left a comment

jiekun commented May 4, 2023

MovieStoreGuy commented May 6, 2023

TylerHelmuth commented May 6, 2023

jiekun commented May 10, 2023

TylerHelmuth commented May 10, 2023

jpkrohling May 10, 2023

TylerHelmuth May 10, 2023

jpkrohling May 10, 2023

jiekun May 11, 2023

jpkrohling May 10, 2023

jiekun May 11, 2023

jpkrohling May 10, 2023

jpkrohling May 10, 2023

jiekun May 11, 2023 •

edited

Loading

jpkrohling commented May 10, 2023

jiekun commented May 11, 2023

TylerHelmuth left a comment

jiekun commented May 12, 2023

[processor/tailsampling] add OTTL Condition policy #20890

[processor/tailsampling] add OTTL Condition policy #20890

Conversation

jiekun commented Apr 13, 2023 • edited Loading

linux-foundation-easycla bot commented Apr 19, 2023 • edited Loading

jiekun commented Apr 19, 2023

TylerHelmuth left a comment

Choose a reason for hiding this comment

linux-foundation-easycla bot commented Apr 20, 2023 • edited Loading

TylerHelmuth left a comment

Choose a reason for hiding this comment

jiekun commented May 4, 2023

MovieStoreGuy commented May 6, 2023

TylerHelmuth commented May 6, 2023

jiekun commented May 10, 2023

TylerHelmuth commented May 10, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jiekun May 11, 2023 • edited Loading

Choose a reason for hiding this comment

jpkrohling commented May 10, 2023

jiekun commented May 11, 2023

TylerHelmuth left a comment

Choose a reason for hiding this comment

jiekun commented May 12, 2023

jiekun commented Apr 13, 2023 •

edited

Loading

linux-foundation-easycla bot commented Apr 19, 2023 •

edited

Loading

linux-foundation-easycla bot commented Apr 20, 2023 •

edited

Loading

jiekun May 11, 2023 •

edited

Loading