Skip to content

Commit

Permalink
[processor/transform] Add common where clause (open-telemetry#31491)
Browse files Browse the repository at this point in the history
**Description:**  Add global conditions  with where clause 
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->

**Link to tracking Issue:** Fixes open-telemetry#27830

**Testing:** Unit tests

**Documentation:** TODO 

~~The main objective is to extend the `ContextStatements` struct by
adding a new `Conditions` parameter. By introducing `Conditions` to
`ContextStatements`, we can now apply a global condition to all related
statements in `WithStatementSequenceGlobalConditions` function.~~

Thanks in advance for your feedback! If this changes will be fine, I
will add common where clause into another context `span`, `metrics`.
  • Loading branch information
lkwronski committed May 9, 2024
1 parent c0512b9 commit a4b0e59
Show file tree
Hide file tree
Showing 16 changed files with 923 additions and 25 deletions.
27 changes: 27 additions & 0 deletions .chloggen/lkwronski.issue-27830-common-where-clause.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Use this changelog template to create an entry for release notes.

# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
change_type: enhancement

# The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver)
component: processor/transform

# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
note: Allow common where clause

# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
issues: [27830]

# (Optional) One or more lines of additional information to render under the primary note.
# These lines will be padded with 2 spaces and then inserted directly into the document.
# Use pipe (|) for multiline entries.
subtext:

# If your change doesn't affect end users or the exported elements of any package,
# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
# Optional: The change log or logs in which this entry should be included.
# e.g. '[user]' or '[user, api]'
# Include 'user' if the change is relevant to end users.
# Include 'api' if there is a change to a library API.
# Default: '[user]'
change_logs: [user]
10 changes: 10 additions & 0 deletions internal/filter/expr/matcher.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,16 @@ func Not[K any](matcher BoolExpr[K]) BoolExpr[K] {
return notMatcher[K]{matcher: matcher}
}

type alwaysTrueMatcher[K any] struct{}

func (alm alwaysTrueMatcher[K]) Eval(_ context.Context, _ K) (bool, error) {
return true, nil
}

func AlwaysTrue[K any]() BoolExpr[K] {
return alwaysTrueMatcher[K]{}
}

type orMatcher[K any] struct {
matchers []BoolExpr[K]
}
Expand Down
17 changes: 17 additions & 0 deletions internal/filter/filterottl/filter.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import (
"github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/contexts/ottllog"
"github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/contexts/ottlmetric"
"github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/contexts/ottlresource"
"github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/contexts/ottlscope"
"github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/contexts/ottlspan"
"github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/contexts/ottlspanevent"
)
Expand Down Expand Up @@ -111,3 +112,19 @@ func NewBoolExprForResource(conditions []string, functions map[string]ottl.Facto
c := ottlresource.NewConditionSequence(statements, set, ottlresource.WithConditionSequenceErrorMode(errorMode))
return &c, nil
}

// NewBoolExprForScope creates a BoolExpr[ottlscope.TransformContext] that will return true if any of the given OTTL conditions evaluate to true.
// The passed in functions should use the ottlresource.TransformContext.
// If a function named `match` is not present in the function map it will be added automatically so that parsing works as expected
func NewBoolExprForScope(conditions []string, functions map[string]ottl.Factory[ottlscope.TransformContext], errorMode ottl.ErrorMode, set component.TelemetrySettings) (expr.BoolExpr[ottlscope.TransformContext], error) {
parser, err := ottlscope.NewParser(functions, set)
if err != nil {
return nil, err
}
statements, err := parser.ParseConditions(conditions)
if err != nil {
return nil, err
}
c := ottlscope.NewConditionSequence(statements, set, ottlscope.WithConditionSequenceErrorMode(errorMode))
return &c, nil
}
51 changes: 51 additions & 0 deletions internal/filter/filterottl/filter_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ import (
"github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/contexts/ottllog"
"github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/contexts/ottlmetric"
"github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/contexts/ottlresource"
"github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/contexts/ottlscope"
"github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/contexts/ottlspan"
"github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/contexts/ottlspanevent"
)
Expand Down Expand Up @@ -270,3 +271,53 @@ func Test_NewBoolExprForResource(t *testing.T) {
})
}
}

func Test_NewBoolExprForScope(t *testing.T) {
tests := []struct {
name string
conditions []string
expectedResult bool
}{
{
name: "basic",
conditions: []string{
"true == true",
},
expectedResult: true,
},
{
name: "multiple conditions resulting true",
conditions: []string{
"false == true",
"true == true",
},
expectedResult: true,
},
{
name: "multiple conditions resulting false",
conditions: []string{
"false == true",
"true == false",
},
expectedResult: false,
},
{
name: "With Converter",
conditions: []string{
`IsMatch("test", "pass")`,
},
expectedResult: false,
},
}

for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
resBoolExpr, err := NewBoolExprForScope(tt.conditions, StandardScopeFuncs(), ottl.PropagateError, componenttest.NewNopTelemetrySettings())
assert.NoError(t, err)
assert.NotNil(t, resBoolExpr)
result, err := resBoolExpr.Eval(context.Background(), ottlscope.TransformContext{})
assert.NoError(t, err)
assert.Equal(t, tt.expectedResult, result)
})
}
}
5 changes: 5 additions & 0 deletions internal/filter/filterottl/functions.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ import (
"github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/contexts/ottllog"
"github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/contexts/ottlmetric"
"github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/contexts/ottlresource"
"github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/contexts/ottlscope"
"github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/contexts/ottlspan"
"github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/contexts/ottlspanevent"
"github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/ottlfuncs"
Expand All @@ -40,6 +41,10 @@ func StandardDataPointFuncs() map[string]ottl.Factory[ottldatapoint.TransformCon
return ottlfuncs.StandardConverters[ottldatapoint.TransformContext]()
}

func StandardScopeFuncs() map[string]ottl.Factory[ottlscope.TransformContext] {
return ottlfuncs.StandardConverters[ottlscope.TransformContext]()
}

func StandardLogFuncs() map[string]ottl.Factory[ottllog.TransformContext] {
return ottlfuncs.StandardConverters[ottllog.TransformContext]()
}
Expand Down
32 changes: 28 additions & 4 deletions processor/transformprocessor/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@

The transform processor modifies telemetry based on configuration using the [OpenTelemetry Transformation Language](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl).

For each signal type, the processor takes a list of statements associated to a [Context type](#contexts) and executes the statements against the incoming telemetry in the order specified in the config.
Each statement can access and transform telemetry using functions and allow the use of a condition to help decide whether the function should be executed.
For each signal type, the processor takes a list of conditions and statements associated to a [Context type](#contexts) and executes the conditions and statements against the incoming telemetry in the order specified in the config.
Each condition and statement can access and transform telemetry using functions and allow the use of a condition to help decide whether the function should be executed.

- [Config](#config)
- [Grammar](#grammar)
Expand All @@ -28,8 +28,8 @@ Each statement can access and transform telemetry using functions and allow the

The transform processor allows configuring multiple context statements for traces, metrics, and logs.
The value of `context` specifies which [OTTL Context](#contexts) to use when interpreting the associated statements.
The statement strings, which must be OTTL compatible, will be passed to the OTTL and interpreted using the associated context.
Each context will be processed in the order specified and each statement for a context will be executed in the order specified.
The conditions and statement strings, which must be OTTL compatible, will be passed to the OTTL and interpreted using the associated context. The conditions string should contain a string with a WHERE clause body without the `where` keyword at the beginning.
Each context will be processed in the order specified and each condition and statement for a context will be executed in the order specified. Conditions are executed first, if a context doesn't meet the conditions, the associated statement will be skipped.

The transform processor also allows configuring an optional field, `error_mode`, which will determine how the processor reacts to errors that occur while processing a statement.

Expand All @@ -46,6 +46,9 @@ transform:
error_mode: ignore
<trace|metric|log>_statements:
- context: string
conditions:
- string
- string
statements:
- string
- string
Expand All @@ -67,6 +70,27 @@ Valid values for `context` are:
| metric_statements | `resource`, `scope`, `metric`, and `datapoint` |
| log_statements | `resource`, `scope`, and `log` |

`conditions` is a list comprised of multiple where clauses, which will be processed as global conditions for the accompanying set of statements.

```yaml
transform:
error_mode: ignore
metric_statements:
- context: metric
conditions:
- type == METRIC_DATA_TYPE_SUM
statements:
- set(description, "Sum")

log_statements:
- context: log
conditions:
- IsMap(body) and body["object"] != nil
statements:
- set(body, attributes["http.route"])
```


### Example

The example takes advantage of context efficiency by grouping transformations with the context which it intends to transform.
Expand Down
33 changes: 33 additions & 0 deletions processor/transformprocessor/config_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,39 @@ func TestLoadConfig(t *testing.T) {
},
},
},
{
id: component.NewIDWithName(metadata.Type, "with_conditions"),
expected: &Config{
ErrorMode: ottl.PropagateError,
TraceStatements: []common.ContextStatements{
{
Context: "span",
Conditions: []string{`attributes["http.path"] == "/animal"`},
Statements: []string{
`set(name, "bear")`,
},
},
},
MetricStatements: []common.ContextStatements{
{
Context: "datapoint",
Conditions: []string{`attributes["http.path"] == "/animal"`},
Statements: []string{
`set(metric.name, "bear")`,
},
},
},
LogStatements: []common.ContextStatements{
{
Context: "log",
Conditions: []string{`attributes["http.path"] == "/animal"`},
Statements: []string{
`set(body, "bear")`,
},
},
},
},
},
{
id: component.NewIDWithName(metadata.Type, "ignore_errors"),
expected: &Config{
Expand Down
Loading

0 comments on commit a4b0e59

Please sign in to comment.