[ottl] New function `Fail` to abort a list of statements conditionally

[ChristianCiach]

Component(s)

pkg/ottl

Is your feature request related to a problem? Please describe.

I am new to Otelcol, so maybe I am doing something wrong. But I quite often encounter a situation where it doesn't make sense to continue the rest of the ottl statements and I would rather propagate an error.

Describe the solution you'd like

I am thinking about something like that:

      transform/logs-parse-json:
        log_statements:
          - error_mode: propagate
            statements:
              - merge_maps(log.cache, ParseJSON(log.body), "upsert")
              - set(log.severity_text, log.cache["level"])
              - fail("Missing log level") where log.severity_text == ""

Describe alternatives you've considered

No response

Additional context

No response

[github-actions]

Pinging code owners:

• pkg/ottl: @TylerHelmuth @evan-bradley @edmocosta

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[TylerHelmuth]

This is a pretty cool idea. Today the way to handle it is some conditions, but a way to break out sooner would be cool.

[TylerHelmuth]

This current API would need to rely on fail return an error. Depending on the error_mode that could be annoying, but thats probably the best way to do it in a non-breaking way.

[ChristianCiach]

Good point. This would only really make sense with error_mode: propagate. With ignore it would just log the error and continue with the rest of the statements block. But I would be okay with that. I had situations in mind where I want to propagate an unexpected situation.

But you are right in that won't give us an option to exit a block of statements early without propagating an error to the logs. You either propagate the error (and make it visible to the logs) or you are forced to run the rest of the statements (silently or not). I guess there are ways to model around this, for example by using a silent block to, for example, parse some json (which can fail), set something like log.cache["jsonparsed"] = true and then make the next block of statements conditional on that attribute.

Still, I think manually failing a block of statements based on a boolean expressions is something that I would like to have.

I didn't really think this through before creating this issue...

Edit: Maybe a new error_mode like abort_silently would be nice, but this would be a separate issue for the transform-processor.

[ChristianCiach]

We could probably add a special type of error that aborts the current sequence of statements without it being logged. Something like:

      transform/logs-parse-json:
        log_statements:
          - error_mode: propagate
            statements:
              - merge_maps(log.cache, ParseJSON(log.body), "upsert")
              - set(log.severity_text, log.cache["level"])
              - break() where log.severity_text != ""
              - set(log.severity_text, log.cache["log.level"]
              - break() where log.severity_text != ""

It happens to me all the time that I know that there is no sense in continuing the rest of the statements because I already found the attribute I was looking for and there is no need burning CPU cycles by continuing the rest of the regexps.

This could also be done by adding the error_mode: abort_silently mentioned above if/when we get fail(), but then I would have to split my statements into a lot of log_statements-elements, making it hard to read.

Of course, I could also do it like this:

      transform/logs-parse-json:
        log_statements:
          - statements:
              - merge_maps(log.cache, ParseJSON(log.body), "upsert")
              - set(log.severity_text, log.cache["level"])
              - set(log.severity_text, log.cache["log.level"]) where log.severity_text == ""
              - set(log.severity_text, log.cache["lvl"]) where log.severity_text == ""

But the where conditions can get out of hand quickly the longer the sequence of statements becomes. I would rather have a way to signal "we are done here!" to the interpreter.

Most other log-collectors also have poor support to express conditions like these. This always makes me think that I must be doing something wrong. Vector seems to be the only tool of its kind that cleanly supports these best-effort parsing situations, due to its VRL syntax.

[edmocosta]

I think the fail editor is nice idea, but make something like the break or any other flow control to work seamless with the transform process and OTTL statements sequence would be a more complex issue, and it definitely would need a broadly discussion.

If your current use-case is selecting the first non empty value, I'm wondering if some FirstNotEmpty([...]) converter would do the work for now, so you don't need to wait for this proposal to go through.

transform/logs-parse-json:
  log_statements:
    statements:
      - merge_maps(log.cache, ParseJSON(log.body), "upsert")
      - set(log.severity_text, FirstNotEmpty([log.cache["level"], log.cache["log.level"], log.cache["lvl"]])

[ChristianCiach]

If your current use-case is selecting the first non empty value, I'm wondering if some FirstNotEmpty([...]) converter would do the work for now

Yes, this would reduce a lot of noise from my configurations. There are still some places where I need to match the same attribute against multiple regexps (where I only expect one to actually match), but the same idea could be applied in these cases.