[ansi_stripper] stripper doesn't strip color?

[KannarFr]

Hi,

ansi_stripper transform doesn't remove ansi color from field?

For the following test:

[transforms.ansi_stripper_simple_yo]
  inputs = []
  type = "ansi_stripper"
  field = "message"
[[tests]]
  name = "ansi_stripper_simple_yo"
  [tests.input]
    insert_at = "ansi_stripper_simple_yo"
    type = "raw"
    value = "\t\r\\033[31mHelloWorld"
  [[tests.outputs]]
    extract_from = "ansi_stripper_simple_yo"
    [[tests.outputs.conditions]]
      "message.equals" = "HelloWorld"

It outputs:

check transform 'ansi_stripper_simple_yo' failed conditions:
  condition[0]: predicates failed: [ message.equals: "HelloWorld" ]
payloads (events encoded as JSON):
  input: {"message":"\t\r\\033[31mHelloWorld","timestamp":"2020-04-02T03:52:42.774856684Z"}
  output: {"message":"\\033[31mHelloWorld","timestamp":"2020-04-02T03:52:42.774856684Z"}

Maybe it's a wanted behavior and we should write ansi_color_stripper? WDYT?

[binarylogic]

Hi @KannarFr thanks for reporting. Yes, this should probably stip color as well. We'll get this fixed.

[binarylogic]

@JeanMertz given that you built this transform could you take a look? It should be relatively simple, but if this test case is correct this transform is not really usable. Thanks!

[KannarFr]

@JeanMertz I think there is an issue on the parsing:

value = "\x1b[32mfoo\x1b[m bar"

throws:

Apr 07 15:07:22.938 ERROR vector::unit_test: Failed to execute tests/behavior/transforms/ansi_stripper.toml tests:
invalid escape character in string: `x` at line 25

But the strip_ansi_escapes creates give this example using bytes:

fn work() -> io::Result<()> {
  let bytes_with_colors = b"\x1b[32mfoo\x1b[m bar";
  let plain_bytes = strip_ansi_escapes::strip(&bytes_with_colors)?;
  io::stdout().write_all(&plain_bytes)?;
  Ok(())
}

[JeanMertz]

Thanks for the reproducible example @KannarFr. I'll take a look at this sometime tomorrow.

[KannarFr]

@JeanMertz (just a ping :p) maybe i can help you?

[JeanMertz]

@KannarFr apologies. I haven't found the time yet, but it's still on my list to do.

What I plan on doing is to check if there's something in the test set-up for this component that differs from your test-case:

https://github.com/timberio/vector/blob/5f78ebb7c3e21f878859b72dff6450c45c7359dc/src/transforms/ansi_stripper.rs#L109-L129

If so, I want to capture your example in a test, and see what's going on.

[KannarFr]

I think here is the difference: the test you quoted here doesn't work if you pass its values to a vector test using this transform. As you said.

I think it's a parsing issue, bytes vs String or something like that.

Waiting for your feedback.

[JeanMertz]

Hey @KannarFr,

I discussed this with @LucioFranco. The problem is that the input string uses the octal escape character \033, which isn't supported by Rust natively. For this to be a valid escape character, we'll need to rewrite it.

Basically something similar to what happens in the rust-cexpr crate, here.

Luckily, we're already using the Nom crate in Vector, so we can mostly duplicate that piece of code just for this specific use-case of handling the octal escape character.

I won't get to it this week, but it's scheduled for next.

[KannarFr]

Ok cool, thanks :).

[KannarFr]

Can I help you @JeanMertz?