Testing vector config by reading vector tap from stdin

[godber]

I thought it would be handy to tap my log stream and pipe it straight into a local instance of vector configured to read from stdin and send to console. The problem is the stdin source creates a whole new event for each event coming from the tap. I was hoping to find a raw option on the stdin source that just read in existing events over stdin without creating a new event.

Is there another source that does this? Or perhaps I'm not understanding the descriptions of the decoding.codec options, maybe one of those does what I am asking. Is this possible in some other way I've not considered? Basically I want to do:

vector tap foo | vector -C config/ -w

So I can quickly iterate on my transforms.

sources:
  in:
    type: stdin

# transforms:
# ...

sinks:
  out:
    inputs:
      - in
    type: console
    encoding:
      codec: json

[jszwedko]

HI @godber !

Good question. I think to do what you want we'd need to add the native_json codec to vector tap so that you could do:

vector tap --format native_json foo | vector -C config/ -w

With:

sources:
  in:
    type: stdin
    decoding:
      codec: "native_json"

# transforms:
# ...

sinks:
  out:
    inputs:
      - in
    type: console
    encoding:
      codec: native_json

The native_json codec is meant to be a representation of events that Vector can natively decode.

If that makes sense to you, would you want to open a feature request to add a native_json codec to vector tap?

If you just use the stdin source and console sink, and not vector tap, you should be able to use the native_json encoding right now.

[godber]

Thanks for such a quick response! I think I am partly understanding what you're saying. The native_json decoding.code will skip the creation of the new event from the stdin source?

I am not sure I get this:

If you just use the stdin source and console sink, and not vector tap, you should be able to use the native_json encoding right now.

Or more likely I've not given you enough context to explain all of what I am doing. I am taping off of the live vector pipeline running in a pod in kubernetes. The whole point of what I'm trying to achieve is reduce the time it takes to develop transforms. So being able to tap off of arbitrary places in our log processing pipeline is pretty amazing. I don't want to modify the existing pipeline necessarily.

For now we are going to try writing some remap transforms to undo the nested events we're getting from the stdin source. Basically converting it back to the kafka source type event we'd actually be seeing in the middle of the pipeline.

[godber]

For now, we've decided to just stick with the following transform, which puts the record back to the way it was before the stdin source creates a NEW event from it:

transforms:
  fix_logs:
    type: remap
    inputs: [in]
    source: |-
      del(.host)
      del(.source_type)
      del(.timestamp)
      . = parse_json!(.message)