Add VRL script for the ingest API source

[fmassot]

Currently, it's possible to define a transform script for ingested documents for any source but not for the ingest API source.

It would be nice to allow a user to modify the ingest API source with CLI.

[evanxg852000]

I can think of two options:

• update the ingest-api-source using quickwit source ingest-api --add-vrl --path path/to/script.vrl @fmassot pointed out that we don't handle update nicely right now
• quickwit index ingest --transform path/to/script.vrl ... performs the transform on the client side

[evanxg852000]

@fmassot So my point on raising this is that it will look half-backed to blog about supporting other data formats during data ingestion when it's not possible on the ingest API nor on the ingest CLI.

[fulmicoton]

Why don't we put that config at the docmapper level? @fmassot @trinity-1686a @guilload

[guilload]

I like the idea. Having per-source transforms is nice, but at this point, multi-source users are inexistent as far as I know. How do we go about this? Do we want to add doc mapper transforms and

Option 1: Remove source transforms?
The node panics when a source transform is found in an index config. The error message explains how to migrate to doc mapper transforms. Without proper index/source config update support, the migration path is painful for our users.

Option 2: Deprecate source transforms?
The node emits a warning when a source transform is found in an index config. The warning message explains how to migrate to doc mapper transforms. In +2 releases, we remove source transforms. In the meantime, doc mapper transforms take precedence over source transforms.

Option 3: Keep source transforms?
Doc mapper transforms take precedence over source transforms?

Option 4: None of the above. We allow users to add/update/delete a transform for the ingest API source via command-line quickwit source ... until we have proper support for index/source config update support via REST API or UI.

I started writing this comment preferring option 2, but now I'm leaning towards 4.

[fmassot]

I like option 4. I think we plan to have updates on index/source in 0.7. So I'm not even sure if it's worth doing all of that.