feat(http sink): templateable uri

[jorgehermo9]

Closes #1155

Testing

First, spin up a dummy http server that will receive the requests of the http sink, for example, using dummyhttp

dummyhttp -v -p 8081

Then, using the following Vector config:

[sources.in]
type = "http_server"
address = "0.0.0.0:8080"
decoding.codec = "json"

[sinks.out]
type = "http"
inputs = ["in"]
uri = "http://localhost:8081/id/{{id}}"
encoding.codec = "json"
batch.timeout_secs = 5

We listen for incoming requests at 8080 and then forward them to localhost:8081 taking the id field of the events to render the uri.

Then, we send requests with a payload that contains that id field to the http source:

curl -X POST -d '{"id":5,"message":"test2"}' localhost:8080

and we can see requests reaching the dummyhttp server with the path /id/5

If we send multiple curl commands within the same batch time window (5s) we can see the events grouped in the same request (for example, sending 3 events with id=5 and 2 events with id=3)

[jorgehermo9]

Couldn't use run_and_assert_sink_error because there were 2 errors being emitted (ServiceCallError and SinkRequestBuildError), and this assert failed https://github.com/vectordotdev/vector/blob/master/lib/vector-common/src/event_test_util.rs#L31 because the assert just checks that there is just one event name that contains the word "Error"

vector/src/test_util/components.rs

Line 91 in 3b583b1

events: &["Error"],

[pront]

It would be nice to refactor it so we can re-use it here

[jorgehermo9]

Refactored in 83d8fa6, let me know what you think

[pront]

Hi @jorgehermo9, thank you for this PR! I will review it today. Please add a PR description a Vector config you used to test this.

[pront]

It would be nice to refactor it so we can re-use it here

[jorgehermo9]

Hi @pront, thank you very much for your prompt review! I had no time today, but will address everything tomorrow; split the ConcurrementMap fix in another PR, open the two issues I mentioned, specifying a config to test this in the PR description... Than ks!

[jorgehermo9]

Please add a PR description a Vector config you used to test this.

Updated the PR's description with a test config & steps!

[jorgehermo9]

Still have a few comments left, I'll address them through the week, I had busy days lately, sorry! I'll ping when everything is done :)

[pront]

Still have a few comments left, I'll address them through the week, I had busy days lately, sorry! I'll ping when everything is done :)

👍 thanks!

[jorgehermo9]

Hi @pront,

About the request_builder conversation (https://github.com/vectordotdev/vector/pull/23288/files/e5824624cabfeca1e30ef470ce25fea5abeda513#r2174023559)

I was mistaken and the event was being notified with EventStatus::Rejected. Fixed it in 618a8e8. The problem was that I was using Event::with_batch_notifier in the tests intead of Event::add_batch_notifier. I don't quite understand why Event::with_batch_notifier was not working as expected, as I saw it was being used in other places such as

vector/src/test_util/mod.rs

Line 260 in 618a8e8

stream.map(move |log| vec![log.with_batch_notifier_option(&batch)].into())

. Do you have any feedback about that previous usage? Or was it a bug?

And about the partitioned_batched issue, with 618a8e8 now the status is set to EventStatus::Delivered, opened #23366 to track this issue.

I think the only one thing left is the run_and_assert_sink_error refactor in this conversation #23288 (comment).

Sorry for the delay!