How to sink original data through sink type http

[ngocchien]

My configuration for vector :

[sources.dummy_logs]
type = "generator" # required
lines = ["{\"messages\":[{\"key\":\"my-key\",\"payload\":\"data at here\",\"eventTime\":1603045262772,\"sequenceId\":2}]}"]
sequence = false # optional, default, relevant when [`format`](#format)
batch_interval = 3
format = "shuffle"

[transforms.remap_data]
type = "remap"
inputs = ["dummy_logs"]
source = """
. = parse_json!(.message) \n"""

# Print parsed logs to stdout
[sinks.http]
type = "http" 
inputs = ["remap_data"]
encoding.codec = "json"
uri = "http://192.168.197.135:8080"


[sinks.console]
type = "console"
inputs = ["remap_data"]
encoding.codec = "json"
#encoding.only_fields = ["timestamp", "thread_id", "query_id", "severity", "message" ]

Original data and print out console :

{
    "messages": [
        {
            "eventTime": 1603045262772,
            "key": "my-key",
            "payload": "data at here",
            "sequenceId": 2
        }
    ]
}

But my data received on my API :

[{"messages":[{"eventTime":1603045262772,"key":"my-key","payload":"data at here","sequenceId":2}]}]

Through encoding: JSON (sink type http), Vector changes object JSON to array JSON.
How to keep original data to send to my API with HTTP Content-Type: application/json

Thanks all.
Chien Nguyen.

[jszwedko]

HI @ngocchien !

I think you want to configure the framing on the http sink to be newline-delimited:

[sinks.http]
type = "http" 
inputs = ["remap_data"]
encoding.codec = "json"
framing.method = "newline_delimited"
uri = "http://192.168.197.135:8080"

By default the http sink will frame JSON-encoded messages as JSON. This differs from the console sink which is stream-based where the http sink is batch-based. Admittedly this nuance is a bit confusing.

[jszwedko]

Also if you want to limit Vector to sending one event at a time you can use batch.max_events = 1 in the http sink.

[ngocchien]

HI @ngocchien !

I think you want to configure the framing on the http sink to be newline-delimited:

[sinks.http]
type = "http" 
inputs = ["remap_data"]
encoding.codec = "json"
framing.method = "newline_delimited"
uri = "http://192.168.197.135:8080"

By default the http sink will frame JSON-encoded messages as JSON. This differs from the console sink which is stream-based where the http sink is batch-based. Admittedly this nuance is a bit confusing.

Thanks, @jszwedko.
Your solution can not resolve my problem.
Your configuration will change HTTP request Content-Type to : application/x-ndjson, this Content-Type is incorrect with my service(pulsar service). I can not use sink type pulsar, because it does not support TLS.

[ngocchien]

And I have tried changing request.headers.Content-Type = "application/json", this is append more 1 Content-Type in headers (not override).
This is the data I got on the pulsar service.

[jszwedko]

Gotcha, yeah, we do have an open issue to make it possible to override the Content-Type header: #9768 . I think you may need for that to be resolved first.

[ngocchien]

Thank you so much. @jszwedko