enhancement(vector source): add `headers` option #22695

anil-db · 2025-03-19T18:52:50Z

Summary

This PR adds option to include http headers in vector source log events similar to http source.
most of function and structs are reused from http source.

Use case: we have a use case where between vector sink in edge node and vector source in aggregator, there is a proxy.
this proxy is responsible for validation and adding a header for Provenance purpose for log source.
we need to attach this Provenance header value to log event in aggregator which later will go to ETL pipeline.

GRPC protocol provide these headers as metadata and it seems tonic's Request struct exposes them as metadata but vector source just drops/ignores that field.

This pr is trying to use that metadata to add the header in log event is configure.
by default header config is empty string hence no data is added.

Change Type

Bug fix
New feature
Non-functional (chore, refactoring, docs)
Performance

Is this a breaking change?

Yes
No

How did you test this PR?

tested manually
with below config

{
    "acknowledgements": {
        "enabled": true
    },  
    "api": {
        "address": "0.0.0.0:8686",
        "enabled": true
    },  
    "data_dir": "/vector-data-dir",
    "schema": {
        "log_namespace": false
    },  
    "sinks": {
        "print": {
            "encoding": {
                "codec": "json"
            },  
            "inputs": [
                "agentTier2"
            ],  
            "type": "console"
        }   
    },  
    "sources": {
        "agentTier2": {
            "address": "0.0.0.0:6001",
            "tls": {
                "alpn_protocols": [
                    "h2"
                ],  
                "ca_file": "${VECTOR_TLS_CA_FILE}",
                "crt_file": "${VECTOR_TLS_CRT_FILE}",
                "enabled": true,
                "key_file": "${VECTOR_TLS_KEY_FILE}",
                "verify_certificate": false
            },  
            "headers": ["*"],
            "type": "vector"
        }   
    }   
}

a sample log message with envoy header added as event property.

{"application_timestamp":"2025-03-20T18:28:24.061466172Z","content-type":"application/grpc","grpc-encoding":"gzip","message":{"crc32":null,"instance_id":"logging-agent-4dljt","is_delivered_by_logging_agent":true,"length":2738,"message":"a pproto messafe","metadata":{"container_name":null,"org_id":null,"pod_id":"logging-agent-4dljt","public_host_name":null, "stream_offset":17770549,"upload_time":1742495304062,"workspace_id":null},"message_tag":"MESSAGE_TAG_ARCHIVED_MESSAGE","parse_success":true,"source_type":"vector","te":"trailers","timestamp":"2025-03-20T18:28:24.914968113Z","x-envoy-expected-rq-timeout-ms":"60000","x-forwarded-proto":"https","x-request-id":"e9f83ed8-9eef-4154-ba09-a2e95c4544cb"}

Does this PR include user facing changes?

Yes. Please add a changelog fragment based on our guidelines.
No. A maintainer will apply the "no-changelog" label to this PR.

Checklist

Please read our Vector contributor resources.
- make check-all is a good command to run locally. This check is
  defined here. Some of these
  checks might not be relevant to your PR. For Rust changes, at the very least you should run:
  - cargo fmt --all
  - cargo clippy --workspace --all-targets -- -D warnings
  - cargo nextest run --workspace (alternatively, you can run cargo test --all)
If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
run dd-rust-license-tool write to regenerate the license inventory and commit the changes (if any). More details here.

References

pront

Thanks @anil-db

changelog.d/add_headers_vector_source.enhancement.md

pront · 2025-03-24T20:44:54Z

src/sources/vector/mod.rs

+    ///
+    /// Specifying "*" results in all headers included in the log event.
+    ///
+    /// These headers are not included in the JSON payload if a field with a conflicting name exists.


I wonder if we want to expose this as an option e.g. override_headers

This specification is exactly same as for http_server it should be consistent across both in my option. https://vector.dev/docs/reference/configuration/sources/http_server/#headers

Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com>

pront

This looks reasonable, thanks @anil-db

pront · 2025-03-28T14:28:42Z

Hi @anil-db, this broke one one of nightly checks. Can you take a look please?
https://github.com/vectordotdev/vector/actions/runs/14119249793/job/39556308314

A forward fix should be doable here. Alternatively, I can revert the PR and investigate this later.

This reverts commit 2a85007.

Revert " enhancement(vector source): add `headers` option (#22695)" This reverts commit 2a85007.

anil-db · 2025-03-31T22:47:46Z

Hi @anil-db, this broke one one of nightly checks. Can you take a look please? https://github.com/vectordotdev/vector/actions/runs/14119249793/job/39556308314

A forward fix should be doable here. Alternatively, I can revert the PR and investigate this later.

do you know how to run nightly build locally. I think it is feature dependency issue. I have to add feature dependency in sources-vector to sources-http-server because it is using some code from there.

pront · 2025-04-01T18:53:02Z

Hi @anil-db, this broke one one of nightly checks. Can you take a look please? https://github.com/vectordotdev/vector/actions/runs/14119249793/job/39556308314
A forward fix should be doable here. Alternatively, I can revert the PR and investigate this later.

do you know how to run nightly build locally. I think it is feature dependency issue. I have to add feature dependency in sources-vector to sources-http-server because it is using some code from there.

It's the make check-component-features check that failed. It has to do with the feature gates. I don't have time to dive into this right now but if you reopen a PR with a fix, I will be able to run this check on the CI before merging.

pront · 2025-05-09T14:22:29Z

Hi @anil-db, this broke one one of nightly checks. Can you take a look please? https://github.com/vectordotdev/vector/actions/runs/14119249793/job/39556308314
A forward fix should be doable here. Alternatively, I can revert the PR and investigate this later.

do you know how to run nightly build locally. I think it is feature dependency issue. I have to add feature dependency in sources-vector to sources-http-server because it is using some code from there.

It's the make check-component-features check that failed. It has to do with the feature gates. I don't have time to dive into this right now but if you reopen a PR with a fix, I will be able to run this check on the CI before merging.

Hi @anil-db, let me know if you are interested in reviving this PR and if the above makes sense. Happy to help.

allow options to add header in vector source events

55488a9

anil-db requested review from a team as code owners March 19, 2025 18:52

github-actions bot added domain: sources Anything related to the Vector's sources domain: external docs Anything related to Vector's external, public documentation labels Mar 19, 2025

add change log

29db7ef

anil-db force-pushed the master branch from a714f1f to 29db7ef Compare March 19, 2025 18:54

pront changed the title ~~enhancement(vector source): allow options to add headers in vector source. log event similar to http source~~ enhancement(vector source): add headers option Mar 24, 2025

pront reviewed Mar 24, 2025

View reviewed changes

Update changelog.d/add_headers_vector_source.enhancement.md

d31cdd5

Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com>

anil-db requested a review from pront March 26, 2025 22:41

pront approved these changes Mar 27, 2025

View reviewed changes

pront enabled auto-merge March 27, 2025 19:54

pront added this pull request to the merge queue Mar 27, 2025

Merged via the queue into vectordotdev:master with commit 2a85007 Mar 27, 2025
70 of 71 checks passed

pront changed the title ~~enhancement(vector source): add headers option~~ enhancement(vector source): add headers option Mar 28, 2025

pront added a commit that referenced this pull request Mar 28, 2025

Revert " enhancement(vector source): add headers option (#22695)"

37a3857

This reverts commit 2a85007.

pront mentioned this pull request Mar 28, 2025

chore(vector source): Revert add headers option" #22741

Merged

github-merge-queue bot pushed a commit that referenced this pull request Mar 28, 2025

chore(vector source): Revert add headers option" (#22741)

8bc821f

Revert " enhancement(vector source): add `headers` option (#22695)" This reverts commit 2a85007.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

enhancement(vector source): add `headers` option #22695

enhancement(vector source): add `headers` option #22695

Uh oh!

anil-db commented Mar 19, 2025 •

edited

Loading

Uh oh!

pront left a comment

Uh oh!

Uh oh!

pront Mar 24, 2025

Uh oh!

anil-db Mar 26, 2025

Uh oh!

pront left a comment

Uh oh!

Uh oh!

pront commented Mar 28, 2025

Uh oh!

anil-db commented Mar 31, 2025 •

edited

Loading

Uh oh!

pront commented Apr 1, 2025

Uh oh!

pront commented May 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

enhancement(vector source): add headers option #22695

enhancement(vector source): add headers option #22695

Uh oh!

Conversation

anil-db commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Change Type

Is this a breaking change?

How did you test this PR?

Does this PR include user facing changes?

Checklist

References

Uh oh!

pront left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pront Mar 24, 2025

Choose a reason for hiding this comment

Uh oh!

anil-db Mar 26, 2025

Choose a reason for hiding this comment

Uh oh!

pront left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pront commented Mar 28, 2025

Uh oh!

anil-db commented Mar 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pront commented Apr 1, 2025

Uh oh!

pront commented May 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

enhancement(vector source): add `headers` option #22695

enhancement(vector source): add `headers` option #22695

anil-db commented Mar 19, 2025 •

edited

Loading

anil-db commented Mar 31, 2025 •

edited

Loading