aws_kinesis_firehose source: include attributes from X-Amz-Firehose-Common-Attributes header in the log event

[tchanturia]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

When configuring Amazon Data Firehose in AWS it is possible to add additional parameters, which AWS includes in each request as a JSON object in X-Amz-Firehose-Common-Attributes header (see HTTP Headers - X-Amz-Firehose-Common-Attributes in https://docs.aws.amazon.com/firehose/latest/dev/httpdeliveryrequestresponse.html).
Example parameters:

These key-value pairs can then be used to provide additional metadata, that is not available as part of CloudWatch log events, and used as a source to provide additional tags (for example when outputting logs to Datadog) or perform additional transformations.

Attempted Solutions

1. Configure aws_kinesis_firehose source and output incoming requests into console:

sources:
  kinesis_firehose:
    type: aws_kinesis_firehose
    address: 0.0.0.0:3000
    access_keys:
      - SECRET[vector_secrets.firehose_access_key]
    store_access_key: false

sinks:
  console:
    inputs: [kinesis_firehose]
    type: console
    encoding:
      codec: json

2. Create a new Amazon Data Firehose stream with Direct PUT and the vector aws kinesis firehose endpoint as the destination
3. Add parameters to the Amazon Data Firehose configuration

Key	Value
appgroup	tymur_test
aws_account	1234567890
environment	testing
team	platform

4. Create a CloudWatch subscription filter on a CloudWatch log group with the created Amazon Data Firehose as destination
5. Send a test log to the CloudWatch log group

Expected:
The configured Amazon Data Firehose parameters are present in the vector log event

Actual:
The configured Amazon Data Firehose parameters are absent in the vector log event

Example log event (current version):

{
    "message": "{\"messageType\":\"DATA_MESSAGE\",\"owner\":\"1234567890\",\"logGroup\":\"/aws/lambda/VectorSandboxFunction\",\"logStream\":\"2026/03/12/VectorSandboxFunction[$LATEST]9263c13cf4394836afaf10e2ae8a9bf7\",\"subscriptionFilters\":[\"tooling-vector-subscription\"],\"logEvents\":[{\"id\":\"39546388019874692505746695330530399880790906995599671298\",\"timestamp\":1773321369659,\"message\":\"this is unstructured log\\n\"}]}",
    "request_id": "91b20a85-8bab-4dd0-b70d-2973f7f2f0b5",
    "source_arn": "arn:aws:firehose:eu-west-1:1234567890:deliverystream/vector-sandbox-logs-to-vector-logs",
    "source_type": "aws_kinesis_firehose",
    "timestamp": "2026-03-12T13:16:27.300Z"
}

Proposal

1. Add new common_attributes config to aws_kinesis_firehose source that would allow to specify which common attributes to include in the log event. Similar to headers in the HTTP server source: https://vector.dev/docs/reference/configuration/sources/http_server/#headers
   If none configured, none would be included. This will make sure backwards compatibility and won't break equivalency (unit test) assertions in the existing vector configurations.
2. Parse JSON structure from X-Amz-Firehose-Common-Attributes header in aws_kinesis_firehouse source
3. Based on the common_attributes config, include matching common attributes as an object in the metadata of the log event in common_attributes property. By inserting these common attributes into common_attributes property, it would allow for easily merging these key value pairs into log even tags, for example. I know headers and query parameters in the HTTP server are inserted in the root of the JSON object instead, so, please, let me know what you think.

Example configuration with new common_attributes property:

sources:
  kinesis_firehose:
    type: aws_kinesis_firehose
    address: 0.0.0.0:3000
    access_keys:
      - SECRET[vector_secrets.firehose_access_key]
    store_access_key: false
    common_attributes:
      - "*"

sinks:
  console:
    inputs: [kinesis_firehose]
    type: console
    encoding:
      codec: json

Example log event (proposal):

{
    "common_attributes": {
        "appgroup": "tymur_testing",
        "aws_account": "1234567890",
        "environment": "testing",
        "team": "platform"
    },
    "message": "{\"messageType\":\"DATA_MESSAGE\",\"owner\":\"1234567890\",\"logGroup\":\"/aws/lambda/VectorSandboxFunction\",\"logStream\":\"2026/03/12/VectorSandboxFunction[$LATEST]ed4b6ced439c4a6aad416c7292d73b97\",\"subscriptionFilters\":[\"tooling-vector-subscription\"],\"logEvents\":[{\"id\":\"39546641090849776225118528402209126727950828071435239426\",\"timestamp\":1773332717754,\"message\":\"this is unstructured log\\n\"}]}",
    "request_id": "beef8396-42d2-4d89-9eae-15fd3f1bb3ec",
    "source_arn": "arn:aws:firehose:eu-west-1:1234567890:deliverystream/vector-sandbox-logs-to-vector-logs",
    "source_type": "aws_kinesis_firehose",
    "timestamp": "2026-03-12T16:25:42.540Z"
}

References

No response

Version

0.54.0