Fluent Bit Plugin for Amazon Kinesis Data Streams
A Fluent Bit output plugin for Amazon Kinesis Data Streams.
region: The region which your Kinesis Data Stream is in.
stream: The name of the Kinesis Data Stream that you want log records sent to.
partition_key: A partition key is used to group data by shard within a stream. A Kinesis Data Stream uses the partition key that is associated with each data record to determine which shard a given data record belongs to. For example, if your logs come from Docker containers, you can use container_id as the partition key, and the logs will be grouped and stored on different shards depending upon the id of the container they were generated from. As the data within a shard are coarsely ordered, you will get all your logs from one container in one shard roughly in order. If you don't set a partition key or put an invalid one, a random key will be generated, and the logs will be directed to random shards. If the partition key is invalid, the plugin will print an warning message.
data_keys: By default, the whole log record will be sent to Kinesis. If you specify key name(s) with this option, then only those keys and values will be sent to Kinesis. For example, if you are using the Fluentd Docker log driver, you can specify
data_keys logand only the log message will be sent to Kinesis. If you specify multiple keys, they should be comma delimited.
role_arn: ARN of an IAM role to assume (for cross account access).
endpoint: Specify a custom endpoint for the Kinesis Streams API.
append_newline: If you set append_newline as true, a newline will be addded after each log record.
The plugin requires
This plugin uses the AWS SDK for Go, and uses its default credential provider chain. If you are using the plugin on Amazon EC2 or Amazon ECS, the plugin will use your EC2 instance role or ECS Task role permissions. The plugin can also retrieve credentials from a (shared credentials file)[https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html], or from the standard
AWS_SESSION_TOKEN environment variables.
FLB_LOG_LEVEL: Set the log level for the plugin. Valid values are:
error(case insensitive). Default is
info. Note: Setting log level in the Fluent Bit Configuration file using the Service key will not affect the plugin log level (because the plugin is external).
SEND_FAILURE_TIMEOUT: Allows you to configure a timeout if the plugin can not send logs to Kinesis Streams. The timeout is specified as a Golang duration, for example:
5m30s. If the plugin has failed to make any progress for the given period of time, then it will exit and kill Fluent Bit. This is useful in scenarios where you want your logging solution to fail fast if it has been misconfigured (i.e. network or credentials have not been set up to allow it to send to Kinesis Streams).
Fluent Bit Versions
This plugin has been tested with Fluent Bit 1.2.0+. It may not work with older Fluent Bit versions. We recommend using the latest version of Fluent Bit as it will contain the newest features and bug fixes.
Example Fluent Bit Config File
[INPUT] Name forward Listen 0.0.0.0 Port 24224 [OUTPUT] Name kinesis Match * region us-west-2 stream my-kinesis-stream-name partition_key container_id append_newline true
AWS for Fluent Bit
We distribute a container image with Fluent Bit and this plugin.
You can use our SSM Public Parameters to find the Amazon ECR image URI in your region:
aws ssm get-parameters-by-path --path /aws/service/aws-for-fluent-bit/
For more see our docs.