CloudWatch Logs output throws a SerializationException error

[txigreman]

Bug Report

CloudWatch Logs output throws a SerializationException error when using log_key option pointing to a field that stores an object.
It works if the field contains a string.

To Reproduce

• Steps to reproduce the problem:
  Run this command with proper AWS credentials:

docker run --rm -e "AWS_ACCESS_KEY_ID=xxx" -e "AWS_SECRET_ACCESS_KEY=xxx" fluent/fluent-bit:2.2.2 \
  -i 'mem' \
  -F 'nest' -p 'Operation=nest' -p 'Wildcard=Mem.*' -p 'Nest_under=mem' -p 'Remove_prefix=Mem.' -m '*' \
  -o 'stdout' -m '*' \
  -o 'cloudwatch_logs' -p 'region=eu-west-1' -p 'log_key=mem' -p 'log_group_name=/test' -p 'log_stream_name=test' -p 'auto_create_group=true' -m '*'

• Log output:

[date] [ info] [fluent bit] version=2.2.2, commit=eeea396e88, pid=1
[date] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[date] [ info] [cmetrics] version=0.6.6
[date] [ info] [ctraces ] version=0.4.0
[date] [ info] [input:mem:mem.0] initializing
[date] [ info] [input:mem:mem.0] storage_strategy='memory' (memory only)
[date] [ info] [output:stdout:stdout.0] worker #0 started
[date] [ info] [sp] stream processor started
[date] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] worker #0 started
[date] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Creating log stream test in log group /test
[0] mem.0: [[1706864948.685588521, {}], {"Swap.total"=>4194304, "Swap.used"=>0, "Swap.free"=>4194304, "mem"=>{"total"=>16331076, "used"=>4951620, "free"=>11379456}}]
[date] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Log Stream test already exists
[date] [error] [output:cloudwatch_logs:cloudwatch_logs.1] PutLogEvents API responded with error='SerializationException'
[date] [error] [output:cloudwatch_logs:cloudwatch_logs.1] Failed to send log events
[date] [error] [output:cloudwatch_logs:cloudwatch_logs.1] Failed to send log events
[date] [error] [output:cloudwatch_logs:cloudwatch_logs.1] Failed to send events
[date] [ warn] [engine] failed to flush chunk '1-1706864948.685606204.flb', retry in 11 seconds: task_id=0, input=mem.0 > output=cloudwatch_logs.1 (out_id=1)

Expected behavior
The field mem should be encoded in json, and stored in cloudwatch logs.

Your Environment

• Version used: 2.2.2
• Docker image: fluent/fluent-bit:2.2.2

[thamjieying]

I am facing the same issue to, but previously when i did not make modification to "log" key it was working fine.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[txigreman]

Still reproducible in version 3.0.3:

docker run --rm -e "AWS_ACCESS_KEY_ID=xxx" -e "AWS_SECRET_ACCESS_KEY=xxx" fluent/fluent-bit:3.0.3 \
  -i 'mem' \
  -F 'nest' -p 'Operation=nest' -p 'Wildcard=Mem.*' -p 'Nest_under=mem' -p 'Remove_prefix=Mem.' -m '*' \
  -o 'stdout' -m '*' \
  -o 'cloudwatch_logs' -p 'region=eu-west-1' -p 'log_key=mem' -p 'log_group_name=/test' -p 'log_stream_name=test' -p 'auto_create_group=true' -m '*'

Note: configure proper AWS credentials.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.