[Q&A] CloudWatch Logs cost increased 5x after upgrading Fluentd Kubernetes DaemonSet image from v1.15.3 to v1.18.0

[Govind10g]

What is a problem?

Hi everyone,

We're seeing a significant increase in our Amazon CloudWatch Logs cost after upgrading our Fluentd DaemonSet image and are trying to determine the root cause.

Environment
Kubernetes
Fluentd DaemonSet
Old Image: fluent/fluentd-kubernetes-daemonset:v1.15.3-debian-cloudwatch-1.0
New Image: fluent/fluentd-kubernetes-daemonset:v1.18.0-debian-cloudwatch-1.5

What changed?

The only intentional changes were:

Upgraded the Fluentd Docker image from v1.15.3-debian-cloudwatch-1.0 to v1.18.0-debian-cloudwatch-1.5.
Increased the CloudWatch Logs retention period from 7 days to 30 days.

No changes were made to:

Application logging
Fluentd ConfigMap
Helm values
DaemonSet configuration
CloudWatch log group configuration (other than retention)

Observations
Before the upgrade, CloudWatch Logs cost was approximately USD 800–900/day.
After the upgrade, the cost increased to approximately USD 4,500–5,000/day.
The increase is not due to storage; Cost Explorer shows the increase is related to CloudWatch Logs ingestion / PutLogEvents.
We have verified that the application is not generating duplicate logs.
There are no duplicate Fluentd pods or duplicate DaemonSets.

What we've investigated
Retention should only affect storage costs, not ingestion.

We suspect the issue may be related to changes in:
fluent-plugin-cloudwatch-logs
fluent-plugin-kubernetes_metadata_filter
Buffering/batching behavior
Record serialization or metadata enrichment

Questions
Has anyone experienced a similar increase after upgrading from the 1.15.x CloudWatch image to the 1.18.x CloudWatch image?
Were there any known changes in the bundled CloudWatch plugin that affect batching or PutLogEvents behavior?
Did the Kubernetes metadata filter start including significantly more metadata by default?
Are there any changes in the default buffer configuration or output plugin behavior between these image versions?
Is there a way to compare the exact plugin versions and generated Fluentd configuration between these two images?

Any guidance, known issues, GitHub discussions, or troubleshooting suggestions would be greatly appreciated.

Describe the configuration of Fluentd

No response

Describe the logs of Fluentd

No response

Environment

- Fluentd version:
- TD Agent version:
- Fluent Package version:
- Docker image (tag):
- Operating system:
- Kernel version:

[Watson1978]

Thanks for the detailed write-up. I went through the version delta between the two images and, as a first note, I don't see any change in this specific jump that would by itself multiply CloudWatch Logs ingestion ~5x. Here is what I checked.

So the increase is most likely driven by event count or duplicate ingestion rather than the upgrade itself. A quick reminder on the cost model: CloudWatch Logs ingestion is billed on message bytes + ~26 bytes of per-event overhead, so both the number of events and the bytes per event matter. One important default worth knowing: when message_keys is not set, this plugin serializes the entire record with JSON.generate(record), so every event carries the full Kubernetes metadata block. If events get split (e.g. multi-line stack traces emitted per line), each split line re-carries that whole metadata block, which inflates bytes quickly.

To pin down the cause, could you check the following?

1. Compare the IncomingBytes and IncomingLogEvents CloudWatch metrics for the affected log group(s) before vs after. This immediately tells us whether it is "more events" (per-event overhead / splitting / duplication) or "bigger events" (metadata bloat).
2. Pick one application log line and confirm whether it appears more than once in CloudWatch. The claim that the app is not producing duplicates does not rule out duplicate ingestion from tail re-reads or retries.
3. Open one stored event and check whether the body is just the message or the full JSON record including all kubernetes.* metadata. If it is the full record, setting message_keys (for example message_keys ["log"]) to send only the message field is usually the single largest reduction, and it is independent of the version question.
4. Dump the effective merged config inside the pod (the ConfigMap @includes the image-provided conf, so the effective config can differ across image versions even if your ConfigMap did not change), and compare old vs new.

If you can share your Fluentd config (ConfigMap), a sample stored event, and the relevant FLUENT_* env vars, I can give a much more concrete answer.