v2.1.0
feat: add log_all_statements opt-out to silence verbose default Postgres logging @ivan-pinatti (#92)
## Summary- Add a
log_all_statementsbool variable (defaulttrue, preserves current behavior). - When
true,src/cluster-regional.tfcontinues to prependlog_statement=all
andlog_min_duration_statement=0tocluster_parameters— no change for existing consumers. - When
false, the hardcoded pair is skipped, so consumers can opt out and control both
parameters themselves viacluster_parameters(e.g.log_statement=none,
log_min_duration_statement=1000).
Motivation
The current defaults log every executed statement plus its duration (including
sub-millisecond ones). On an active cluster this routinely produces 20M+ log
lines per day, which can spike CloudWatch Logs and any downstream log-ingest
(Datadog, Splunk, ELK) by tens to hundreds of GB per day. We observed this
first-hand: a freshly deployed Aurora Postgres cluster running Metabase
produced ~22M postgresql log events per day and drove a ~120 GB/day
Datadog indexed-log spike until we patched the module locally.
The values are hardcoded and concat-ed before var.cluster_parameters,
so a user attempting to override them ends up with duplicate parameter names
that AWS rejects at the parameter-group level. In practice there is no
opt-out today without patching the component. This PR adds that opt-out
without changing the default behavior.
How to reproduce the problem this PR fixes
Against a freshly deployed Aurora-Postgres cluster using the current main
(no PR applied):
-
Deploy the component with defaults — no
cluster_parametersoverride.
The hardcoded concat insrc/cluster-regional.tfforces
log_statement=allandlog_min_duration_statement=0onto the cluster
parameter group. Verify:aws rds describe-db-cluster-parameters \ --db-cluster-parameter-group-name <your-pg-name> \ --query "Parameters[?ParameterName=='log_statement' || ParameterName=='log_min_duration_statement']" # expected: log_statement=all, log_min_duration_statement=0
-
Run any real workload against the cluster for 24 hours. Count the log
events shipped to CloudWatch:aws cloudwatch get-metric-statistics \ --namespace AWS/Logs --metric-name IncomingLogEvents \ --dimensions Name=LogGroupName,Value=/aws/rds/cluster/<cluster>/postgresql \ --start-time $(date -u -d '24 hours ago' +%FT%TZ) \ --end-time $(date -u +%FT%TZ) \ --period 86400 --statistic Sum
On our production Metabase cluster this returned ~22 million events/day
— every statement logged, plus a duration line for each. Ingested into
Datadog this drove a ~120 GB/day indexed-log spike. -
Now try to turn it off from the catalog — e.g. append
{ name = "log_statement", value = "none", apply_method = "immediate" }
tovar.cluster_parameters. Apply fails at the AWS parameter-group level
with a duplicate-parameter-name error, because the hardcoded values are
prepended viaconcat()and cannot be overridden. This is the problem:
no opt-out exists today short of patching the component.
Validating the fix in this PR
Same cluster, this PR applied:
-
Leave
log_all_statementsunset — it defaults totrue. The parameter
group showslog_statement=allandlog_min_duration_statement=0exactly
as before. No behavior change for existing consumers. -
Set
log_all_statements = falseand supply your own values:log_all_statements: false cluster_parameters: - { name: log_statement, value: none, apply_method: immediate } - { name: log_min_duration_statement, value: "1000", apply_method: immediate }
aws rds describe-db-cluster-parameters …now shows exactly the user's
values, no duplicates. The parameters areSIGHUP-reloadable so the
change applies live (no reboot). -
After 24 hours, rerun the CloudWatch query from step 2 of reproduction.
In our production cluster the event count dropped from ~22M/day to
~200k/day — a ~99 % reduction — driven entirely by this single
parameter-group change.
Follow-up (out of scope)
Longer term, flipping the default to false in a major version bump would be
the cleanest fix, since log_statement=all + log_min_duration_statement=0
is a debug-mode combination, not a safe production default. Happy to open a
separate PR for that if maintainers agree.
Summary by CodeRabbit
- New Features
- Added new configuration option to control automatic PostgreSQL statement logging for Aurora database clusters. By default, statement logging remains enabled to preserve existing behavior and maintain backward compatibility. Users can now optionally disable this automatic logging configuration if their logging requirements differ from the defaults.