Skip to content

v2.1.0

Choose a tag to compare

@cloudposse-releaser cloudposse-releaser released this 28 May 19:44
9135a84
feat: add log_all_statements opt-out to silence verbose default Postgres logging @ivan-pinatti (#92) ## Summary
  • Add a log_all_statements bool variable (default true, preserves current behavior).
  • When true, src/cluster-regional.tf continues to prepend log_statement=all
    and log_min_duration_statement=0 to cluster_parameters — no change for existing consumers.
  • When false, the hardcoded pair is skipped, so consumers can opt out and control both
    parameters themselves via cluster_parameters (e.g. log_statement=none,
    log_min_duration_statement=1000).

Motivation

The current defaults log every executed statement plus its duration (including
sub-millisecond ones). On an active cluster this routinely produces 20M+ log
lines per day, which can spike CloudWatch Logs and any downstream log-ingest
(Datadog, Splunk, ELK) by tens to hundreds of GB per day. We observed this
first-hand: a freshly deployed Aurora Postgres cluster running Metabase
produced ~22M postgresql log events per day and drove a ~120 GB/day
Datadog indexed-log spike until we patched the module locally.

The values are hardcoded and concat-ed before var.cluster_parameters,
so a user attempting to override them ends up with duplicate parameter names
that AWS rejects at the parameter-group level. In practice there is no
opt-out today without patching the component. This PR adds that opt-out
without changing the default behavior.

How to reproduce the problem this PR fixes

Against a freshly deployed Aurora-Postgres cluster using the current main
(no PR applied):

  1. Deploy the component with defaults — no cluster_parameters override.
    The hardcoded concat in src/cluster-regional.tf forces
    log_statement=all and log_min_duration_statement=0 onto the cluster
    parameter group. Verify:

    aws rds describe-db-cluster-parameters \
      --db-cluster-parameter-group-name <your-pg-name> \
      --query "Parameters[?ParameterName=='log_statement' || ParameterName=='log_min_duration_statement']"
    # expected: log_statement=all, log_min_duration_statement=0
  2. Run any real workload against the cluster for 24 hours. Count the log
    events shipped to CloudWatch:

    aws cloudwatch get-metric-statistics \
      --namespace AWS/Logs --metric-name IncomingLogEvents \
      --dimensions Name=LogGroupName,Value=/aws/rds/cluster/<cluster>/postgresql \
      --start-time $(date -u -d '24 hours ago' +%FT%TZ) \
      --end-time $(date -u +%FT%TZ) \
      --period 86400 --statistic Sum

    On our production Metabase cluster this returned ~22 million events/day
    — every statement logged, plus a duration line for each. Ingested into
    Datadog this drove a ~120 GB/day indexed-log spike.

  3. Now try to turn it off from the catalog — e.g. append
    { name = "log_statement", value = "none", apply_method = "immediate" }
    to var.cluster_parameters. Apply fails at the AWS parameter-group level
    with a duplicate-parameter-name error, because the hardcoded values are
    prepended via concat() and cannot be overridden. This is the problem:
    no opt-out exists today short of patching the component.

Validating the fix in this PR

Same cluster, this PR applied:

  1. Leave log_all_statements unset — it defaults to true. The parameter
    group shows log_statement=all and log_min_duration_statement=0 exactly
    as before. No behavior change for existing consumers.

  2. Set log_all_statements = false and supply your own values:

    log_all_statements: false
    cluster_parameters:
      - { name: log_statement, value: none, apply_method: immediate }
      - { name: log_min_duration_statement, value: "1000", apply_method: immediate }

    aws rds describe-db-cluster-parameters … now shows exactly the user's
    values, no duplicates. The parameters are SIGHUP-reloadable so the
    change applies live (no reboot).

  3. After 24 hours, rerun the CloudWatch query from step 2 of reproduction.
    In our production cluster the event count dropped from ~22M/day to
    ~200k/day — a ~99 % reduction — driven entirely by this single
    parameter-group change.

Follow-up (out of scope)

Longer term, flipping the default to false in a major version bump would be
the cleanest fix, since log_statement=all + log_min_duration_statement=0
is a debug-mode combination, not a safe production default. Happy to open a
separate PR for that if maintainers agree.

Summary by CodeRabbit

  • New Features
    • Added new configuration option to control automatic PostgreSQL statement logging for Aurora database clusters. By default, statement logging remains enabled to preserve existing behavior and maintain backward compatibility. Users can now optionally disable this automatic logging configuration if their logging requirements differ from the defaults.

Review Change Stack