v0.41.2

• Add two additional log_line_prefix settings
  • %p-%s-%c-%l-%h-%u-%d-%m
  • %m [%p][%b][%v][%x] %q[user=%u,db=%d,app=%a]
• Change schema collection message to warning during test run
  • This helps discover schema collection issues, e.g. due
    to connection restrictions or other permission problems
• Fix issue with multiple domain constraints
• Upgrade gopsutil to v3.21.10
  • This adds support for M1 Macs, amongst other improvements
    for OS metris collection

v0.41.1

• Fix schema stats for databases with some custom data types
• Fix tracking of index scans over time

v0.41.0

• Add support for custom data types
• Track column stats for improved Index Advisor recommendations
• Vacuum activity: Correctly handle duplicate tables
• Citus: fix broken relation stats query
• Retry API requests in case of temporary network issues
• Update to Go 1.17
• Update to pg_query_go v2.1.0
  • Improve normalization of GROUP BY clauses

v0.40.0

• Update to pg_query_go v2.0.4
  • Normalize: Don't touch "GROUP BY 1" and "ORDER BY 1" expressions, keep original text
  • Fingerprint: Cache list item hashes to fingerprint complex queries faster
    (this change also significantly reduces memory usage for complex queries)
• Install script: Support CentOS in addition to RHEL

v0.39.0

• Docker: Use Docker's USER command to set user, to support running as non-root
  • This enables the collector container to run in environments that require the
    whole container to run as a non-root user, which previously was not the case.
  • For compatibility reasons the container can still be run as root explicitly,
    in which case the setpriv command is used to drop privileges. setpriv replaces
    gosu since its available for installation in most distributions directly, and
    fulfills the same purpose here.
• Selfhosted: Support running log discovery with non-localhost db_host settings
  • Previously this was prevented by a fixed check against localhost/127.0.0.1,
    but sometimes one wants to refer to the local server by a non-local IP address
• AWS: Add support for AssumeRoleWithWebIdentity
  • This is useful when running the collector inside EKS in order to access
    AWS resources, as recommended by AWS: https://docs.aws.amazon.com/eks/latest/userguide/specify-service-account-role.html
• Statement stats retrieval: Get all rows first, before fingerprinting queries
  • This avoids showing a bogus ClientWrite event on the Postgres server side whilst
    the collector is running the fingerprint method. There is a trade-off here,
    because we now need to retrieve all statement texts (for the full snapshot) before
    doing the fingerprint, leading to a slight increase in memory usage. Nonetheless,
    this improves debuggability, and avoids bogus statement timeout issues.
• Track additional meta information about guided setup failures
• Fix reporting of replication statistics for more than 1 follower

v0.38.1

• Update to pg_query_go 2.0.2
  • Normalize: Fix handling of two subsequent DefElems (resolves rare crashes)
• Redact primary_conninfo setting if present and readable
  • This can contain sensitive information (full connection string to the
    primary), and pganalyze does not do anything with it right now. In the
    future, we may partially redact this and use primary hostname
    information, but for now, just fully redact it.

v0.38.0

• Update to pg_query 2.0 and Postgres 13 parser
  • This is a major upgrade in terms of supported syntax (Postgres 10 to 13),
    as well as a major change in the fingerprints, which are now shorter and
    not compatible with the old format.
  • When you upgrade to this version of the collector you will see a break
    in statistics, that is, you will see new query entries in pganalyze after
    adopting this version of the collector.
• Amazon RDS: Support long log events beyond 2,000 lines
  • Resolves edge cases where very long EXPLAIN plans would be ignored since
    they exceeded the previous 2,000 limit
  • We now ensure that we go back up to 10 MB in the file with each log
    download that happens, with support for log events that exceed the RDS API
    page size limit of 10,000 log lines
• Self-managed: Also check for the process name "postmaster" when looking for
  Postgres PID (fixes data directory detection for RHEL-based systems)

v0.37.1

• Docker builds: Increase stack size to 2MB to prevent rare crashes
  • Alpine has a very small stack size by default (80kb) which is less than
    the default that Postgres expects (100kb). Since there is no good reason
    to reduce it to such a small amount, increase to usually common Linux
    default of 2MB stack size.
  • This would have surfaced as a hard crash of the Docker container with
    error code 137 or 139, easily confused with out of memory errors, but
    clearly distinct from it.
• Reduce timeout for accessing EC2 instance metadata service
  • Previously we were re-using our shared HTTP client, which has a rather
    high timeout (120 seconds) that causes the HTTP client to wait around
    for a long time. This is generally intentional (since it includes the
    time spent downloading a request body), but is a bad idea when running
    into EC2's IDMSv2 service that has a network-hop based limit. If that
    hop limit is exceeded, the requests just go to nowhere, causing the
    client to wait for a multiple of 120 seconds (~10 minutes were observed).
• Don't use pganalyze query marker for "--test-explain" command
  • The marker means the resulting query gets hidden from the EXPLAIN plan
    list, which is what we don't want for this test query - it's intentional
    that we can see the EXPLAIN plan we're generating for the test.

v0.37.0

• Add support for receiving logs from remote servers over syslog
  • You can now specify the new "db_log_syslog_server" config setting, or
    "LOG_SYSLOG_SERVER" environment variable in order to setup the collector
    as a syslog server that can receive logs from a remote server via syslog
    to the server that runs the collector.
  • Note that the format of this setting is "listen_address:port", and its
    recommended to use a high port number to avoid running the collector as root.
  • For example, you can specify "0.0.0.0:32514" and then send syslog messages
    to the collector's server address at port 32514.
  • Note that you need to use protocol RFC5424, with an unencrypted TCP
    connection. Due to syslog not being an authenticated protocol it is
    recommended to only use this integration over private networks.
• Add support for "pid=%p,user=%u,db=%d,app=%a,client=%h " and
  "user=%u,db=%d,app=%a,client=%h " log_line_prefix settings
  • This prefix misses a timestamp, but is useful when sending data over syslog.
• Log parsing: Correctly handle %a containing commas/square brackets
  • Note that this does not support all cases since Go's regexp engine
    does not support negative lookahead, so we can't handle an application
    name containing a comma if the log_line_prefix has a comma following %a.
• Ignore CSV log files in log directory #83
  • Some Postgres installations are configured to log both standard-format
    log files and CSV log files to the same directory, but the collector
    currently reads all files specified in a db_log_location, which works
    poorly with this setup.
• Tweak collector sample config file to match setup instructions
• Improvements to "--discover-log-location"
  • Don't keep running if there's a config error
  • Drop the log_directory helper command and just fetch the setting from Postgres
  • Warn and only show relative location if log_directory is inside
    the data directory (this requires special setup steps to resolve)
• Improvements to "--test-logs"
  • Run privilege drop test when running log test as root, to allow running
    "--test-logs" for a complete log setup test, avoiding the need to run
    a full "--test"
• Update pg_query_go to incorporate memory leak fixes
• Check whether pg_stat_statements exists in a different schema, and give a
  clear error message
• Drop support for Postgres 9.2
  • Postgres 9.2 has been EOL for almost 4 years
• Update to Go 1.16
  • This introduces a change to Go's certificate handling, which may break
    certain older versions of Amazon RDS certificates, as they do not
    include a SAN. When this is the case you will see an error message like
    "x509: certificate relies on legacy Common Name field".
  • As a temporary workaround you can run the collector with the
    GODEBUG=x509ignoreCN=0 environment setting, which ignores these incorrect
    fields in these certificates. For a permanent fix, you need to update
    your RDS certificates to include the correct SAN field: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL-certificate-rotation.html

v0.36.0

• Config parsing improvements:
  • Fail fast when pganalyze section is missing in config file
  • Ignore duplicates in db_name config setting
    • Previously this could cause malformed snapshots that would be submitted
      correctly but could not be processed
  • Validate db_url parsing to avoid collector crash with invalid URLs
• Include pganalyze-collector-setup program (see 0.35 release notes) in supported packages
• Rename <unidentified queryid> query text placeholder to <query text unavailable>
  • This makes it clearer what the underlying issue is
• Revert to using <truncated query> instead of <unparsable query> in some situations
  • When a query is cut off due to pg_stat_activity limit being reached,
    show <truncated query>, to make it clear that increasing track_activity_query_size
    would solve the issue
• Ignore I/O stats for AWS Aurora utility statements
  • AWS Aurora appears to report incorrect blk_read_time and blk_write_time values
    for utility statements (i.e., non-SELECT/INSERT/UPDATE/DELETE); we zero these out for now
• Fix log-based EXPLAIN bug where query samples could be dropped if EXPLAIN failed
• Add U140 log event (inconsistent range bounds)
  • e.g.: ERROR: range lower bound must be less than or equal to range upper bound
• Fix issue where incomplete schema information in snapshots was not marked correctly
  • This could lead to schema objects disappearing and being re-created
• Fix trailing newline handling for GCP and self-hosted log streams
  • This could lead to queries being poorly formatted in the UI, or some queries
    with single-line comments being ignored
• Include additional collector configuration settings in snapshot metadata for diagnostics
• Ignore "insufficient privilege" queries w/o queryid
  • Previously, these could all be aggregated together yielding misleading stats