switch postgresql drivers to support more column types out of the box.

[james-lawrence]

previous pr/discussion: #1617

currently lib/pq doesn't handle name, oid types, and I suspect other builtin postgersql types consistently making adding custom queries for some tables difficult.

It is doubtful pq will change the current output of these types to be friendlier to the end user as the maintainer is pretty hardline about not changing behaviour, even when its for the better.

switching to pgx allows support for all the currently ignored columns except stats_reset, and that is a problem with writing output not the db driver. it likely could be fixed in a follow up pr.

copied the ParseURL from lib/pq as it was a dependency whose logic I didn't wish to change, but at the same time didn't want a dependency on the driver just for that function.

[sparrc]

this is a bit of an implementation detail that doesn't need to be part of the changelog. What does this mean for the end-user of the postgres plugin?

[james-lawrence]

sorry, I should change the wording to actually allow oid/name types =)

[sparrc]

what does this change do for the postgres plugin user? can you update the readme(s)?

[james-lawrence]

will look into the failures tomorrow night.

[james-lawrence]

I'll fix the conflicts saturday, any timeline for this getting merged into mainline?

[sparrc]

I'll set this to the 1.2 milestone, though I can't guarantee we'll have time to get it reviewed in time.

[sparrc]

@james-lawrence can you describe how you've gone about testing change? how have you verified that this won't break current users of the postgres & postgres_extensible plugins?

[james-lawrence]

Honestly I havent done a full on regression test.

• This PR only exposes new functionality that previously was not enabled.
• All the changes are only in the internal representation within the plugin, there were not any changes to the emitted data. In particular the types string (emits a string, same as original), int and int32 (emits a base 10 string same as when only int64 was supported) have been added internally but the resulting output is the same.
• I made sure to use the code from libpq for forming the db connection.

I will do a smoke test when I fix the merge conflict of running a configuration using master and then upgrading to this version.

[james-lawrence]

I'll fix the conflict this weekend.

[james-lawrence]

I compared the output of master and this branch only 1 line changed and it looks to be a simple reordering of the columns:

master: pg.stat.disk.usage.relation,server=application_name\=telegraf\ dbname\=postgres\ host\=localhost\ user\=james,db=postgres,table_schema=pg_catalog,table_name=pg_index,host=localhost index_bytes=32768i,toast_bytes=0i,table_bytes=57344i,oid="2610",row_estimate=121,total_bytes=90112i 1484406693000000000
branch: pg.stat.disk.usage.relation,table_name=pg_index,host=localhost,server=application_name\=telegraf\ dbname\=postgres\ host\=localhost\ user\=james,db=postgres,table_schema=pg_catalog table_bytes=57344i,oid="2610",row_estimate=121,total_bytes=90112i,index_bytes=32768i,toast_bytes=0i 1484406714000000000

here they are reordered, they match exactly:

master: pg.stat.disk.usage.relation,server=application_name\=telegraf\ dbname\=postgres\ host\=localhost\ user\=james,db=postgres,table_schema=pg_catalog,table_name=pg_index,host=localhost index_bytes=32768i,toast_bytes=0i,table_bytes=57344i,oid="2610",row_estimate=121,total_bytes=90112i 1484406693000000000
branch: pg.stat.disk.usage.relation,server=application_name\=telegraf\ dbname\=postgres\ host\=localhost\ user\=james,db=postgres,table_schema=pg_catalog,table_name=pg_index,host=localhost index_bytes=32768i,toast_bytes=0i,table_bytes=57344i,oid="2610",row_estimate=121,total_bytes=90112i 1484406714000000000

This was the configuration I used for both master and this branch.

# Telegraf Configuration
#
# Telegraf is entirely plugin driven. All metrics are gathered from the
# declared inputs, and sent to the declared outputs.
#
# Plugins must be declared in here to be active.
# To deactivate a plugin, comment out the name and any variables.
#
# Use 'telegraf -config telegraf.conf -test' to see what metrics a config
# file would generate.
#
# Environment variables can be used anywhere in this config file, simply prepend
# them with $. For strings the variable must be within quotes (ie, "$STR_VAR"),
# for numbers and booleans they should be plain (ie, $INT_VAR, $BOOL_VAR)


# Global tags can be specified here in key="value" format.
[global_tags]
  # dc = "us-east-1" # will tag all metrics with dc=us-east-1
  # rack = "1a"
  ## Environment variables can be used as tags, and throughout the config file
  # user = "$USER"


# Configuration for telegraf agent
[agent]
  ## Default data collection interval for all inputs
  interval = "10s"
  ## Rounds collection interval to 'interval'
  ## ie, if interval="10s" then always collect on :00, :10, :20, etc.
  round_interval = true

  ## Telegraf will send metrics to outputs in batches of at most
  ## metric_batch_size metrics.
  ## This controls the size of writes that Telegraf sends to output plugins.
  metric_batch_size = 1000

  ## For failed writes, telegraf will cache metric_buffer_limit metrics for each
  ## output, and will flush this buffer on a successful write. Oldest metrics
  ## are dropped first when this buffer fills.
  ## This buffer only fills when writes fail to output plugin(s).
  metric_buffer_limit = 10000

  ## Collection jitter is used to jitter the collection by a random amount.
  ## Each plugin will sleep for a random time within jitter before collecting.
  ## This can be used to avoid many plugins querying things like sysfs at the
  ## same time, which can have a measurable effect on the system.
  collection_jitter = "0s"

  ## Default flushing interval for all outputs. You shouldn't set this below
  ## interval. Maximum flush_interval will be flush_interval + flush_jitter
  flush_interval = "10s"
  ## Jitter the flush interval by a random amount. This is primarily to avoid
  ## large write spikes for users running a large number of telegraf instances.
  ## ie, a jitter of 5s and interval 10s means flushes will happen every 10-15s
  flush_jitter = "0s"

  ## By default, precision will be set to the same timestamp order as the
  ## collection interval, with the maximum being 1s.
  ## Precision will NOT be used for service inputs, such as logparser and statsd.
  ## Valid values are "ns", "us" (or "µs"), "ms", "s".
  precision = ""

  ## Logging configuration:
  ## Run telegraf with debug log messages.
  debug = false
  ## Run telegraf in quiet mode (error log messages only).
  quiet = false
  ## Specify the log file name. The empty string means to log to stderr.
  logfile = ""

  ## Override default hostname, if empty use os.Hostname()
  hostname = ""
  ## If set to true, do no set the "host" tag in the telegraf agent.
  omit_hostname = false


###############################################################################
#                            OUTPUT PLUGINS                                   #
###############################################################################

# Configuration for influxdb server to send metrics to
[[outputs.influxdb]]
  ## The full HTTP or UDP endpoint URL for your InfluxDB instance.
  ## Multiple urls can be specified as part of the same cluster,
  ## this means that only ONE of the urls will be written to each interval.
  # urls = ["udp://localhost:8089"] # UDP endpoint example
  urls = ["http://localhost:8086"] # required
  ## The target database for metrics (telegraf will create it if not exists).
  database = "telegraf" # required

  ## Retention policy to write to. Empty string writes to the default rp.
  retention_policy = ""
  ## Write consistency (clusters only), can be: "any", "one", "quorum", "all"
  write_consistency = "any"

  ## Write timeout (for the InfluxDB client), formatted as a string.
  ## If not provided, will default to 5s. 0s means no timeout (not recommended).
  timeout = "5s"


###############################################################################
#                            INPUT PLUGINS                                    #
###############################################################################
# # Read metrics from one or many postgresql servers
[[inputs.postgresql_extensible]]
  # specify address via a url matching:
  # postgres://[pqgotest[:password]]@localhost[/dbname]?sslmode=...
  # or a simple string:
  #   host=localhost user=pqotest password=... sslmode=... dbname=app_production
  #
  # All connection parameters are optional.  #
  # Without the dbname parameter, the driver will default to a database
  # with the same name as the user. This dbname is just for instantiating a
  # connection with the server and doesn't restrict the databases we are trying
  # to grab metrics for.
  #
  address = "postgres://localhost/postgres?sslmode=disable&application_name=telegraf"
  # A list of databases to pull metrics about. If not specified, metrics for all
  # databases are gathered.
  databases = ["postgres"]
  #
  # Define the toml config where the sql queries are stored
  # New queries can be added, if the withdbname is set to true and there is no
  # databases defined in the 'databases field', the sql query is ended by a 'is
  # not null' in order to make the query succeed.
  # Be careful that the sqlquery must contain the where clause with a part of
  # the filtering, the plugin will add a 'IN (dbname list)' clause if the
  # withdbname is set to true
  # Example :
  # The sqlquery : "SELECT * FROM pg_stat_database where datname" become
  # "SELECT * FROM pg_stat_database where datname IN ('postgres', 'pgbench')"
  # because the databases variable was set to ['postgres', 'pgbench' ] and the
  # withdbname was true.
  # Be careful that if the withdbname is set to false you don't have to define
  # the where clause (aka with the dbname)
  # the tagvalue field is used to define custom tags (separated by comas)
  #
  # Structure :
  # [[inputs.postgresql_extensible.query]]
  #   sqlquery string
  #   version string
  #   withdbname boolean
  #   tagvalue string (coma separated)
  [[inputs.postgresql_extensible.query]]
    sqlquery="SELECT * FROM pg_stat_database"
    version=901
    withdbname=false
    tagvalue="datname"
    measurement="pg.stat.database"

  [[inputs.postgresql_extensible.query]]
    sqlquery="SELECT * FROM pg_stat_bgwriter"
    version=901
    withdbname=false
    tagvalue=""
    measurement="pg.stat.bgwriter"

  [[inputs.postgresql_extensible.query]]
    sqlquery="SELECT relid, schemaname, relname, seq_scan, seq_tup_read, idx_scan, idx_tup_fetch, n_tup_ins, n_tup_upd, n_tup_del, n_tup_hot_upd, n_live_tup, n_dead_tup, n_mod_since_analyze, vacuum_count, autovacuum_count, analyze_count, autoanalyze_count FROM pg_stat_user_tables"
    version=950
    withdbname=false
    tagvalue="relname"
    measurement="pg.stat.table"

  [[inputs.postgresql_extensible.query]]
    sqlquery="SELECT relid, schemaname, relname, heap_blks_read, heap_blks_hit, idx_blks_read, idx_blks_hit, COALESCE(toast_blks_read, 0) AS toast_blks_read, COALESCE(toast_blks_hit, 0) AS toast_blks_hit, COALESCE(tidx_blks_read, 0) AS tidx_blks_read, COALESCE(tidx_blks_hit, 0) AS tidx_blks_hit FROM pg_statio_user_tables"
    version=901
    withdbname=false
    tagvalue="relname"
    measurement="pg.stat.io.table"

  [[inputs.postgresql_extensible.query]]
    sqlquery="SELECT * FROM pg_stat_xact_user_tables"
    version=901
    withdbname=false
    tagvalue="relname"
    measurement="pg.stat.xact.table"

  [[inputs.postgresql_extensible.query]]
    sqlquery="SELECT * FROM pg_stat_user_indexes"
    version=901
    withdbname=false
    tagvalue="relname,indexrelname"
    measurement="pg.stat.indexes"

  [[inputs.postgresql_extensible.query]]
    sqlquery="SELECT * FROM pg_statio_user_indexes"
    version=901
    withdbname=false
    tagvalue="relname,indexrelname"
    measurement="pg.stat.io.indexes"

  [[inputs.postgresql_extensible.query]]
    sqlquery="""
    SELECT *, total_bytes - index_bytes - toast_bytes AS table_bytes FROM (
      SELECT c.oid,nspname AS table_schema, relname AS TABLE_NAME,
             c.reltuples AS row_estimate,
             pg_total_relation_size(c.oid) AS total_bytes,
             pg_indexes_size(c.oid) AS index_bytes,
             COALESCE(pg_total_relation_size(reltoastrelid), 0) AS toast_bytes
      FROM pg_class c
      LEFT JOIN pg_namespace n ON n.oid = c.relnamespace
      WHERE relkind = 'r'
    ) AS a"""
    version=901
    withdbname=false
    tagvalue="table_schema,table_name"
    measurement="pg.stat.disk.usage.relation"

[sparrc]

thanks for testing that out @james-lawrence, I'll merge this now and hopefully give it some soak time before the 1.3 release

[anilkulkarni87]

Can we connect to redshift using the postgresql_extensible plugin?