Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[#158456560] Add a lot of metrics to the collector #3

Merged
merged 17 commits into from
Jun 22, 2018

Commits on Jun 21, 2018

  1. Add --rm to docker run

    keymon committed Jun 21, 2018
    Configuration menu
    Copy the full SHA
    9d184a2 View commit details
    Browse the repository at this point in the history
  2. cloudwatch: Add more basic metrics

    Add a bunch of additional basic metrics as listed in [1]
    
    We do not test these, as there is not such a point because we mock
    the service.
    
    In this commit, we add the most valuable metrics in our opinion.
    
    [1] https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/rds-metricscollected.html
    keymon committed Jun 21, 2018
    Configuration menu
    Copy the full SHA
    eb294d9 View commit details
    Browse the repository at this point in the history
  3. cloudwatch: Add more RDS specific metrics

    Add more metrics available in cloudwatch for RDS[1]
    
    We add them in a separate commit as we are not sure about the real
    value of these metrics.
    
    [1] https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/rds-metricscollected.html
    keymon committed Jun 21, 2018
    Configuration menu
    Copy the full SHA
    3861553 View commit details
    Browse the repository at this point in the history
  4. Allow emit metrics with tags

    Add Tags as an attribute of the metrics.Metric object, which would
    be added when emitting the metric with loggregator.
    keymon committed Jun 21, 2018
    Configuration menu
    Copy the full SHA
    e65ac27 View commit details
    Browse the repository at this point in the history
  5. Add "source" tag to the current collectors

    we add tags source=cloudwatch and source=sql to each existing collector.
    keymon committed Jun 21, 2018
    Configuration menu
    Copy the full SHA
    c7bbbd2 View commit details
    Browse the repository at this point in the history
  6. sql_collector allows include tags in queries

    We want to add dynamic tags like database name, table name and so on
    to the metrics retrieved using SQL queries.
    
    For that we extend the sql connector, so that it will consider
    any additional column not defined as `MetricQueryMeta` as tags,
    and add it to the returned metrics.
    keymon committed Jun 21, 2018
    Configuration menu
    Copy the full SHA
    79e2ae2 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    72f1c7b View commit details
    Browse the repository at this point in the history
  8. postgres: sql query for dbsize and test DB

    We want to include the dbname tag with the current name, because
    we might allow multiple DBs per service instance in the future.
    
    Currently we only offer one DB per instance, which shall be returned
    by the `brokerinfo.ConnectionString()`, so we only return only
    `current_database()`.
    
    We change the test to create a temporary test DB with a random DB
    and delete it after the test. We had to kill all active connection
    before DROP that database using SQL commands, because golang
    sql.DB does keep a pool of connections and we did not find a way
    to close them explicitely.
    keymon committed Jun 21, 2018
    Configuration menu
    Copy the full SHA
    c67cc8e View commit details
    Browse the repository at this point in the history
  9. postgres: DB deadlocks counter metric

    The deadlocks counter metric keeps a counter of how many deadlocks
    have occurred per DB. A deadlock means that one transaction had to
    be cancelled.
    
    We add a test that simulates a deadlock.
    
    We only report `current_database()` for the time being as our broker
    only offers one DB per instance, but we might allow more later. We
    include the tag `dbname`.
    
    The example code to simulate the deadlock has been borrowed from:
    https://www.depesz.com/2012/02/09/waiting-for-9-2-deadlock-counter/
    keymon committed Jun 21, 2018
    Configuration menu
    Copy the full SHA
    28b1ca0 View commit details
    Browse the repository at this point in the history
  10. postgres: seq_scan and idx_scan per table

    We get the stats collector generated metric seq_scan and idx_scan
    for each table. We add the tags for the table_name.
    
    The dbname is forced to be `current_database()` but we might emit
    for all the DBs if we offer more databases per instance.
    
    We add a index to the films table and try to run queries that does
    a scan and a index to check the counter.
    keymon committed Jun 21, 2018
    Configuration menu
    Copy the full SHA
    ce71d31 View commit details
    Browse the repository at this point in the history
  11. postgres: idx_scan includes index_name tag

    Use the pg_stat_user_indexes tables to enrich the metric with the
    index name.
    keymon committed Jun 21, 2018
    Configuration menu
    Copy the full SHA
    1b4937e View commit details
    Browse the repository at this point in the history
  12. postgres: max_connections setting

    Return the max_connections allowed in the postgred server.
    This number is per configuration in the RDS parameter group[1],
    and it is constant.
    
    We report this value in the marketplace description, but it
    might be handy for the tenant to create dynamic dashboards and
    alerts, rather than hardcode values.
    
    [1] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithParamGroups.html
    keymon committed Jun 21, 2018
    Configuration menu
    Copy the full SHA
    941699d View commit details
    Browse the repository at this point in the history
  13. postgres: table_size per table

    Report the table_size in bytes per table. We include the table_name
    tag.
    
    The dbname tag is forced to be `current_database()` but we might emit
    for all the DBs if we offer more databases per instance.
    keymon committed Jun 21, 2018
    Configuration menu
    Copy the full SHA
    30e72a5 View commit details
    Browse the repository at this point in the history
  14. postgres: Add more metrics from pg_stat_database

    Add counter of commit/rollback transactions, read/hit block couter,
    read/write time and temporary bytes.
    
    See [1]
    
    [1] https://www.postgresql.org/docs/9.2/static/monitoring-stats.html#PG-STAT-DATABASE-VIEW
    keymon committed Jun 21, 2018
    Configuration menu
    Copy the full SHA
    570ef97 View commit details
    Browse the repository at this point in the history
  15. postgres: Metric for blocked_connections

    We test this with the deadlock test.
    keymon committed Jun 21, 2018
    Configuration menu
    Copy the full SHA
    4b9ff46 View commit details
    Browse the repository at this point in the history
  16. postgres: report max age of running transactions

    max_tx_age, transactions lasting too long indicate trouble[1]:
    transactions lasting too long, transactions not closed properly,
    slow queries or operations, etc.
    
    This metric will report the maximum transaction age in the moment
    the query is reported, but it will likely miss transactions shorter
    than the metric collection interval.
    
    [1] https://russ.garrett.co.uk/2015/10/02/postgres-monitoring-cheatsheet/
    keymon committed Jun 21, 2018
    Configuration menu
    Copy the full SHA
    4ed4817 View commit details
    Browse the repository at this point in the history

Commits on Jun 22, 2018

  1. ci/blackbox: Test for any cloudwatch metrics

    The CPUUTilization takes a while to appear. Consider the test
    OK if any cloudwatch metric is received.
    keymon committed Jun 22, 2018
    Configuration menu
    Copy the full SHA
    3508ce7 View commit details
    Browse the repository at this point in the history