Add collector for core RabbitMQ metrics

[gerhard]

This was done by @dcorbacho in dcorbacho@db4762d, I'm just submitting the PR
This is a follow-up from #63

This collector puts the minimum amount of pressure on RabbitMQ since the
metrics are not aggregated. Metrics are read from ETS tables, they are
strictly node local, and the amount of data is minimal. We wouldn't even
need rabbitmq_management_agent for the majority of core metrics, meaning
that prometheus_rabbitmq_exporter wouldn't need to depend on
rabbitmq_management if it was configured to use just this collector.

The other benefit of core metrics is that they are constantly updated,
since every object writes metrics on its own interval (typically every
5s). When /api/metrics is requested, raw values from ETS are read &
returned - no aggregation, no delay. What you get for core metrics is
what happened at most 5s ago.

Our end-goal is to expose raw, per-node metrics. Cluster metrics are
just an aggregation of per-node metrics, which should be done in
Prometheus/Grafana, just-in-time. This would help visualise cluster
imbalances, such as a subset of nodes running hotter than others, as
well as off-load all metrics processing so that RabbitMQ can spend as
many reductions on message-related work. We are imagining the equivalent
of Linux's /proc subsystem, but for RabbitMQ + Erlang + Mnesia.

I would be very excited to enable a mode for this plugin that only
exposes node-local metrics and doesn't require rabbitmq_management. Is
that something that you would like to have a go at @deadtricker?

For completeness, I'm adding the metrics that this collector adds to the
/api/metrics output of a freshly booted node:

# TYPE rabbitmq_core_connection_recv_oct counter
# HELP rabbitmq_core_connection_recv_oct Count of octects received on the connection.
# TYPE rabbitmq_core_connection_send_oct counter
# HELP rabbitmq_core_connection_send_oct Count of octects sent on the connection.
# TYPE rabbitmq_core_connection_reductions counter
# HELP rabbitmq_core_connection_reductions Count of reductions that take place on the queue process.
# TYPE rabbitmq_core_queue_messages_ready gauge
# HELP rabbitmq_core_queue_messages_ready Number of messages ready to be delivered to clients.
# TYPE rabbitmq_core_queue_messages_unacknowledge gauge
# HELP rabbitmq_core_queue_messages_unacknowledge Number of messages delivered to clients but not yet acknowledged.
# TYPE rabbitmq_core_queue_messages gauge
# HELP rabbitmq_core_queue_messages Sum of ready and unacknowledged messages (queue depth).
# TYPE rabbitmq_core_queue_reductions counter
# HELP rabbitmq_core_queue_reductions Count of reductions that take place on the queue process.
# TYPE rabbitmq_core_channel_exchange_publish gauge
# HELP rabbitmq_core_channel_exchange_publish Count of messages published.
# TYPE rabbitmq_core_channel_exchange_confirm gauge
# HELP rabbitmq_core_channel_exchange_confirm Count of messages confirmed.
# TYPE rabbitmq_core_channel_exchange_return_unroutable gauge
# HELP rabbitmq_core_channel_exchange_return_unroutable Count of messages returned to publisher as unroutable.
# TYPE rabbitmq_core_channel_process_reductions counter
# HELP rabbitmq_core_channel_process_reductions Count of reductions that take place on the channel process.
# TYPE rabbitmq_core_queue_disk_reads gauge
# HELP rabbitmq_core_queue_disk_reads Total number of times messages have been read from disk by this queue.
# TYPE rabbitmq_core_queue_disk_writes gauge
# HELP rabbitmq_core_queue_disk_writes Total number of times messages have been written to disk by this queue.
# TYPE rabbitmq_core_node_io_read_count counter
# HELP rabbitmq_core_node_io_read_count Read operations since node start.
rabbitmq_core_node_io_read_count{node="rabbit@08c5bed93480"} 1
# TYPE rabbitmq_core_node_io_read_bytes gauge
# HELP rabbitmq_core_node_io_read_bytes Bytes read since node start.
rabbitmq_core_node_io_read_bytes{node="rabbit@08c5bed93480"} 1
# TYPE rabbitmq_core_node_io_read_time gauge
# HELP rabbitmq_core_node_io_read_time Total time of read operations.
rabbitmq_core_node_io_read_time{node="rabbit@08c5bed93480"} 29
# TYPE rabbitmq_core_node_io_write_count counter
# HELP rabbitmq_core_node_io_write_count Write operations since node start.
rabbitmq_core_node_io_write_count{node="rabbit@08c5bed93480"} 0
# TYPE rabbitmq_core_node_io_write_bytes gauge
# HELP rabbitmq_core_node_io_write_bytes Bytes written since node start.
rabbitmq_core_node_io_write_bytes{node="rabbit@08c5bed93480"} 0
# TYPE rabbitmq_core_node_io_write_time gauge
# HELP rabbitmq_core_node_io_write_time Total time of write operations.
rabbitmq_core_node_io_write_time{node="rabbit@08c5bed93480"} 0
# TYPE rabbitmq_core_node_io_sync_count counter
# HELP rabbitmq_core_node_io_sync_count Sync operations since node start.
rabbitmq_core_node_io_sync_count{node="rabbit@08c5bed93480"} 0
# TYPE rabbitmq_core_node_io_sync_time gauge
# HELP rabbitmq_core_node_io_sync_time Total time of sync operations.
rabbitmq_core_node_io_sync_time{node="rabbit@08c5bed93480"} 0
# TYPE rabbitmq_core_node_io_seek_count counter
# HELP rabbitmq_core_node_io_seek_count Seek operations since node start.
rabbitmq_core_node_io_seek_count{node="rabbit@08c5bed93480"} 0
# TYPE rabbitmq_core_node_io_seek_time gauge
# HELP rabbitmq_core_node_io_seek_time Total time of seek operations.
rabbitmq_core_node_io_seek_time{node="rabbit@08c5bed93480"} 0
# TYPE rabbitmq_core_node_io_reopen_count counter
# HELP rabbitmq_core_node_io_reopen_count Times files have been reopened by the file handle cache.
rabbitmq_core_node_io_reopen_count{node="rabbit@08c5bed93480"} 0
# TYPE rabbitmq_core_node_mnesia_ram_tx_count counter
# HELP rabbitmq_core_node_mnesia_ram_tx_count Mnesia transactions in RAM since node start.
rabbitmq_core_node_mnesia_ram_tx_count{node="rabbit@08c5bed93480"} 11
# TYPE rabbitmq_core_node_mnesia_disk_tx_count counter
# HELP rabbitmq_core_node_mnesia_disk_tx_count Mnesia transactions in disk since node start.
rabbitmq_core_node_mnesia_disk_tx_count{node="rabbit@08c5bed93480"} 12
# TYPE rabbitmq_core_node_msg_store_read_count counter
# HELP rabbitmq_core_node_msg_store_read_count Read operations in the message store since node start.
rabbitmq_core_node_msg_store_read_count{node="rabbit@08c5bed93480"} 0
# TYPE rabbitmq_core_node_msg_store_write_count counter
# HELP rabbitmq_core_node_msg_store_write_count Write operations in the message store since node start.
rabbitmq_core_node_msg_store_write_count{node="rabbit@08c5bed93480"} 0
# TYPE rabbitmq_core_queue_index_journal_write_count counter
# HELP rabbitmq_core_queue_index_journal_write_count Write operations in the queue index journal since node start.
rabbitmq_core_queue_index_journal_write_count{node="rabbit@08c5bed93480"} 0
# TYPE rabbitmq_core_queue_index_write_count counter
# HELP rabbitmq_core_queue_index_write_count Queue index write operations since node start.
# TYPE rabbitmq_core_queue_index_read_count counter
# HELP rabbitmq_core_queue_index_read_count Queue index read operations since node start.
# TYPE rabbitmq_core_queue_io_file_handle_open_attempt_count counter
# HELP rabbitmq_core_queue_io_file_handle_open_attempt_count File descriptor open attempts.
rabbitmq_core_queue_io_file_handle_open_attempt_count{node="rabbit@08c5bed93480"} 10
# TYPE rabbitmq_core_queue_io_file_handle_open_attempt_time gauge
# HELP rabbitmq_core_queue_io_file_handle_open_attempt_time Total time of file descriptor open attempts.
rabbitmq_core_queue_io_file_handle_open_attempt_time{node="rabbit@08c5bed93480"} 264
# TYPE rabbitmq_core_node_fd_used gauge
# HELP rabbitmq_core_node_fd_used File descriptors used.
rabbitmq_core_node_fd_used{node="rabbit@08c5bed93480"} 81
# TYPE rabbitmq_core_node_sockets_used gauge
# HELP rabbitmq_core_node_sockets_used Sockets used.
rabbitmq_core_node_sockets_used{node="rabbit@08c5bed93480"} 0
# TYPE rabbitmq_core_node_mem_used gauge
# HELP rabbitmq_core_node_mem_used Memory used in bytes.
rabbitmq_core_node_mem_used{node="rabbit@08c5bed93480"} 97021952
# TYPE rabbitmq_core_node_disk_free gauge
# HELP rabbitmq_core_node_disk_free Disk free in bytes.
rabbitmq_core_node_disk_free{node="rabbit@08c5bed93480"} 55178477568
# TYPE rabbitmq_core_node_proc_used gauge
# HELP rabbitmq_core_node_proc_used Erlang processes used.
rabbitmq_core_node_proc_used{node="rabbit@08c5bed93480"} 399
# TYPE rabbitmq_core_node_gc_num counter
# HELP rabbitmq_core_node_gc_num GC runs.
rabbitmq_core_node_gc_num{node="rabbit@08c5bed93480"} 5141
# TYPE rabbitmq_core_node_gc_bytes_reclaimed counter
# HELP rabbitmq_core_node_gc_bytes_reclaimed Bytes reclaimed by GC.
rabbitmq_core_node_gc_bytes_reclaimed{node="rabbit@08c5bed93480"} 379204736
# TYPE rabbitmq_core_node_context_switches counter
# HELP rabbitmq_core_node_context_switches Context switches since node start.
rabbitmq_core_node_context_switches{node="rabbit@08c5bed93480"} 190670
# TYPE rabbitmq_core_connection_created counter
# HELP rabbitmq_core_connection_created Connections created.
rabbitmq_core_connection_created{node="rabbit@08c5bed93480"} 0
# TYPE rabbitmq_core_connection_closed counter
# HELP rabbitmq_core_connection_closed Connections closed.
rabbitmq_core_connection_closed{node="rabbit@08c5bed93480"} 0
# TYPE rabbitmq_core_channel_created counter
# HELP rabbitmq_core_channel_created Channels created.
rabbitmq_core_channel_created{node="rabbit@08c5bed93480"} 0
# TYPE rabbitmq_core_channel_closed counter
# HELP rabbitmq_core_channel_closed Channels closed.
rabbitmq_core_channel_closed{node="rabbit@08c5bed93480"} 0
# TYPE rabbitmq_core_queue_declared counter
# HELP rabbitmq_core_queue_declared Queues declared.
rabbitmq_core_queue_declared{node="rabbit@08c5bed93480"} 0
# TYPE rabbitmq_core_queue_deleted counter
# HELP rabbitmq_core_queue_deleted Queues deleted.
rabbitmq_core_queue_deleted{node="rabbit@08c5bed93480"} 0
# TYPE rabbitmq_core_node_node_send_bytes counter
# HELP rabbitmq_core_node_node_send_bytes Count of bytes sent to node.
# TYPE rabbitmq_core_node_node_recv_bytes counter
# HELP rabbitmq_core_node_node_recv_bytes Count of bytes received from node.
# TYPE rabbitmq_core_channel_queue_get counter
# HELP rabbitmq_core_channel_queue_get Count of messages delivered in acknowledgement mode in response to basic.get.
# TYPE rabbitmq_core_channel_queue_get_no_ack counter
# HELP rabbitmq_core_channel_queue_get_no_ack Count of messages delivered in no-acknowledgement mode in response to basic.get.
# TYPE rabbitmq_core_channel_queue_deliver counter
# HELP rabbitmq_core_channel_queue_deliver Count of messages delivered in acknowledgement mode to consumers.
# TYPE rabbitmq_core_channel_queue_deliver_no_ack counter
# HELP rabbitmq_core_channel_queue_deliver_no_ack Count of messages delivered in no-acknowledgement mode to consumers.
# TYPE rabbitmq_core_channel_queue_redeliver counter
# HELP rabbitmq_core_channel_queue_redeliver Count of subset of delivered messages which had the redelivered flag set.
# TYPE rabbitmq_core_channel_queue_ack counter
# HELP rabbitmq_core_channel_queue_ack Count of messages acknowledged.
# TYPE rabbitmq_core_channel_queue_get_empty counter
# HELP rabbitmq_core_channel_queue_get_empty Count of basic.get operations on empty queues.
# TYPE rabbitmq_core_channel_consumer_count gauge
# HELP rabbitmq_core_channel_consumer_count Consumers count.
# TYPE rabbitmq_core_channel_messages_unacknowledged gauge
# HELP rabbitmq_core_channel_messages_unacknowledged Count of messages unacknowledged.
# TYPE rabbitmq_core_channel_messages_unconfirmed gauge
# HELP rabbitmq_core_channel_messages_unconfirmed Count of messages unconfirmed.
# TYPE rabbitmq_core_channel_messages_uncommited gauge
# HELP rabbitmq_core_channel_messages_uncommited Count of messages uncommited.
# TYPE rabbitmq_core_channel_messages_prefetch_count gauge
# HELP rabbitmq_core_channel_messages_prefetch_count Limit to the number of unacknowledged messages on every connection on a channel.
# TYPE rabbitmq_core_channel_messages_global_prefetch_count gauge
# HELP rabbitmq_core_channel_messages_global_prefetch_count Global limit to the number of unacknowledged messages shared between all connections on a channel.
# TYPE rabbitmq_core_connection_recv_count counter
# HELP rabbitmq_core_connection_recv_count Count of bytes received on the connection.
# TYPE rabbitmq_core_connection_send_count counter
# HELP rabbitmq_core_connection_send_count Count of bytes send on the connection.
# TYPE rabbitmq_core_node_fd_total gauge
# HELP rabbitmq_core_node_fd_total File descriptors available.
rabbitmq_core_node_fd_total{node="rabbit@08c5bed93480"} 1048576
# TYPE rabbitmq_core_node_sockets_total gauge
# HELP rabbitmq_core_node_sockets_total Sockets available.
rabbitmq_core_node_sockets_total{node="rabbit@08c5bed93480"} 943626
# TYPE rabbitmq_core_node_mem_limit gauge
# HELP rabbitmq_core_node_mem_limit Memory usage high watermark.
rabbitmq_core_node_mem_limit{node="rabbit@08c5bed93480"} 13475676160
# TYPE rabbitmq_core_node_disk_free_limit gauge
# HELP rabbitmq_core_node_disk_free_limit Free disk space low watermark.
rabbitmq_core_node_disk_free_limit{node="rabbit@08c5bed93480"} 50000000
# TYPE rabbitmq_core_node_proc_total gauge
# HELP rabbitmq_core_node_proc_total Erlang processes limit.
rabbitmq_core_node_proc_total{node="rabbit@08c5bed93480"} 1048576
# TYPE rabbitmq_core_node_uptime gauge
# HELP rabbitmq_core_node_uptime Time in milliseconds since node start.
rabbitmq_core_node_uptime{node="rabbit@08c5bed93480"} 24942
# TYPE rabbitmq_core_node_run_queue gauge
# HELP rabbitmq_core_node_run_queue Runtime run queue.
rabbitmq_core_node_run_queue{node="rabbit@08c5bed93480"} 1
# TYPE rabbitmq_core_node_processors gauge
# HELP rabbitmq_core_node_processors Logical processors.
rabbitmq_core_node_processors{node="rabbit@08c5bed93480"} 10
# TYPE rabbitmq_core_node_net_ticktime gauge
# HELP rabbitmq_core_node_net_ticktime Periodic tick interval between all pairs of nodes to maintain the connections and to detect disconnections.
rabbitmq_core_node_net_ticktime{node="rabbit@08c5bed93480"} 60
# TYPE rabbitmq_core_channel_queue_exchange_publish counter
# HELP rabbitmq_core_channel_queue_exchange_publish Count of messages published.
# TYPE rabbitmq_core_connections gauge
# HELP rabbitmq_core_connections RabbitMQ Connections count.
rabbitmq_core_connections 0
# TYPE rabbitmq_core_channels gauge
# HELP rabbitmq_core_channels RabbitMQ Channels count.
rabbitmq_core_channels 0
# TYPE rabbitmq_core_consumers gauge
# HELP rabbitmq_core_consumers RabbitMQ Consumers count.
rabbitmq_core_consumers 0
# TYPE rabbitmq_core_queues gauge
# HELP rabbitmq_core_queues RabbitMQ Queues count.

[gerhard]

Build passes, check out all these new & beautiful rabbitmq_core_* metrics: https://travis-ci.org/deadtrickster/prometheus_rabbitmq_exporter/builds/499414875#L940-L1140

I'm thinking that it's time to cut a new version after this is merged, would you agree @deadtrickster ?

[deadtrickster]

Yeah, +1 for the new version. Actually at the beginning I was reading ETS tables directly too for some metrics. Was really easy to break. Hopefully I was doing different.

[deadtrickster]

Will merge and release tomorrow with a fresh head.

[deadtrickster]

I would be very excited to enable a mode for this plugin that only
exposes node-local metrics and doesn't require rabbitmq_management. Is
that something that you would like to have a go at @deadtricker?

Yeah, all parts are there. Perhaps a configurable switch? Also need to agree on http interface replacement. This plugin mounts itself inside management api space. But if that configurable switch is on, metrics can be exposed via raw prometheus_httpd using the same default /api/metrics.

[deadtrickster]

Merged, will release soon.