Skip to content

Metrics Data Retrieval

Jump to bottom
Balaji Bal edited this page Oct 26, 2020 · 6 revisions

Metrics Data Retrieval

The following code shows how metrics can be stored on the Data Platform.

from ferris_cli.ferris_cli import MetricMessage
from ferris_cli.ferris_cli import MetricsAPI


# Create and send a MetricMessage
mm = MetricMessage('dataplatform.jupyter.users',28)
print(mm.toJSON())
mapi = MetricsAPI().send(mm)

Metric Model

Each metric message is sent to Kafka in a JSON Format and consists of the following attributes

  • metric_key: A identifier for the specific namespace/metrics series represented in a dotted such as 'dataplatfrom.data.hive.curatedzone.record_counts.crm'.
  • metric_value: The metric value is an integer value.
  • update_time : The update timestamp is injected by the ferris cli library.

Metrics Flow within Data Platform

When an application emits a metric message within the Data Platform utilizing the ferris-cli library or directly( formatted as a JSON Formatted message) it is sent to the ferris.metrics Kafka Topic.

Applications written in other languages or which do not use the ferris-cli API may send metrics to be aggregated by sending a JSON message such as the following to the Kafka topic ferris.metrics.

{
   "metric_key":"dataplatform.jupyter.users",
   "metric_value":28,
   "update_time":"2020-10-24T12:42:11Z"
}

The metrics are then routed by the 'Elasticsearch Metrics Sink' and stored within the 'ferris-metrics' index in Elasticsearch for medium term storage (limit around 100M Metrics). Elasticsearch serves as the primary source for metrics retrieval for reporting and dashboarding. This design decision is based on the fast response times Elasticsearch provides.

Metrics are also routed to a HDFS and subsequently Hive for long term storage.

Metrics Retrieval API

The metrics retrieval API is an API for retrieving for presentation within dashboards and reports. It is modelled around de-facto conventions used by metrics databases such as Graphite, InfluxDB or Time Series DB.

The metrics retrieval API requires the following parameters to extract either a specific metric or a collection of metrics.

  • metric_type: The type of metrics. Allows for wildcards. For example 'dataplatform.hive.tables.record_counts.*' retrives all metrics under the path 'dataplatform.hive.tables.record_counts'
  • time_from: The start time from which the metrics to be retrieved. Example "2020-08-27T11:47:09Z"
  • time_to: The end time from which the metrics to be retrieved. Example "2020-11-30T11:47:09Z"
  • start_from: The starting record for purpose of paging ( default is zero )
  • result_size: The number of records to be retrieved. (default is 1000 )

Defining metrics namespaces

Defining metrics namespaces are critical in order to provide a manageable and easy to query metrics data set.

Some examples are provided below

  • dpr.minio.* : For metrics around minio
  • dpr.hive.record_counts.cur_zone.table_name
Clone this wiki locally