Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Yaml configuration #211

Merged
merged 25 commits into from
May 23, 2023
Merged

Conversation

DifferentialOrange
Copy link
Member

@DifferentialOrange DifferentialOrange commented May 12, 2023

Run

make CONFIG=config.yml build

to build a dashboard based on your custom configuration. (Use OUTPUT=mydashboard.json to customize output path.) You'll need jsonnet 0.20.0 or newer for that.

Configuration description

# type is 'prometheus' or 'influxdb'
type: prometheus

# title is dashboard artifact title
# default is 'Tarantool dashboard'
title: Tarantool dashboard

# description is dashboard artifact description
# default is 'Dashboard for Tarantool application and database server monitoring, based on grafonnet library.'
description: Dashboard for Tarantool application and database server monitoring, based on grafonnet library.

# grafana_tags are dashboard artifact tags
# They do not related to Prometheus/InfluxDB quieries,
# they are a part of Grafana UI and can be used to search dashboards in the library.
# default is [tarantool]
grafana_tags: [tarantool]

# datasource is the name on metrics datasource
# Use expression like '$prometheus' to work with dynamic datasource set with variable.
# Use string like 'MyDashboard' to set your specific datasource.
# default for prometheus type is '$prometheus', for influxdb type is '$influxdb'
datasource: '$prometheus'

# policy is InfluxDB datasource metrics retention policy
# influxdb type only
# Use expression like '$policy' to work with dynamic policy set with variable.
# Use string like 'MyPolicy' to set your specific policy.
# Use 'default' to work with default InfluxDB datasource policy.
# default is '$policy'
# policy: '$policy'

# measurement is InfluxDB datasource metrics measurement
# influxdb type only
# Use expression like '$measurement' to work with dynamic measurement set with variable.
# Use string like 'MyMeasurement' to set your specific measurement.
# default is '$measurement'
# measurement: '$measurement'

# filters are Prometheus/InfluxQL query filters
# By specifying them, you can filter metrics by Prometheus label conditions
# or InfluxDB tag conditions.
# Use app: ['=', 'myapp'] to display only metrics with 'myapp' app label/tag.
# Use app: ['=~', '/^myapp\d/'] to display only metrics with app label/tag satisfying
# a regular expression. See PromQL/InfluxQL documentation for a regex syntax.
# Use job: ['=~', '$job'] to display only metrics with '$job' variable value job label/tag.
# Beware that only job and alias (label_pairs_alias for InfluxDB) template variables
# would be built automatically, if required.
# default is {}
filters: {
  job: ['=~', '$job'],
  alias: ['=~', '$alias'],
}

# metrics_prefix is an additional prefix for metric names in Prometheus/InfluxQL queries
# Use 'myapp_' if you expose metrics like 'myapp_tnt_uptime_info'
# default is ''
metrics_prefix: ''

# sections are dashboard panels sections to be included
# The list of supported sections:
# - cluster
# - replication
# - http
# - net
# - slab
# - mvcc
# - space
# - vinyl
# - cpu
# - cpu_extended' # for TDG
# - runtime
# - luajit
# - operations
# - crud
# - expirationd
# - tdg_kafka_common
# - tdg_kafka_brokers
# - tdg_kafka_topics
# - tdg_kafka_consumer
# - tdg_kafka_producer
# - tdg_tuples
# - tdg_file_connectors
# - tdg_graphql
# - tdg_iproto
# - tdg_rest_api
# - tdg_tasks
# default is [cluster, replication, http, net, slab, mvcc, space, vinyl, cpu, runtime, luajit, operations, crud, expirationd]
sections:
- cluster
- replication
- http
- net
- slab
- mvcc
- space
- vinyl
- cpu
- runtime
- luajit
- operations
- crud
- expirationd

Example: prometheus dashboard with template variable

type: prometheus
datasource: '$prometheus'
filters: {
  job: ['=~', '$job'],
  alias: ['=~', '$alias'],
}

Example: prometheus static dashboard

type: prometheus
datasource: MyPrometheusWithTarantoolMetrics
filters: {
  job: ['=', 'MyJobWithTarantoolMetrics'],
}

Example: prometheus static dashboard with additional labels

type: prometheus
datasource: MyPrometheusWithTarantoolMetrics
filters: {
  job: ['=', 'MyJobWithTarantoolMetrics'],
  vendor_app_tag: ['=', 'MyCacheApplication'],
}

Example: prometheus static dashboard with specific sections

type: prometheus
datasource: MyPrometheusWithTarantoolMetrics
filters: {
  job: ['=', 'MyJobWithTarantoolMetrics'],
}
sections:
- cluster
- replication
- http
- net
- slab
- space
- runtime
- luajit
- operations

Example: prometheus static TDG dashboard

type: prometheus
title: Tarantool Data Grid dashboard
grafana_tags: [tarantool, TDG]
datasource: MyPrometheusWithTDGMetrics
filters: {
  job: ['=', 'MyJobWithTDGMetrics'],
}
sections:
- cluster
- replication
- http
- net
- slab
- mvcc
- space
- vinyl
- cpu_extended
- runtime
- luajit
- operations
- crud
- tdg_kafka_common
- tdg_kafka_brokers
- tdg_kafka_topics
- tdg_kafka_consumer
- tdg_kafka_producer
- expirationd
- tdg_tuples
- tdg_file_connectors
- tdg_graphql
- tdg_iproto
- tdg_rest_api
- tdg_tasks

Example: influxdb dashboard with template variable

type: influxdb
datasource: '$influxdb'
policy: '$policy'
measurement: '$measurement'
filters: {
  label_pairs_alias: ['=~', '/^$label_pairs_alias$/'],
}

Example: influxdb static dashboard

type: influxdb
datasource: MyInfluxDBWithTarantoolMetrics
policy: default
measurement: MyMeasurementWithTarantoolMetrics

Closes #204
Closes #206
Closes #207

@DifferentialOrange DifferentialOrange force-pushed the DifferentialOrange/yaml-configuration branch 5 times, most recently from a201126 to b517c65 Compare May 16, 2023 12:48
After this patch, code uses a single place to extract the configuration
parameters like datasource type, datasource, datasource parameters,
alias filters etc.

Part of #206
Part of #207
After this patch, code uses a single place to extract the configuration
parameters like datasource type, datasource, datasource parameters,
alias filters etc.

Part of #206
This patch finishes the migration of existing queries to new filter
system.

Part of #206
After this patch, it will be possible to disable some default filters,
like job.

Part of #206
Remove redundant "include all" regex for alias if no condition required
(like in some static built dashboards).

Part of #206
Support custom description and Grafana tags on manual build.
Build template queries based on cfg value.
To build a dashboard from YAML config, run
```
CONFIG=config.yml make build
```

Part of #206
Part of #207
@DifferentialOrange DifferentialOrange force-pushed the DifferentialOrange/yaml-configuration branch from b517c65 to 31445fe Compare May 17, 2023 12:20
@DifferentialOrange DifferentialOrange marked this pull request as ready for review May 17, 2023 12:21
@DifferentialOrange
Copy link
Member Author

I have asked @palage4a to try this implementation to solve their issues. If everything would be alright, I'll merge this PR.

@DifferentialOrange DifferentialOrange force-pushed the DifferentialOrange/yaml-configuration branch from 3fc1eb4 to 5ee07d6 Compare May 23, 2023 16:10
Grafana revisions:
- InfluxDB revision 17 [1],
- Prometheus revision 17 [2],
- InfluxDB TDG revision 6 [3],
- Prometheus TDG revision 6 [4].

This release introduces several breaking changes for those using this
repository as a library after migrating to new build system. See updated
README.

Added:
- Custom filters support on static build
- Option to skip Prometheus job on static build
- Option to static build with custom set of sections
- Custom description and Grafana tags on manual build
- API to build a dashboard from YAML config
- Support for metrics prefix on manual build

Changed:
- Display two digits after decimal point instead of three
- Deprecate make single line static build
- Hide additional columns in overview table

Fixed:
- TDG Kafka brokers rps panels alias condition for InfluxDB
- Fill missing values with null on TDG CPU panels for InfluxDB
- Remove useless kind group in TDG jobs panels for InfluxDB
- CRUD tuples query alias filtering for Prometheus
- TDG tuples query alias filtering
- Policy in alias extraction for InfluxDB dashboards
- Remove redundant "include all" regex for alias if no condition required

1. https://grafana.com/api/dashboards/12567/revisions/17/download
2. https://grafana.com/api/dashboards/13054/revisions/17/download
3. https://grafana.com/api/dashboards/16405/revisions/6/download
4. https://grafana.com/api/dashboards/16406/revisions/6/download
@DifferentialOrange DifferentialOrange force-pushed the DifferentialOrange/yaml-configuration branch from 21bcffd to 742c14f Compare May 23, 2023 16:17
@DifferentialOrange DifferentialOrange merged commit 7869a11 into master May 23, 2023
@DifferentialOrange DifferentialOrange deleted the DifferentialOrange/yaml-configuration branch May 23, 2023 16:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant