diff --git a/docs/admin/monitoring/index.md b/docs/admin/monitoring/index.md index 47a78a32..ce9942d0 100644 --- a/docs/admin/monitoring/index.md +++ b/docs/admin/monitoring/index.md @@ -4,8 +4,9 @@ # Monitoring and diagnostics It is important to continuously monitor your CrateDB database cluster -to detect anomalies and follow usage trends, so you can react to -them properly and timely. +to detect anomalies, so you can react to them promptly. +Collecting statistics and following usage trends is also important +for proper capacity planning. CrateDB provides system information about the cluster as a whole, individual cluster nodes, and about the entities and resources it manages. @@ -72,12 +73,12 @@ and for ad hoc use. Below are a few popular and recommended options. :Prometheus: - The [Crate JMX HTTP Exporter] is a Prometheus exporter that consumes + The {ref}`Crate JMX HTTP Exporter ` is a Prometheus exporter that consumes metrics information from CrateDB's JMX collectors and exposes them via HTTP so they can be scraped by Prometheus, and, for example, subsequently displayed in Grafana, or processed into Alertmanager. - [Monitoring a CrateDB cluster with Prometheus and Grafana] illustrates + {ref}`monitoring-prometheus-grafana` illustrates a full setup for making CrateDB-specific metrics available to Prometheus. The tutorial uses the _Crate JMX HTTP Exporter_ for exposing telemetry information, the _Prometheus SQL Exporter_ for conducting system table @@ -104,5 +105,9 @@ and for ad hoc use. Below are a few popular and recommended options. real-time information about the cluster, its nodes, and their shards. -[Crate JMX HTTP Exporter]: https://github.com/crate/jmx_exporter -[Monitoring a CrateDB cluster with Prometheus and Grafana]: https://community.cratedb.com/t/monitoring-a-self-managed-cratedb-cluster-with-prometheus-and-grafana/1236 +:::{toctree} +:hidden: +Prometheus and Grafana +prometheus-jmx-exporter +prometheus-sql-exporter +::: diff --git a/docs/admin/monitoring/prometheus-grafana.md b/docs/admin/monitoring/prometheus-grafana.md new file mode 100644 index 00000000..819fb514 --- /dev/null +++ b/docs/admin/monitoring/prometheus-grafana.md @@ -0,0 +1,211 @@ +(monitoring-prometheus-grafana)= +# Monitoring a CrateDB cluster with Prometheus and Grafana + +:::{div} sd-text-muted +::: + +:::{rubric} Introduction +::: + +We recommend [^standalone] pairing two standard observability tools: +Use [Prometheus] to collect and store metrics, +and [Grafana] to build dashboards. + +This guide describes how to set up a Grafana dashboard that allows you +to check live and historical data around performance and capacity +metrics in your CrateDB cluster. It uses instructions suitable for +Debian or Ubuntu Linux, but can be adapted for other Linux distributions. + +[^standalone]: {ref}`Containerized ` and [CrateDB Cloud] setups differ. + This tutorial targets standalone and on‑premises installations. + +:::{rubric} Overview +::: + +For a CrateDB environment, you are interested in CrateDB-specific metrics, +such as the number of shards or number of failed queries, and OS metrics, +such as available disk space, memory usage, or CPU usage. +Based on Prometheus, the monitoring stack uses the following exporters +to fulfill those requirements. + +:Node Exporter: + + Exposes a wide variety of hardware and kernel-related metrics. + +:JMX Exporter: + + Consumes metrics information from CrateDB's + JMX collectors and exposes them via HTTP so they can be scraped by Prometheus. + +:SQL Exporter: + + Allows running arbitrary SQL + statements against a CrateDB cluster to retrieve additional + information from CrateDB's system tables. + +## Set up CrateDB cluster + +First things first, you will need a CrateDB cluster. +{ref}`Multi-node setup instructions ` provides +a quick walkthrough for Ubuntu Linux. + +## Set up Prometheus Exporters + +The Node Exporter and the JMX Exporter need to be installed on all +machines that are running CrateDB nodes. + +1. Install the Prometheus Node Exporter. + ```shell + apt install prometheus-node-exporter + ``` + +2. Install the {ref}`prometheus-jmx-exporter`. + +## Set up Prometheus + +You would typically run this on a machine that is not part of the +CrateDB cluster. +The {ref}`prometheus-sql-exporter` also does not need to be installed +on each machine. + +```shell +apt install prometheus prometheus-sql-exporter --no-install-recommends +``` + +For advanced configuration options, see {ref}`prometheus-auth` and +{ref}`prometheus-storage`. + +Now, configure Prometheus to scrape metrics from Node Exporters and +JMX Exporters on all CrateDB nodes, and also metrics from the SQL +Exporter. +```shell +nano /etc/prometheus/prometheus.yml +``` + +:Node Exporter: Port 9100 +:JMX Exporter: Port 8080 +:SQL Exporter: Port 9237 + +```yaml +- job_name: 'node' + static_configs: + - targets: ['ubuntuvm1:9100', 'ubuntuvm2:9100'] + +- job_name: 'cratedb_jmx' + static_configs: + - targets: ['ubuntuvm1:8080', 'ubuntuvm2:8080'] + +- job_name: 'sql_exporter' + static_configs: + - targets: ['localhost:9237'] +``` + +Restart the Prometheus daemon if it was already started. +```shell +systemctl restart prometheus +``` + +## Set up Grafana + +Install Grafana on the same machine where you installed Prometheus. +On a Debian or Ubuntu machine, run the following: +```shell +apt install --yes wget gpg +wget -q -O - https://packages.grafana.com/gpg.key | gpg --dearmor | tee /usr/share/keyrings/grafana.gpg >/dev/null +echo "deb [signed-by=/usr/share/keyrings/grafana.gpg] https://packages.grafana.com/oss/deb stable main" | tee /etc/apt/sources.list.d/grafana.list +apt update +apt install --yes grafana +``` +Then, start Grafana. +```shell +systemctl start grafana-server +``` +For other systems, see the [Grafana installation documentation][grafana-debian]. + +:::{rubric} Data source +::: + +Navigate to `http://:3000/` to access the Grafana login screen. +The default credentials are `admin`/`admin`; change the password immediately. +Navigate to "Add your first data source", then select "Prometheus" and set the +URL to `http://:9090/`. +If you configured basic authentication for Prometheus, this is where you +would need to enter the credentials. +Confirm using "Save & test". + +:::{rubric} Dashboard +::: + +An example dashboard based on the discussed setup is available for easy importing +from [Grafana » CrateDB Monitoring Dashboard]. +In your Grafana installation, on the left-hand side, hover over the “Dashboards” +icon and select “Import”. Specify the dashboard ID **17174** and load the dashboard. +On the next screen, finalize the setup by selecting the previously created +Prometheus data source. + +![CrateDB monitoring dashboard in Grafana|690x396](https://us1.discourse-cdn.com/flex020/uploads/crate/original/1X/0e01a3f0b8fc61ae97250fdeb2fe741f34ac7422.png){width=690px} + +## Alternative implementations + +Build your own dashboard or use an entirely different monitoring approach while +still covering similar metrics discussed in this article. +The list below is a good starting point for troubleshooting most operational issues. + +* CrateDB metrics (with example Prometheus queries based on the Crate JMX HTTP Exporter) + * Thread pools rejected: `sum(rate(crate_threadpools{property="rejected"}[5m])) by (name)` + * Thread pool queue size: `sum(crate_threadpools{property="queueSize"}) by (name)` + * Thread pools active: `sum(crate_threadpools{property="active"}) by (name)` + * Queries per second: `sum(rate(crate_query_total_count[5m])) by (query)` + * Query error rate: `sum(rate(crate_query_failed_count[5m])) by (query)` + * Average Query Duration over the last 5 minutes: `sum(rate(crate_query_sum_of_durations_millis[5m])) by (query) / sum(rate(crate_query_total_count[5m])) by (query)` + * Circuit breaker memory in use: `sum(crate_circuitbreakers{property="used"}) by (name)` + * Number of shards: `crate_node{name="shard_stats",property="total"}` + * Garbage Collector rates: `sum(rate(jvm_gc_collection_seconds_count[5m])) by (gc)` + * Thread pool rejected operations: `crate_threadpools{property="rejected"}` +* Operating system metrics + * CPU utilization + * Memory usage + * Open file descriptors + * Disk usage + * Disk read/write operations and throughput + * Received and transmitted network traffic + +## Appendix + +(prometheus-auth)= +:::{rubric} Prometheus authentication +::: + +By default, Prometheus binds to port 9090 without authentication. Prevent +auto-start during install (e.g., with `policy-rcd-declarative`), then +configure web auth using a YAML file. + +Create `/etc/prometheus/web.yml`: +```yaml +basic_auth_users: + admin: +``` + +Point Prometheus at it (e.g., `/etc/default/prometheus`): + +```shell +ARGS="--web.config.file=/etc/prometheus/web.yml --web.enable-lifecycle" +``` + +Restart Prometheus after setting ownership and 0640 permissions on `web.yml`. + +(prometheus-storage)= +:::{rubric} CrateDB as Prometheus storage +::: + +For a large deployment where you also use Prometheus to monitor other systems, +you may also want to use a CrateDB cluster as the storage for all Prometheus +metrics. The {ref}`CrateDB Prometheus Adapter ` achieves that. + + +[CrateDB Cloud]: https://cratedb.com/products/cratedb-cloud +[Grafana]: https://grafana.com/ +[grafana-debian]: https://grafana.com/docs/grafana/latest/setup-grafana/installation/debian/ +[Grafana » CrateDB Monitoring Dashboard]: https://grafana.com/grafana/dashboards/17174-cratedb-monitoring/ +[Prometheus]: https://prometheus.io/ +[Prometheus Node Exporter]: https://prometheus.io/docs/guides/node-exporter/ diff --git a/docs/admin/monitoring/prometheus-jmx-exporter.md b/docs/admin/monitoring/prometheus-jmx-exporter.md new file mode 100644 index 00000000..2fb83a3c --- /dev/null +++ b/docs/admin/monitoring/prometheus-jmx-exporter.md @@ -0,0 +1,35 @@ +(prometheus-jmx-exporter)= + +# Prometheus JMX Exporter + +The [Crate JMX HTTP Exporter] is a Prometheus exporter that consumes metrics +information from CrateDB's JMX collectors and exposes them via HTTP so they can +be scraped by Prometheus, and, for example, subsequently displayed in Grafana, +or processed into Alertmanager. + +:::{rubric} Setup +::: + +This is very simple, on each node run the following: + +```shell +cd /usr/share/crate/lib +wget https://repo1.maven.org/maven2/io/crate/crate-jmx-exporter/1.2.0/crate-jmx-exporter-1.2.0.jar +nano /etc/default/crate +``` + +then uncomment the `CRATE_JAVA_OPTS` line and change its value to: + +```shell +# Append to existing options (preserve other flags). +CRATE_JAVA_OPTS="${CRATE_JAVA_OPTS:-} -javaagent:/usr/share/crate/lib/crate-jmx-exporter-1.2.0.jar=8080" +``` + +and restart the crate daemon: + +```bash +systemctl restart crate +``` + + +[Crate JMX HTTP Exporter]: https://github.com/crate/jmx_exporter diff --git a/docs/admin/monitoring/prometheus-sql-exporter.md b/docs/admin/monitoring/prometheus-sql-exporter.md new file mode 100644 index 00000000..3d3ab082 --- /dev/null +++ b/docs/admin/monitoring/prometheus-sql-exporter.md @@ -0,0 +1,107 @@ +(prometheus-sql-exporter)= + +# Prometheus SQL Exporter + +The SQL Exporter allows running arbitrary SQL statements against a CrateDB +cluster to retrieve additional information. As the cluster contains information +from each node, we do not need to install the SQL Exporter on every node. +Instead, we install it centrally on the same machine that also hosts Prometheus. + +Please note that it is not the same to set up a data source in Grafana pointing +to CrateDB to display the output from queries in real-time as to use Prometheus +to collect these values over time. + +Installing the package is straight-forward: + +```shell +apt install prometheus-sql-exporter +``` + +For the SQL exporter to connect to the cluster, we need to create a new user +`sql_exporter`. We grant the user reading access to the `sys` schema. Run the +below commands on any CrateDB node: + +```shell +curl -H 'Content-Type: application/json' -X POST 'http://localhost:4200/_sql' -d '{"stmt":"CREATE USER sql_exporter WITH (password = '\''insert_password'\'');"}' +curl -H 'Content-Type: application/json' -X POST 'http://localhost:4200/_sql' -d '{"stmt":"GRANT DQL ON SCHEMA sys TO sql_exporter;"}' +``` + +We then create a configuration file in `/etc/prometheus-sql-exporter.yml` with a +sample query that retrieves the number of shards per node: + +```yaml +jobs: +- name: "global" + interval: '5m' + connections: ['postgres://sql_exporter:insert_password@ubuntuvm1:5433?sslmode=disable'] + queries: + - name: "shard_distribution" + help: "Number of shards per node" + labels: ["node_name"] + values: ["shards"] + query: | + SELECT node['name'] AS node_name, COUNT(*) AS shards + FROM sys.shards + GROUP BY 1; + allow_zero_rows: true + + - name: "heap_usage" + help: "Used heap space per node" + labels: ["node_name"] + values: ["heap_used"] + query: | + SELECT name AS node_name, heap['used'] / heap['max']::DOUBLE AS heap_used + FROM sys.nodes; + + - name: "global_translog" + help: "Global translog statistics" + values: ["translog_uncommitted_size"] + query: | + SELECT COALESCE(SUM(translog_stats['uncommitted_size']), 0) AS translog_uncommitted_size + FROM sys.shards; + + - name: "checkpoints" + help: "Maximum global/local checkpoint delta" + values: ["max_checkpoint_delta"] + query: | + SELECT COALESCE(MAX(seq_no_stats['local_checkpoint'] - seq_no_stats['global_checkpoint']), 0) AS max_checkpoint_delta + FROM sys.shards; + + - name: "shard_allocation_issues" + help: "Shard allocation issues" + labels: ["shard_type"] + values: ["shards"] + query: | + SELECT IF(s.primary = TRUE, 'primary', 'replica') AS shard_type, COALESCE(shards, 0) AS shards + FROM UNNEST([true, false]) s(primary) + LEFT JOIN ( + SELECT primary, COUNT(*) AS shards + FROM sys.allocations + WHERE current_state <> 'STARTED' + GROUP BY 1 + ) a ON s.primary = a.primary; +``` + +*Please note: There exist two implementations of the SQL Exporter: +[burningalchemist/sql_exporter](https://github.com/burningalchemist/sql_exporter) +and [justwatchcom/sql_exporter](https://github.com/justwatchcom/sql_exporter). +They don't share the same configuration options. Our example is based on the +implementation that is shipped with the Ubuntu package, which is +`justwatchcom/sql_exporter.*`. + +To apply the new configuration, we restart the service: + +```shell +systemctl restart prometheus-sql-exporter +``` + +The SQL Exporter can also be used to monitor any business metrics as well, but +be careful with regularly running expensive queries. Below are two more advanced +monitoring queries of CrateDB that may be useful: + +```sql +/* Time since the last successful snapshot (backup) */ +SELECT (NOW() - MAX(started)) / 60000 AS MinutesSinceLastSuccessfulSnapshot +FROM sys.snapshots +WHERE "state" = 'SUCCESS'; +``` diff --git a/docs/install/multi-node.md b/docs/install/multi-node.md index ca2b41b7..cc27a4b6 100644 --- a/docs/install/multi-node.md +++ b/docs/install/multi-node.md @@ -326,11 +326,80 @@ sh$ bin/crate \ [Metadata configuration settings] ::: +(multi-node-setup-example)= -## Expand cluster +## Example -Learn how to add more nodes to an existing cluster at {ref}`scale-expand`. +This walkthrough illustrates how to set up a multi-node CrateDB cluster +on two Ubuntu machines. +First, prepare CrateDB for a production setup by explicitly configuring +its JVM heap size to satisfy bootstrap checks. +```shell +nano /etc/default/crate +``` +```ini +CRATE_HEAP_SIZE=4G +``` + +Then, configure the cluster setup using a YAML file. +```shell +mkdir /etc/crate +nano /etc/crate/crate.yml +``` +```yaml +# Tell CrateDB to respond to requests both from localhost and the local network. +network.host: _local_,_site_ + +# List all the machines that make up our cluster. +# For production use, we recommend to use at least three nodes so that a quorum +# can be established in case of network partition to avoid split-brain scenarios. +discovery.seed_hosts: + - ubuntuvm1:4300 + - ubuntuvm2:4300 + +# List the nodes that are eligible to act as master nodes during bootstrap. +cluster.initial_master_nodes: + - ubuntuvm1 + - ubuntuvm2 + +# Require both nodes to be available for the cluster to operate. +# With more nodes, we could have set `recover_after_data_nodes` +# to a value smaller than the total number of nodes. +gateway.recover_after_data_nodes: 2 +gateway.expected_data_nodes: 2 + +# Configure that the `crate` superuser will be trusted to authenticate without +# password for local connections, but connections from other machines will +# require credentials. +auth.host_based.enabled: true +auth: + host_based: + config: + 0: + user: crate + address: _local_ + method: trust + 99: + method: password +``` + +Install CrateDB from the Debian/Ubuntu package repository. +```bash +apt update +apt install --yes gpg lsb-release wget +wget -O- https://cdn.crate.io/downloads/debian/DEB-GPG-KEY-crate | gpg --dearmor | tee /usr/share/keyrings/crate.gpg >/dev/null +echo "deb [signed-by=/usr/share/keyrings/crate.gpg] https://cdn.crate.io/downloads/debian/stable/ $(lsb_release -cs) main" | tee /etc/apt/sources.list.d/crate.list +apt update +apt install crate -o Dpkg::Options::="--force-confold" +``` +Use `force-confold` to keep the configuration files created earlier. +Otherwise, they would be overwritten. + +Repeat the above steps on the other node and start CrateDB on both machines. +```bash +systemctl start crate +``` (multi-node-other)= @@ -439,6 +508,10 @@ If you are using CrateDB 3.x or below, to configure every node with a list of seed nodes, you can use the [discovery.zen.ping.unicast.hosts] setting instead of {ref}`crate-reference:discovery.seed_hosts`. +## Related sections + +{ref}`scale-expand` describes how to add more nodes to an existing cluster. + [127.0.0.1:4200]: http://127.0.0.1:4200/ [127.0.0.1:4201]: http://127.0.0.1:4201/ diff --git a/docs/integrate/grafana/index.md b/docs/integrate/grafana/index.md index 92f058b8..e5e7c589 100644 --- a/docs/integrate/grafana/index.md +++ b/docs/integrate/grafana/index.md @@ -51,6 +51,18 @@ Connecting to a CrateDB cluster uses the Grafana PostgreSQL data source adapter. :::: +:::{rubric} See also +::: + +::::{grid} 2 + +:::{grid-item-card} Tutorial: Monitoring CrateDB with Prometheus and Grafana +:link: monitoring-prometheus-grafana +:link-type: ref +Production-grade monitoring and graphing of CrateDB metrics. +::: + +:::: :::{toctree} :maxdepth: 1 diff --git a/docs/integrate/prometheus/index.md b/docs/integrate/prometheus/index.md index 5df98929..045c1d94 100644 --- a/docs/integrate/prometheus/index.md +++ b/docs/integrate/prometheus/index.md @@ -93,6 +93,20 @@ usage guide. +:::{rubric} See also +::: + +::::{grid} 2 + +:::{grid-item-card} Tutorial: Monitoring CrateDB with Prometheus and Grafana +:link: monitoring-prometheus-grafana +:link-type: ref +Production-grade monitoring and graphing of CrateDB metrics. +::: + +:::: + + ```{seealso} [CrateDB and Prometheus] ```