Flexible RPC exporter for Prometheus #423
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,32 +1,102 @@ | ||
| # Monitoring | ||
|
|
||
| ## Prometheus | ||
|
|
||
| To monitor RPC return values over time, a Prometheus data exporter can be connected | ||
| to any Bitcoin Tank and configured to scrape any available RPC results. | ||
|
|
||
| The `bitcoin-exporter` image is defined in `resources/images/exporter` and | ||
| maintained in the BitcoinDevProject dockerhub organization. To add the exporter | ||
| in the Tank pod with Bitcoin Core add the `"exporter"` key to the node in the graphml file: | ||
|
|
||
| ```xml | ||
| <node id="0"> | ||
| <data key="version">27.0</data> | ||
| <data key="exporter">true</data> | ||
| </node> | ||
| ``` | ||
|
|
||
| The default metrics are defined in the `bitcoin-exporter` image: | ||
| - Block count | ||
| - Number of inbound peers | ||
| - Number of outbound peers | ||
| - Mempool size (# of TXs) | ||
|
|
||
| Metrics can be configured by setting a `"metrics"` key to the node in the graphml file. | ||
| The metrics value is a space-separated list of labels, RPC commands with arguments, and | ||
| JSON keys to resolve the desired data: | ||
|
|
||
| ``` | ||
| label=method(arguments)[JSON result key][...] | ||
| ``` | ||
|
|
||
| For example, the default metrics listed above are defined as: | ||
|
|
||
| ```xml | ||
| <node id="0"> | ||
| <data key="version">27.0</data> | ||
| <data key="exporter">true</data> | ||
| <data key="metrics">blocks=getblockcount() inbounds=getnetworkinfo()["connections_in"] outbounds=getnetworkinfo()["connections_in"] mempool_size=getmempoolinfo()["size"]</data> | ||
| </node> | ||
| ``` | ||
|
|
||
| The data can be retrieved from the Prometheus exporter on port `9332`, example: | ||
|
|
||
| ``` | ||
| # HELP blocks getblockcount() | ||
| # TYPE blocks gauge | ||
| blocks 704.0 | ||
| # HELP inbounds getnetworkinfo()["connections_in"] | ||
| # TYPE inbounds gauge | ||
| inbounds 0.0 | ||
| # HELP outbounds getnetworkinfo()["connections_in"] | ||
| # TYPE outbounds gauge | ||
| outbounds 0.0 | ||
| # HELP mempool_size getmempoolinfo()["size"] | ||
| # TYPE mempool_size gauge | ||
| mempool_size 0.0 | ||
| ``` | ||
|
|
||
| ## Grafana | ||
|
|
||
| Data from Prometheus exporters can be collected and fed into Grafana for a | ||
| web-based interface. | ||
|
|
||
| ### Install logging infrastructure | ||
|
|
||
| First make sure you have `helm` installed, then run the `install_logging` script: | ||
|
|
||
| ```bash | ||
| resources/scripts/install_logging.sh | ||
| ``` | ||
|
|
||
| To forward port `3000` and view the Grafana dashboard run the `connect_logging` script: | ||
|
|
||
| ```bash | ||
| resources/scripts/connect_logging.sh | ||
| ``` | ||
|
|
||
| The Grafana dashboard (and API) will be accessible without requiring authentication | ||
|
Contributor
There was a problem hiding this comment. I know it's probably because I'm too dumb, but where are the actual logs? When I open grafana on localhost:3000 I don't see any connected logs coming in?
This is with patch to default graph: diffdiff --git a/resources/graphs/default.graphml b/resources/graphs/default.graphml
index 153bd52..8c276a0 100644
--- a/resources/graphs/default.graphml
+++ b/resources/graphs/default.graphml
@@ -6,12 +6,14 @@
<key attr.name="exporter" attr.type="boolean" for="node" id="exporter"/>
<key attr.name="collect_logs" attr.type="boolean" for="node" id="collect_logs"/>
<key attr.name="image" attr.type="string" for="node" id="image"/>
+ <key attr.name="metrics" attr.type="string" for="node" id="metrics"/>
<graph edgedefault="directed">
<node id="0">
<data key="version">27.0</data>
<data key="bitcoin_config">-uacomment=w0</data>
<data key="exporter">true</data>
<data key="collect_logs">true</data>
+ <data key="metrics">blocks=getblockcount() inbounds=getnetworkinfo()["connections_in"] outbounds=getnetworkinfo()["connections_in"] mempool_size=getmempoolinfo()["size"]</data>
</node>
<node id="1">
<data key="version">27.0</data>
Contributor
There was a problem hiding this comment. Should I see running logging containers? This is all I see:
The script appeared to run successfully:
Contributor
There was a problem hiding this comment. kubectl does show preometheus running: ₿ kubectl --namespace warnet-logging get pods -l "release=prometheus"
NAME READY STATUS RESTARTS AGE
prometheus-kube-prometheus-operator-6c5998f7dc-hjvwx 1/1 Running 0 9m27s
prometheus-kube-state-metrics-688d66b5b8-8srsw 1/1 Running 0 9m27s
prometheus-prometheus-node-exporter-8tt7q 1/1 Running 0 9m27sBut I don't see any node exporters?
Contributor
There was a problem hiding this comment. server logs don't appear to contain any errors:
Contributor
There was a problem hiding this comment. I do see a single running exporter (I think):
Contributor
Author
There was a problem hiding this comment. The exporters are inside the tank pods, next to the bitcoin containers.
The stuff in warnet-logging is the grafana api server and the prometheus scraper that reads from the individual tank exporters. I know there is a "node exporter" pod in warnet-logging as well, I dunno what that is actually for and on my system, it never works anyway: As far as seeing something in Grafana right away you're right I didn't document that, I will push another commit today that hopefully makes a default dashboard easy to load
Collaborator
There was a problem hiding this comment. @willcl-ark logs are in Loki not Prometheus. There are no additional containers for Loki to get it's data as it collects it via k8s directly similar to how you can do |
||
| at http://localhost:3000 | ||
|
|
||
| ## Dashboards | ||
|
|
||
| To view the default metrics in the included default dashboard, upload the dashboard | ||
| JSON file to the Grafana server: | ||
|
|
||
| ``` | ||
| curl localhost:3000/api/dashboards/db \ | ||
| -H "Content-Type: application/json" \ | ||
| --data "{\"dashboard\": $(cat resources/configs/grafana/default_dashboard.json)}" | ||
| ``` | ||
|
|
||
| Note the URL in the reply from the server (example): | ||
|
|
||
| ``` | ||
| {"folderUid":"","id":2,"slug":"default-warnet-dashboard","status":"success","uid":"fdu0pda1z6a68b","url":"/d/fdu0pda1z6a68b/default-warnet-dashboard","version":1}( | ||
| ``` | ||
|
|
||
| Open the dashboard in your browser (example): | ||
|
|
||
| `http://localhost:3000/d/fdu0pda1z6a68b/default-warnet-dashboard` | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's add a
metricskey to both default graph and generated graphs fromwarcli graph createso that users can easily add this key without gettingmissing key(at the graph level) errors.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cool done