Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Grafana dashboards for local use #2107

Closed
Tracked by #2028
greg-szabo opened this issue Jan 24, 2024 · 0 comments · Fixed by #2108
Closed
Tracked by #2028

Grafana dashboards for local use #2107

greg-szabo opened this issue Jan 24, 2024 · 0 comments · Fixed by #2108
Labels
enhancement New feature or request needs-triage This issue/PR has not yet been triaged by the team.
Milestone

Comments

@greg-szabo
Copy link
Contributor

Feature Request

Docker Compose to start Granafa and Prometheus servers that monitor locally running nodes.

Summary

The Grafana dashboards in the qa-infra repository have some interesting graphs that we could use during local testing. We could also improve those dashboards easier if we have a locally running Grafana server.

Problem Definition

I have a hard time understanding the health of a node in some cases: how long does consensus take, what's the memory usage, etc. The Comet log file has a lot of information but we should leverage the Prometheus metrics during development better.

With this tool we'd get easier visibility into Prometheus metrics. It's an external tool, no Go code needs to change.

Proposal

  • Docker Compose file that starts up a Grafana and a Prometheus container.
  • They need to connect to each other.
  • Prometheus needs to be able to connect to nodes running locally on the host machine.
  • Pre-populated dashboards would be nice.
@greg-szabo greg-szabo added enhancement New feature or request needs-triage This issue/PR has not yet been triaged by the team. labels Jan 24, 2024
github-merge-queue bot pushed a commit that referenced this issue Jan 24, 2024
Closes #2107
<!--

Please add a reference to the issue that this PR addresses and indicate
which
files are most critical to review. If it fully addresses a particular
issue,
please include "Closes #XXX" (where "XXX" is the issue number).

If this PR is non-trivial/large/complex, please ensure that you have
either
created an issue that the team's had a chance to respond to, or had some
discussion with the team prior to submitting substantial pull requests.
The team
can be reached via GitHub Discussions or the Cosmos Network Discord
server in
the #cometbft channel. GitHub Discussions is preferred over Discord as
it
allows us to keep track of conversations topically.
https://github.com/cometbft/cometbft/discussions

If the work in this PR is not aligned with the team's current
priorities, please
be advised that it may take some time before it is merged - especially
if it has
not yet been discussed with the team.

See the project board for the team's current priorities:
https://github.com/orgs/cometbft/projects/1

-->
This PR introduces a new folder `test/monitoring` which contains a
Docker Compose file that creates a Grafana and a Prometheus server
locally for the developer. It connects up with a localnet installation.

The localnet Docker compose file (`compose.yml`) was extended to allow
Prometheus traffic from the container and documentation was updated to
reflect the changes.

Short way to execute:
```
make build-linux localnet-start monitoring-start
```
---

The JSON files are pre-populated Grafana dashboards. Two of them are
from the qa-infra repository, one is part of host monitoring with
"node-explorer" and one is a new one created for storage experiments
that are currently ongoing.

#### PR checklist

- [ ] Tests written/updated - no Go code change
- [X] Changelog entry added in `.changelog` (we use
[unclog](https://github.com/informalsystems/unclog) to manage our
changelog)
- [X] Updated relevant documentation (`docs/` or `spec/`) and code
comments - test docs were updated
- [X] Title follows the [Conventional
Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec

---------

Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com>
mergify bot pushed a commit that referenced this issue Jan 24, 2024
Closes #2107
<!--

Please add a reference to the issue that this PR addresses and indicate
which
files are most critical to review. If it fully addresses a particular
issue,
please include "Closes #XXX" (where "XXX" is the issue number).

If this PR is non-trivial/large/complex, please ensure that you have
either
created an issue that the team's had a chance to respond to, or had some
discussion with the team prior to submitting substantial pull requests.
The team
can be reached via GitHub Discussions or the Cosmos Network Discord
server in
the #cometbft channel. GitHub Discussions is preferred over Discord as
it
allows us to keep track of conversations topically.
https://github.com/cometbft/cometbft/discussions

If the work in this PR is not aligned with the team's current
priorities, please
be advised that it may take some time before it is merged - especially
if it has
not yet been discussed with the team.

See the project board for the team's current priorities:
https://github.com/orgs/cometbft/projects/1

-->
This PR introduces a new folder `test/monitoring` which contains a
Docker Compose file that creates a Grafana and a Prometheus server
locally for the developer. It connects up with a localnet installation.

The localnet Docker compose file (`compose.yml`) was extended to allow
Prometheus traffic from the container and documentation was updated to
reflect the changes.

Short way to execute:
```
make build-linux localnet-start monitoring-start
```
---

The JSON files are pre-populated Grafana dashboards. Two of them are
from the qa-infra repository, one is part of host monitoring with
"node-explorer" and one is a new one created for storage experiments
that are currently ongoing.

#### PR checklist

- [ ] Tests written/updated - no Go code change
- [X] Changelog entry added in `.changelog` (we use
[unclog](https://github.com/informalsystems/unclog) to manage our
changelog)
- [X] Updated relevant documentation (`docs/` or `spec/`) and code
comments - test docs were updated
- [X] Title follows the [Conventional
Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec

---------

Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com>
(cherry picked from commit e8b4efa)
@adizere adizere added this to the 2024-Q1 milestone Jan 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request needs-triage This issue/PR has not yet been triaged by the team.
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

2 participants