Skip to content

localmetrics: add local Grafana timeseries tooling#50702

Merged
craig[bot] merged 2 commits intocockroachdb:masterfrom
tbg:poc/grafana
Jun 30, 2020
Merged

localmetrics: add local Grafana timeseries tooling#50702
craig[bot] merged 2 commits intocockroachdb:masterfrom
tbg:poc/grafana

Conversation

@tbg
Copy link
Copy Markdown
Member

@tbg tbg commented Jun 26, 2020

Add a docker-compose setup that starts a local Grafana backed by a local
Postgres along with a helper that can import timeseries data into the
Postgres instance which the Grafana instance is configured to display.

Consult scripts/localmetrics/README.md for a quickstart.

This isn't a valuable debug tool just yet, but with a bit of elbow
grease, I believe that it will become an invaluable tool to avoid
the many back-and-forth round-trips we have these days with customers
to exchange screenshots of the Admin UI.

To make it truly useful, we need

  1. timeseries in debug.zip
  2. auto-generate dashboards from ./pkg/ts/catalog.

Both are totally doable, and even without 2) there's already some
utility as it's easy to make ad-hoc panels in Grafana thanks to the
built-in query builder.

Finally, here's a screenshot of the one panel included here right now,
the rate of DistSender batches, taken from sample.csv.

image

Release note: None

@tbg tbg requested review from lunevalex and petermattis June 26, 2020 16:13
@tbg tbg requested a review from a team as a code owner June 26, 2020 16:13
@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

Copy link
Copy Markdown
Collaborator

@petermattis petermattis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

This is neat!

sample.csv is pretty large. Is it it worth including?

Reviewed 3 of 3 files at r1.
Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @lunevalex and @petermattis)

@tbg
Copy link
Copy Markdown
Member Author

tbg commented Jun 27, 2020

sample.csv is pretty large. Is it it worth including?

Don't think it would've mattered, but I moved it to a gist instead, so it's gone now.

TFTR

bors r=petermattis

@craig
Copy link
Copy Markdown
Contributor

craig Bot commented Jun 27, 2020

Build failed

@tbg
Copy link
Copy Markdown
Member Author

tbg commented Jun 27, 2020

bors r=petermattis

Lint

@craig
Copy link
Copy Markdown
Contributor

craig Bot commented Jun 27, 2020

Build failed

@tbg
Copy link
Copy Markdown
Member Author

tbg commented Jun 27, 2020

lint fails with:

[Ensure generated code is up-to-date] pkg/sql/colexec/overloads_test_utils.eg.go:11:2: can’t find import: “github.com/cockroachdb/cockroach/pkg/server/telemetry”

Probably @jordanlewis knows what to make of that, but it doesn't seem to be this PR's fault.

@tbg
Copy link
Copy Markdown
Member Author

tbg commented Jun 27, 2020

bors r=petermattis

master is probably fixed now.

craig Bot pushed a commit that referenced this pull request Jun 27, 2020
50702: localmetrics: add local Grafana timeseries tooling r=petermattis a=tbg



Add a docker-compose setup that starts a local Grafana backed by a local
Postgres along with a helper that can import timeseries data into the
Postgres instance which the Grafana instance is configured to display.

Consult scripts/localmetrics/README.md for a quickstart.

This isn't a valuable debug tool just yet, but with a bit of elbow
grease, I believe that it will become an invaluable tool to avoid
the many back-and-forth round-trips we have these days with customers
to exchange screenshots of the Admin UI.

To make it truly useful, we need

1. [timeseries in debug.zip](#50432)
2. auto-generate dashboards from `./pkg/ts/catalog`.

Both are totally doable, and even without 2) there's already some
utility as it's easy to make ad-hoc panels in Grafana thanks to the
built-in query builder.

Finally, here's a screenshot of the one panel included here right now,
the rate of DistSender batches, taken from `sample.csv`.


![image](https://user-images.githubusercontent.com/5076964/85878083-78756580-b7d8-11ea-88b4-fb515ef26186.png)


Release note: None

Co-authored-by: Tobias Schottdorf <tobias.schottdorf@gmail.com>
@craig
Copy link
Copy Markdown
Contributor

craig Bot commented Jun 27, 2020

Build failed

Add a docker-compose setup that starts a local Grafana backed by a local
Postgres along with a helper that can import timeseries data into the
Postgres instance which the Grafana instance is configured to display.

Consult scripts/localmetrics/README.md for a quickstart.

This isn't a valuable debug tool just yet, but with a bit of elbow
grease, I believe that it will become an invaluable tool to avoid
the many back-and-forth round-trips we have these days with customers
to exchange screenshots of the Admin UI.

To make it truly useful, we need

1. [timeseries in debug.zip](cockroachdb#50432)
2. auto-generate dashboards from `./pkg/ts/catalog`.

Both are totally doable, and even without 2) there's already some
utility as it's easy to make ad-hoc panels in Grafana thanks to the
built-in query builder.

Release note: None
@tbg
Copy link
Copy Markdown
Member Author

tbg commented Jun 30, 2020

bors r=petermattis
lint

@craig
Copy link
Copy Markdown
Contributor

craig Bot commented Jun 30, 2020

Build succeeded

@craig craig Bot merged commit e4b3bb9 into cockroachdb:master Jun 30, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants