UI: Showing Block Size Stats #7233

outofrange · 2024-03-25T21:26:36Z

This PR adds details about block sizes to the blocks UI page.

Each source block row shows an aggregated sum of all chunks and indices of all blocks in this row, respecting the currently selected compaction level.

When selecting a block, the details view shows the following:

total size
index size & percentage of total
chunk size & percentage of total
size per day, based on block total size and duration

All sizes are humanized up to Pebibytes, and display the raw bytes value on mouse over via their title text.

As I wasn't sure about all compatibility, block stats are only calculated and taken into account if

block has .thanos.files array set
individual files have their size_bytes set

At the moment, only chunks and index are counted - meta.json might be present as well, but often hasn't a size assigned to it.
Other rel_path than index and chunks/* are ignored as well.

Fixes #7205

I added CHANGELOG entry for this change.

Changes

displaying size stats (if applicable) for selected block in details view
displaying aggregated block size per source row

Verification

added new tests
ran existing React tests
manual testing proxying a production Thanos instance of 1-2 TiB (~ 2-4 weeks) from about 10 sources

outofrange · 2024-03-25T21:28:19Z

pkg/ui/react-app/src/thanos/pages/blocks/BlockDetails.tsx

+          {sizeStats && (
+            <>
+              <div data-testid="total-size">
+                <b>Total size:</b>&nbsp;


Those non breaking spaces are mostly because Prettier (?) trims my trailing spaces at the end of the line :(

A block can have a list of contained files set in `.thanos.files`. If the `files` array is set, all referenced files with `size_bytes` set are counted: - sum of all `chunk/*` file sizes - size of index file - total size (sum of both) Shows statistics about the selected block in the block details view: - Total size of block - Size of index (and percentage of total) - Size of all chunks (and percentage of total) - Daily growth, based on total size and block duration Output is humanized up to Pebibytes and fixed to two decimal places; raw bytes are accessible through mouse over / title text. Signed-off-by: Markus Möslinger <markus.moeslinger@socra.dev>

Added total size of all blocks from a source to the row title, beneath the source name. The shown total size is humanized up to pebibytes and fixed to two decimal places; raw bytes value is accessible through mouse over / title text. The shown value will refresh with selected compaction levels, but doesn't take block filter into account. I thought about showing daily growth as well, but just summing all milliseconds of all blocks doesn't work with overlapping blocks / multiple resolutions. Signed-off-by: Markus Möslinger <markus.moeslinger@socra.dev>

Signed-off-by: Markus Möslinger <markus.moeslinger@socra.dev>

outofrange · 2024-03-25T22:14:31Z

pkg/ui/react-app/src/thanos/pages/blocks/BlockDetails.tsx

+                </span>
+              </div>
+              <div data-testid="daily-bytes">
+                <b>Daily:</b>&nbsp;


Not too happy about Daily as label, but it's shorter than something like Total per day.

Automatic code formatting duplicated some comments near import statements. Signed-off-by: Markus Möslinger <markus.moeslinger@socra.dev>

yeya24

I love this change. Great work.

GiedriusS

Very nice! Thank you 🙇

outofrange · 2024-03-26T07:27:36Z

That was fast, thank you to both of you too! :)

douglascamata · 2024-03-26T08:50:15Z

@outofrange this is awesome stuff, as others said... thank you!

If you want another idea for a contribution, here's one: each block stream (indexed from 1 to n) has a set of external labels. Currently we have to click on any block inside a stream to see them and this is annoying when you are looking for an specific stream. It would be super helpful to be able to see the external labels there together with the block stream's index and size (that you just added), so that you could just use your browser "find text" feature to find the block stream you want. 💡

outofrange · 2024-03-26T12:55:20Z

@outofrange this is awesome stuff, as others said... thank you!

If you want another idea for a contribution, here's one: each block stream (indexed from 1 to n) has a set of external labels. Currently we have to click on any block inside a stream to see them and this is annoying when you are looking for an specific stream. It would be super helpful to be able to see the external labels there together with the block stream's index and size (that you just added), so that you could just use your browser "find text" feature to find the block stream you want. 💡

I created this ticket for further discussions about this: #7237

* feat(ui): added BlockSizeStats calculation to blocks page A block can have a list of contained files set in `.thanos.files`. If the `files` array is set, all referenced files with `size_bytes` set are counted: - sum of all `chunk/*` file sizes - size of index file - total size (sum of both) Shows statistics about the selected block in the block details view: - Total size of block - Size of index (and percentage of total) - Size of all chunks (and percentage of total) - Daily growth, based on total size and block duration Output is humanized up to Pebibytes and fixed to two decimal places; raw bytes are accessible through mouse over / title text. Signed-off-by: Markus Möslinger <markus.moeslinger@socra.dev> * feat(ui): added aggregated BlockSizeStats to blocks row title Added total size of all blocks from a source to the row title, beneath the source name. The shown total size is humanized up to pebibytes and fixed to two decimal places; raw bytes value is accessible through mouse over / title text. The shown value will refresh with selected compaction levels, but doesn't take block filter into account. I thought about showing daily growth as well, but just summing all milliseconds of all blocks doesn't work with overlapping blocks / multiple resolutions. Signed-off-by: Markus Möslinger <markus.moeslinger@socra.dev> * chore(docs): added UI block size PR to CHANGELOG.md Signed-off-by: Markus Möslinger <markus.moeslinger@socra.dev> * chore(ui): removed comments Automatic code formatting duplicated some comments near import statements. Signed-off-by: Markus Möslinger <markus.moeslinger@socra.dev> --------- Signed-off-by: Markus Möslinger <markus.moeslinger@socra.dev>

pull-request-size bot added the size/L label Mar 25, 2024

outofrange commented Mar 25, 2024

View reviewed changes

outofrange marked this pull request as ready for review March 25, 2024 21:39

outofrange added 3 commits March 25, 2024 23:05

chore(docs): added UI block size PR to CHANGELOG.md

ce31f00

Signed-off-by: Markus Möslinger <markus.moeslinger@socra.dev>

outofrange force-pushed the feature/7205-blocks-page-size-stats branch from cc14368 to ce31f00 Compare March 25, 2024 22:05

outofrange commented Mar 25, 2024

View reviewed changes

chore(ui): removed comments

ca7ed42

Automatic code formatting duplicated some comments near import statements. Signed-off-by: Markus Möslinger <markus.moeslinger@socra.dev>

yeya24 approved these changes Mar 26, 2024

View reviewed changes

GiedriusS approved these changes Mar 26, 2024

View reviewed changes

GiedriusS merged commit 6c613fc into thanos-io:main Mar 26, 2024
20 checks passed

outofrange mentioned this pull request Mar 26, 2024

Blocks UI: Find sources for external labels #7237

Open

douglascamata mentioned this pull request Mar 26, 2024

Block Viewer: Show size of the blocks in the UI #3221

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

UI: Showing Block Size Stats #7233

UI: Showing Block Size Stats #7233

outofrange commented Mar 25, 2024 •

edited

Loading

outofrange Mar 25, 2024

outofrange Mar 25, 2024

yeya24 left a comment

GiedriusS left a comment

outofrange commented Mar 26, 2024

douglascamata commented Mar 26, 2024 •

edited

Loading

outofrange commented Mar 26, 2024

UI: Showing Block Size Stats #7233

UI: Showing Block Size Stats #7233

Conversation

outofrange commented Mar 25, 2024 • edited Loading

Changes

Verification

outofrange Mar 25, 2024

Choose a reason for hiding this comment

outofrange Mar 25, 2024

Choose a reason for hiding this comment

yeya24 left a comment

Choose a reason for hiding this comment

GiedriusS left a comment

Choose a reason for hiding this comment

outofrange commented Mar 26, 2024

douglascamata commented Mar 26, 2024 • edited Loading

outofrange commented Mar 26, 2024

outofrange commented Mar 25, 2024 •

edited

Loading

douglascamata commented Mar 26, 2024 •

edited

Loading