Managing chunk sizes when the amount of metrics scales up over time seems tedious, and I can’t find any official guidance on it

[jedwards1211]

See also timescale/timescaledb#6720

Is it easy to find the information you need?

No

Are the instructions clear?

No

How could we improve the Timescale documentation site?

A lot of timescale examples will have a hypertable with columns like a metric id, time and value.

This is all well and good if the number of metric ids remains constant. But what about a use case where we need to add more metric ids as new users connect new devices to the system? The chunks will just get bigger and bigger as more and more devices are inserting data for more metric ids. Without any adjustment, the chunks will eventually become too big and the system will collapse.

Is there currently any official advice for how to deal with this, and is there any long term plan for making this more convenient/automatically managed in TimescaleDB? The best thing I can think of is to make my own jobs to periodically scale down the size of new chunks to be inversely proportional to the number of metrics as they increase. But I haven’t seen any official discussion of this, which is surprising, given how fundamental it is to the architecture of a whole category of systems.

To add to this, according to the official guidance about being able to fit all the active chunks in 25% of main memory, if we decide to create new hypertables in the future, ideally we should scale down all of the existing chunks. This seems…not super flexible, to say the least, and gives me hesitation about how easy it will be to adopt timescale long term. But on the other hand, I’m thinking…surely the architects of TimescaleDB have considered this, and have a good idea about how to deal with such cases?

[github-actions]

Thank you for the report. We welcome documentation contributions!

• For information about how to propose a change, see the contributing guide in our GitHub repository.
• For information on style and word usage, see the style guide