Skip to content
This repository has been archived by the owner on May 17, 2021. It is now read-only.

influxdb persistence: enable time resolution of 1 second #5933

Open
JsBergbau opened this issue Jan 20, 2020 · 1 comment
Open

influxdb persistence: enable time resolution of 1 second #5933

JsBergbau opened this issue Jan 20, 2020 · 1 comment

Comments

@JsBergbau
Copy link

In the influxdb persistence file a data point is stored at MilliSeconds precision. See InfluxDBPersistenceService.java Line 244

influxDB.write(dbName, TimeUnit.MILLISECONDS, serie);

This should be configurable to seconds level. Seconds compared to milliseconds enable better compression with the build in compression functions and thus saves space. Resolution to ms is hardly needed.

@JsBergbau
Copy link
Author

Some furhter information to get an impression how much space could be saved:
I've converted a series of about 276000 datapoints from nanoseconds resolution to seconds resolution. the /data folder of influxdb was 509 MB bevor I've started. About 12h later, when all compactions should have been run size shrank to 467 MB. That's saving 42 MB or about 8 % for this series.
I've counted all the datapoints. Currently there are about 73 million datapoints stored in this influxdb. So this table is only 0,3772 % of all datapoints and still 8 % disk space is saved.
Since most of the other measurements in this influxdb are from Openhab and Openhab uses ms as time resolution the expected saved space should be in between from nanoseconds to seconds, so about half.

I know disk space is cheap this time so you could argument thats not worth it. However influxdb has a serious problem on 32 bit systems. When the database reaches about 3.6 GB. "When the TSM files approach this limit, Influx fails with "Cannot allocate memory" errors." from influxdata/influxdb#10486 See also influxdata/influxdb#12362

Since a lot OpenHab installations are on a RaspberryPI with 32 Bit we'll get to this limit sooner or later. So we should use all possibilities to extend this limit. When changing from ns resolution to seconds resolution saves so much space we should implement this feature saving the timestamps only at precision level of seconds.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant