mgr/influx: Various time fixes #20494
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
A interval of 5 seconds (previous default) provides a high resolution, but also a very large number of entries in the Influx database. By setting the default 30 seconds the amount of entries is reduced to 1/6th of that and thus saves storage and other resources. If users want they can change the interval to anything they like. Signed-off-by: Wido den Hollander <wido@42on.com>
Without a retention policy it is up to the administrator to set a retention policy on the data. This can easily be forgotten resulting in a ever growing InfluxDB database. Users can change the retention policy if they want to afterwards, this is just the default policy set by the InfluxDB module when it initially creates the database. Signed-off-by: Wido den Hollander <wido@42on.com>
By fetching the current time once and storing it into a variable we save a lot of system calls. On large clusters this can be a lot of system calls. In addition we also make sure that all points gathered in the same loop/run have exactly the same timestamp. Otherwise there will be a difference in time between the items in InfluxDB which then causes problems when creating graphs with for example Grafana. Signed-off-by: Wido den Hollander <wido@42on.com>
|
This all looks fine, although I'm a bit concerned that this was painfully slow at 5s intervals on a 2000 OSD cluster -- suggests that the code to gather and send the report was taking multiple seconds to execute. |
|
@jcsp Well, Influx also has to accept the data and the JSON needed to be send. The thing is, I don't have this cluster at my disposal to test and debug everything on it, it's in production. However, having a 30 second interval means you have a 6th of the data compared to the 5s interval. People can still use a 5s interval, but I don't think it's wise to make it the default |
jcsp
approved these changes
Mar 26, 2018
|
(tested this locally) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixes three things:
The default interval of 5 seconds is a nice to have, but it breaks large systems. Collecting and sending all this information on a 2000 OSD cluster is killing for the Manager. We shouldn't set that by default.
In addition a optimization here is to only fetch the time once, store it in a variable and us it in that run. Otherwise values have different times, but we also perform a lot of syscalls for fetching the time.
The last fix is to set a default retention of 8 weeks in InfluxDB. Users can change this however they like, it will not be modified afterwards, but it's to prevent users setting this up and having a DB grow for ever and breaking at some point in the future.