This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
Automatically chose retention policy based on time range #4262
Comments
torkelo
added
type/feature-request
datasource/InfluxDB
prio/low
|
So I started looking into this and did some prototyping for how this could work. I'm currently not focusing on the UI bits as that is dependent on how/where exactly this will be implemented. var rangediff = options.range.to.diff(options.range.from) / 1000;
options.targets.forEach(function(entry){
console.log(entry.policy);
//if (entry.policy === '_auto_') {
if (rangediff >= 10800) {
entry.policy = 'agg10m';
} else {
entry.policy = 'default';
}
//}
});This woks and forces all queries to use the 'agg10m' policy if the time range selected is >=3h and 'default' if it's not. |
|
@dennisjac , I am not sure if settings is best place to add this. This should go into the datasource definition. During the next week, I'll try to setup grafana development env and try to play with this. |
Can't retention policies be defined on a per-measurement (table) basis? |
|
@anlutro , I guess you could override them there but I'm not sure how many people define different retention policies for individual measurements. The reason I'm interested in this is because right now I'd pretty much have to create each dashboard three times for each of the three retention policies I'm using and then then manually select the right one based on the amount of time I want to display. This is extremely cumbersome at best and if you have more than a few dashboards (or hosts) completely unfeasible. This is what my retention policies and continuous queries currently look like: With this definition I get the appropriate "density" of data for low-, mid- and long-term views of the data and thanks to the back reference in the CQ's everything stays perfectly dynamic. With the auto-selection in Grafana the last piece would fit into place to make all of this work end-to-end. |
|
Yep, I'm in the same situation. I think I might want to store different measurements for different lengths of time, but I'd be fine with keeping it consistent on the database level. As an easier, temporary solution I was thinking about allowing template variables in the RP/measurement. |
|
I wasn't aware that you couldn't use template variables in the retention policy. |
|
btw here is an update version of the patch that I'm using right now. determineAutoPolicy(interval) {
// >48h = agg5m
// >1month = agg1h
var retentionPolicies = [
{interval: 0, policy: 'default'},
{interval: 172800, policy: 'agg5m'},
{interval: 2592000, policy: 'agg1h'}
];
var prevEntry = retentionPolicies[0];
for (var idx = 0; idx < retentionPolicies.length; idx++) {
if (retentionPolicies[idx].interval > interval) {
return prevEntry.policy;
}
prevEntry = retentionPolicies[idx];
}
return prevEntry.policy;
}and this bit at the beginning of the query() function: var rangediff = options.range.to.diff(options.range.from) / 1000;
options = _.cloneDeep(options);
for (var idx = 0; idx < options.targets.length; idx++) {
if (options.targets[idx].policy === '_auto_') {
options.targets[idx].policy = this.determineAutoPolicy(rangediff);
}
}I'm cloning the options variable because I need to modify it yet also need to preserve the original "auto" value so that when a new time range is selected the correct policy can be chosen again. |
|
@anlutro best check InfluxDB docs |
|
I think that could be solved more generically by having time range-dependent variable in template. I can already set up templated variable and just use say That way if that would be just generic variable it could be used for any data source, not only influx |
|
I just stumbled on this problem trying to setup grafana, was there any progress on this ? |
|
AFAIK the best you can do is to make a template variable with your retention periods, then use it everywhere |
|
but this is static, right ? the rentention policy needs to be chosen dependening on the range selected to really be useful. |
|
Yes. As I said, there is no other way to do it right now unless Grafana introduces conditional variables ( |
|
I would prefer not but I have no issue patching my grafana installation, however I could not find where exactly I should add my custom code. I have tried (I searched for $timeFilter as reference):
After that I rebuilt the assets with |
|
Sorry, writing JS wants me want to vomit so all we have is just a template field in each dashboard with "default" and "longterm" options for retention |
|
I not a big fan of javascript but the build process used is grafana is not helping either... I finally found where is the replace code for my use case, it was shadowed by the fact that $timeFilter replacement is done in two locations on the javascript side and once in the go side, I really don't like that design but it does not matter much, I now have what I need to patch this myself. |
|
I don't think implementing this at the datasource level would work. It doesn't seem so far-fetched to have multiple or different retention policies for individual measurements. For example you could have a RP+CQ storing the average of a measurement and a separate RP+CQ storing the maximum. In that case, a graph for max(foo) would want to use the aggregated-max RP. I thought of trying to implement this a while ago but I don't know Go at all. I now realize it might be possible with JS code alone, maybe I should give it a try. |
|
It would help to look at what people do with it. We have 2 , "default" that keeps full resolution for last month and "longerm" that has downsampled data (to 10 minute intervals) and keeps it for much longer but I dunno how common setup like that is |
|
In my case I use influxdb as datasource, in in I have multiple retention policies to reduce the number of datapoints for older records: Depending on the size of the range I want to show and/or the periodI want to show I want to use the best retention policy, If I want to show the last 2 weeks there is not much point usind data from raw since it will fetch way too much points, also I I want to show data from last month raw simply will not return any data. For now I just patched grafana to implement this logic in the influxdb datasource but this is not configurable, I also expose a few scopedVars to be able to create the correct queries. If you are interested I can show you my patch, it might help find a more generic solution. |
|
I'm also running a patched grafana for some time now. Influxdb with more than a single retention policy is infeasible to use without such a modification. With this patch against v5.2.1 you can scroll sideways or zoom and correct retention policy and interval (as configured) is chosen automatically. I'm posting it here as this issue is often referenced in other discussions. diff --git a/public/app/plugins/datasource/influxdb/datasource.ts b/public/app/plugins/datasource/influxdb/datasource.ts
index f971ac2..3f05ae5 100644
--- a/public/app/plugins/datasource/influxdb/datasource.ts
+++ b/public/app/plugins/datasource/influxdb/datasource.ts
@@ -16,6 +16,9 @@ export default class InfluxDatasource {
basicAuth: any;
withCredentials: any;
interval: any;
+ retentionPolicy: any;
+ retentionBefore: any;
+ retentionInterval: any;
supportAnnotations: boolean;
supportMetrics: boolean;
responseParser: any;
@@ -34,6 +37,9 @@ export default class InfluxDatasource {
this.basicAuth = instanceSettings.basicAuth;
this.withCredentials = instanceSettings.withCredentials;
this.interval = (instanceSettings.jsonData || {}).timeInterval;
+ this.retentionPolicy = (instanceSettings.jsonData || {}).retentionPolicy;
+ this.retentionBefore = (instanceSettings.jsonData || {}).retentionBefore;
+ this.retentionInterval = (instanceSettings.jsonData || {}).retentionInterval;
this.supportAnnotations = true;
this.supportMetrics = true;
this.responseParser = new ResponseParser();
@@ -47,6 +53,20 @@ export default class InfluxDatasource {
var queryModel;
var i, y;
+ var timestamp = new Date().getTime() / 1000;
+ var beforeTs = timestamp - options.range.from.unix();
+ var retentionPolicy = this.retentionPolicy ? this.retentionPolicy.split(";") : {};
+ var retentionBefore = this.retentionBefore ? this.retentionBefore.split(";") : {};
+ var retentionInterval = this.retentionInterval ? this.retentionInterval.split(";") : {};
+ var policy, interval;
+
+ for (y = 0; y < retentionBefore.length; y++) {
+ if (beforeTs > parseInt(retentionBefore[y])) {
+ policy = retentionPolicy[y];
+ interval = retentionInterval[y];
+ }
+ }
+
var allQueries = _.map(targets, target => {
if (target.hide) {
return '';
@@ -54,6 +74,11 @@ export default class InfluxDatasource {
queryTargets.push(target);
+ if (policy) {
+ scopedVars.autopolicy = { value: policy };
+ scopedVars.autointerval = { value: interval };
+ }
+
// backward compatibility
scopedVars.interval = scopedVars.__interval;
diff --git a/public/app/plugins/datasource/influxdb/influx_query.ts b/public/app/plugins/datasource/influxdb/influx_query.ts
index 2ef7417..f4395c3 100644
--- a/public/app/plugins/datasource/influxdb/influx_query.ts
+++ b/public/app/plugins/datasource/influxdb/influx_query.ts
@@ -23,6 +23,18 @@ export default class InfluxQuery {
target.groupBy = target.groupBy || [{ type: 'time', params: ['$__interval'] }, { type: 'fill', params: ['null'] }];
target.select = target.select || [[{ type: 'field', params: ['value'] }, { type: 'mean', params: [] }]];
+ if (target.policy === 'auto') {
+ target.policy = 'default';
+ if (typeof scopedVars !== 'undefined' && typeof scopedVars.autopolicy !== 'undefined') {
+ target.policy = scopedVars.autopolicy.value;
+ var interval = kbn.interval_to_seconds(scopedVars.autointerval.value);
+ if (interval > kbn.interval_to_seconds(scopedVars.__interval.value)) {
+ scopedVars.__interval = { value: kbn.secondsToHms(interval) };
+ scopedVars.interval = scopedVars.__interval;
+ }
+ }
+ }
+
this.updateProjection();
}
diff --git a/public/app/plugins/datasource/influxdb/partials/config.html b/public/app/plugins/datasource/influxdb/partials/config.html
index a70a1de..19d80a7 100644
--- a/public/app/plugins/datasource/influxdb/partials/config.html
+++ b/public/app/plugins/datasource/influxdb/partials/config.html
@@ -49,3 +49,23 @@
</div>
</div>
</div>
+
+<h4 class="page-heading">Retention Policy "auto"</h4>
+
+<div class="gf-form-group">
+ <div class="gf-form max-width-32">
+ <span class="gf-form-label width-11">Retention Policy</span>
+ <input type="text" class="gf-form-input width-20" ng-model="ctrl.current.jsonData.retentionPolicy" spellcheck='false' placeholder="sample5m;sample1h;sample1d"></input>
+ <i class="fa fa-question-circle" bs-tooltip="'Retention policies (delimited by semicolons)'" data-placement="right"></i>
+ </div>
+ <div class="gf-form max-width-32">
+ <span class="gf-form-label width-11">Before Timestamp</span>
+ <input type="text" class="gf-form-input width-20" ng-model="ctrl.current.jsonData.retentionBefore" spellcheck='false' placeholder="86400;2592000;31536000"></input>
+ <i class="fa fa-question-circle" bs-tooltip="'Before which timestamp to use above retention polices (in seconds, delimited by semicolons)'" data-placement="right"></i>
+ </div>
+ <div class="gf-form max-width-32">
+ <span class="gf-form-label width-11">Minimum Interval</span>
+ <input type="text" class="gf-form-input width-20" ng-model="ctrl.current.jsonData.retentionInterval" spellcheck='false' placeholder="30s;5m;1h"></input>
+ <i class="fa fa-question-circle" bs-tooltip="'Mimimum group by time interval to use with above policies (delimited by semicolons)'" data-placement="right"></i>
+ </div>
+</div>
diff --git a/public/app/plugins/datasource/influxdb/query_builder.ts b/public/app/plugins/datasource/influxdb/query_builder.ts
index 2b19aa8..84ba90b 100644
--- a/public/app/plugins/datasource/influxdb/query_builder.ts
+++ b/public/app/plugins/datasource/influxdb/query_builder.ts
@@ -52,7 +52,7 @@ export class InfluxQueryBuilder {
if (!measurement.match('^/.*/')) {
measurement = '"' + measurement + '"';
- if (policy && policy !== 'default') {
+ if (policy && policy !== 'default' && policy !== 'auto') {
policy = '"' + policy + '"';
measurement = policy + '.' + measurement;
}
@@ -69,7 +69,7 @@ export class InfluxQueryBuilder {
measurement = '"' + measurement + '"';
}
- if (policy && policy !== 'default') {
+ if (policy && policy !== 'default' && policy !== 'auto') {
policy = '"' + policy + '"';
measurement = policy + '.' + measurement;
}
|
|
In Grafana v6.0.2 I'm using an workaround which seems to work pretty well. The system I use has four retention policies defined: Because in Grafana there is no conditional variable, the idea was to have in InfluxDB a mapping between the time interval and the corresponding retention policy and to return the proper retention policy in Grafana via a "Query" variable. So, in InfluxDB I've created an unlimited duration retention policy where I've manually defined a measurement to keep this mapping: The mapping measurement is the following: Then, in Grafana I've defined the following variable: The |
|
@talek Awesome! No patching needed, works for me! |
|
@talek I can't figure out the last step you mentioned, namely prefixing all measurements with the $rp variable. Do you have same field names between retention policies? Or are they like field, d_field, m_field, etc. |
This works nicely, thanks @talek ! I found one minor bug however, your variable checks the time range while influxdb retention policies care about time since now, i.e. this works if Any ideas on how to improve this? I had the following in mind, but I can't compare ints and time: |
|
I'm very beginner with influxdb and grafana and tried your workaround:
which works very good but when restarting influxdb the retention policy "forever" is still there but all inserts are gone. Can someone point me in the right direction? |
I ran into this myself, and ended up with a slightly more complex variant that uses two variables: period and time. Here's the table of retention policies: The important bit is "period" - this is the upper bound for when to use this retention policy. If I'm looking at 24h or less, using "raw" data is fine. If I'm looking at 90d of data, use hourly data, etc. Next is time. This just matches the retention time. With this, I can use the following query:
Because influx will sort by time, this means it will pick the highest resolution (first aggregate) available that is still in range. This means if I view 90 days of data, I will use the ret_1h aggregate, but if I zoom in on a single day 60 days ago, I'll use ret_1m. The "Resolution" field is redundant, mainly for my personal sanity. I'm honestly not sure what the Correct solution to this bug is. The only downside with what I've got now is that I need to define this variable in every dashboard, but if there was a global templating variable thingamajing, that might be a better solution in general than trying to find a single one-size-fits all for picking retention policies. |
I've been digging into this one as well, as it's an important use case for us - I don't know why it's not a fundamental use case for which a method should be expected out of the box. It's completely natural to expect a dashboard view where the same query can be used regardless of time period selected, and have the tsdb responsible for rollups in a way that allows it. Score one for the venerable RRDtool and Graphite As you say,
I tried all sorts of combinations before various lightbulb moments came together: you can compare against the Create points whose time values are offsets from the epoch, of the ns values of the retention policies you've created: Shame about the verbose numerics - they can't be abbreviated using time unit specifiers [ed: Kristian used We need the name of the RP which is the smallest one satisfying That The Grafana query for the InfluxDB datasource looks like This feels like a disgraceful hack - but might be the only workaround at the moment. It does seem to work quite smoothly. In my tests, as I step out to a 1 hour view, I see fine granularity (10s) data from my |
|
I think you meant to write
and I had to set |
Not only that, but |
|
Hallo |
|
It seems to me that you have a negative timestamp. Remove the minus. |
|
I tried, but I have the same error: |
|
mmm does it work if you remove the following: ,idx=1? |
|
without idx=1 the error changes: |
|
this might not be the best place to fix this. Anyway, try copying this. If it does not work, I can't help you. sorry. |
|
Thanks @hackery for the easy fix problem! |
aocenas
added
prio/medium
aocenas
added this to Backlog features
in Observability (deprecated, use Observability Squad)
|
In additionally to @hackery's workaround I write a quite similar workaround in Flux for InfluxDB 2.x: I hope this helps someone! :-) |
|
I know we have workarounds detailed here but this seems like a basic requirement I keep running into. Is there any chance this is prioritized at some point? |
|
Same here, it would be nice to have this possibility without implementing these workarounds
|
|
This would be handy for us as well. We currently use a custom intermediate Grafana datasource which handles the rollup selection, but it seems like this should be a very common use case. |
Additionally update the grafana dashboard to automatically select retention period according to selected time range. grafana/grafana#4262 (comment)
|
@torkelo is this on the roadmap for Grafana? |
|
hi @R-Studio , as far as i know, this is not on the roadmap currently. |
|
(as the current approach is to handle feature requests in the discussions-section, we are moving this issue to the discussions) |
This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
and others
Hi,
I'm currently looking at how to make Grafana use the aggregated retention policies in InfluxDB.
Issue #3943 apparently adds a way to set the retention policy statically for a query but this only works if you have a hand full of queries. The moment you have more dashboards this is not really a viable option.
What I'd rather like to propose is the ability to add a table in the settings that defines a retention policy based on time ranges. For example you would define the values like this:
1 month => retention_one_value_per_day
1 week => retention_one_value_per_hour
1 day => retention_one_value_per_10s
All queries in all dashboards would then use the retention policy closest to the selected time range as default (which could still be overridden on a per-query basis by explicitly selecting a retention policy there?). If the user doesn't define this table then the default retention policy would always be used.
With this approach most people would be able to set up this table once and then all dashboards would automatically always use the right retention policy for their data which is I think what 99% of the people out there expect. Also for anyone who doesn't specify this value Grafana will behave just as before.
The text was updated successfully, but these errors were encountered: