Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Request for merge for change that ignores data points that utilize undefined tag values, tag keys, or metric names as opposed to throwing an exception (optional based on query string param) #243

Open
wants to merge 3 commits into from

2 participants

@davidkbainbridge

Overview

This change adds a query string parameter, 'silent'. If present then any query that contains a tag key, tag value, or metric name that has not been registered in TSDB is ignored (dropped) as opposed to throwing an exception. Thus an entire query will not fail if a member of it has one of these values that does not exist.

Justification

A query that specifies tag information or a metric name does not need to fail and should instead return no data as this is essentially the same as specifying a filter that does not match any targets and thus should be treated as that way.

This is very helpful when you are using OpenTSDB to collect data and at time of query you can not determine if a given metric or tag pair has been added (--auto-metric).

This is also helpful when a single query contains multiple data points some of which are valid (i.e. have valid tag / metric information) and some don't. Instead of the whole query failing, valid data is returned for those data points for which the filter matches targets.

Backward Compatibility

This capability is activated by the presence of a 'silent' option on the end of the query string, i.e. '&silent'. Thus if this option is not include the existing behavior (exception) will be the default.

davidkbainbr... and others added some commits
davidkbainbridge Initial checkin of code to ignore those parts of the query if tag inf…
…ormation or metric have not yet been created in TSDB
1a96abc
@davidkbainbridge davidkbainbridge add a query param option to use as a switch to ignore bad queries, wh…
…ere bad is defined as queries that contain tag value, tag keys, or metric names that have not been registered
24e5137
@davidkbainbridge davidkbainbridge referenced this pull request in zenoss/query
Merged

Review please feature/ignorebad #42

@manolama
Owner

Sorry for the delay. So the use case would be if a user executes something like "?start=1d-ago&m=sum:foo{host=web01|web02}" but "web02" has not been assigned a UID yet, instead of throwing the exception "no UID found for web02" we should just return the data for "host=web01"? That sounds reasonable as an optional config like you implemented. Would you mind changing the flag from "silent" to "ignore_missing" or "ignore_nsun" for "no such unique name"?

@davidkbainbridge
@manolama manolama referenced this pull request from a commit in manolama/opentsdb
@manolama manolama Add the tsd.query.skip_unresolved_tagvs command line option and code …
…that allows

queries to complete even if an explicitly declared tag value does not exist such
as a query "m=sum:sys.cpu.user{host=web01|web02}" where "web02" has not been
assigned a UID yet.
This solves the most common use case from #243. Metrics and tag names should
still throw an exception as those are assigned much less frequently than values.
Additional flags can be given to handle these situations.
fcc0ae8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Oct 4, 2013
  1. Initial checkin of code to ignore those parts of the query if tag inf…

    davidkbainbridge authored
    …ormation or metric have not yet been created in TSDB
  2. @davidkbainbridge

    add a query param option to use as a switch to ignore bad queries, wh…

    davidkbainbridge authored
    …ere bad is defined as queries that contain tag value, tag keys, or metric names that have not been registered
Commits on Nov 22, 2013
  1. @davidkbainbridge
This page is out of date. Refresh to see the latest.
Showing with 44 additions and 18 deletions.
  1. +44 −18 src/tsd/GraphHandler.java
View
62 src/tsd/GraphHandler.java
@@ -342,6 +342,7 @@ private String getGnuplotBasePath(final HttpQuery query) {
qs.remove("png");
qs.remove("json");
qs.remove("ascii");
+ qs.remove("ignore_unknown_tags");
return cachedir + Integer.toHexString(qs.hashCode());
}
@@ -850,30 +851,55 @@ private static void respondAsciiQuery(final HttpQuery query,
if (rate) {
i--; // Move to the next part.
}
- final Query tsdbquery = tsdb.newQuery();
+ Query tsdbquery = tsdb.newQuery();
try {
tsdbquery.setTimeSeries(metric, parsedtags, agg, rate);
} catch (NoSuchUniqueName e) {
- throw new BadRequestException(e.getMessage());
- }
- // downsampling function & interval.
- if (i > 0) {
- final int dash = parts[1].indexOf('-', 1); // 1st char can't be `-'.
- if (dash < 0) {
- throw new BadRequestException("Invalid downsampling specifier '"
- + parts[1] + "' in m=" + m);
+
+ // A query was specified with a tag key, tag value, or metric
+ // that does not exist in TSDB. Based on a query option the
+ // entire query can fail, or this metric query will be ignored
+ // and not results returned for it, as if a tag key, tag value
+ // or metric name does not exists there can be no matching
+ // data points.
+ if (!query.hasQueryStringParam("ignore_unknown_tags")) {
+ throw new BadRequestException(e.getMessage());
+ } else {
+ tsdbquery = null;
}
- Aggregator downsampler;
- try {
- downsampler = Aggregators.get(parts[1].substring(dash + 1));
- } catch (NoSuchElementException e) {
- throw new BadRequestException("No such downsampling function: "
- + parts[1].substring(dash + 1));
+ }
+
+ // Only continue if tsdbquery != null, because that indicates that
+ // the query is valid (tag key, tag values, and metrics exist)
+ if (tsdbquery != null) {
+ // downsampling function & interval.
+ if (i > 0) {
+ final int dash = parts[1].indexOf('-', 1); // 1st char can't be `-'.
+ if (dash < 0) {
+ throw new BadRequestException("Invalid downsampling specifier '"
+ + parts[1] + "' in m=" + m);
+ }
+ Aggregator downsampler;
+ try {
+ downsampler = Aggregators.get(parts[1].substring(dash + 1));
+ } catch (NoSuchElementException e) {
+ throw new BadRequestException("No such downsampling function: "
+ + parts[1].substring(dash + 1));
+ }
+ final int interval = parseDuration(parts[1].substring(0, dash));
+ tsdbquery.downsample(interval, downsampler);
}
- final int interval = parseDuration(parts[1].substring(0, dash));
- tsdbquery.downsample(interval, downsampler);
+ tsdbqueries[nqueries++] = tsdbquery;
}
- tsdbqueries[nqueries++] = tsdbquery;
+ }
+
+ // If the actual number of parsed queries is < the number of queries
+ // specified that means that some where invalid and the caller was OK with
+ // that so shink the result arrray
+ if (nqueries < ms.size()) {
+ final Query[] shrink = new Query[nqueries];
+ System.arraycopy(tsdbqueries, 0, shrink, 0, nqueries);
+ return shrink;
}
return tsdbqueries;
}
Something went wrong with that request. Please try again.