[0.9.3-nightly] regexs fails with empty tags

[gallir]

If some tags are missing/no used in some points they are no selected by regexes like the "All" used from grafana and similar.

Example, for

select * from serie where sparse_tag =~ /.*/

only those points what stored some value in sparse_tag will be retrieved. The same happens with expression like "sparse_tag = ''" but it's annoying with regexes because you cannot create filters (in grafana, for example, they are called "template variables") if the tag is not stored in every point in the measurement.

I don't know if this is a bug or a constraint, anyway it will be great if it worked as expected.

[beckettsean]

@gallir the regex as written only matches actual values, not the absence of a value. Does a query like this address your needs? I haven't directly tested it but I think it should work.

select * from serie where sparse_tag =~ /.*/ OR sparse_tag !~ /.*/

[gallir]

Nope, it does not work. Test with actual values (with a last week nightly build because 0.9.2 stable died with "select * ..."):

> select count(value) from data where collection='confirmation' and service =~ /.*/  OR sparse_tag !~ /.*/
name: data
----------
time            count
1970-01-01T00:00:00Z    297166

> select count(value) from data where collection='booking' and service =~ /.*/  OR sparse_tag !~ /.*/
> 

Gosh! when testing different options while I was writing this I found one that works, /^$/

> select count(value) from data where service !~ /^$/
name: data
----------
time            count
1970-01-01T00:00:00Z    516220

Unfortunately Grafana doesn't accept this regex :(

But it's strange, it selects also empty tags when the regex requires the opposite.

[jsternberg]

This may be really difficult for us to do. While we keep an index of what values correspond to which tag, we don't keep track of values that don't have a specific tag. I don't think there would be any practical way we could do that either as it would require us to know what tags are allowed to exist and will exist for every point ahead of time. The only way I can think of would be to iterate through all of the possible series, but I don't think that would be a good idea for performance.

Something we may be able to do is allow an empty string as a tag value. If we allowed that, then it would likely be possible to index the empty tag value and probably wouldn't even require the query engine to change to support this kind of functionality.

@pauldix what are your thoughts on this? This might also be something to consider if/when we redesign the line protocol like in #6037.

[pauldix]

If you just leave out the WHERE clause then you'll get everything. We wouldn't want to look at the regex to see if it should match against an empty tag.

One thing we may want to do is to make sure WHERE some_tag = '' or WHERE some_tag is null work properly

[jsternberg]

@pauldix the second part is what I was referring to. When a tag isn't specified when writing a metric, it's not indexed anywhere so figuring out WHERE some_tag = '' is difficult because we would have to iterate through all of the possible series.

We could make it possible by allowing tags to be the empty string so the user would write:

> insert cpu,host=server01,region= value=2

And then this would work:

> select value from cpu where host = 'server01' and region =~ /.*/

But anything that doesn't have a region would get ignored since it wasn't indexed when it was written. It would be difficult to implement WHERE some_tag is null because of this indexing. One benefit to this is we could differentiate between a tag that is nil and one that is the empty string pretty easily.

[pauldix]

No, they shouldn't be able to write tags with an empty string. If they're matching on the tag empty string in the query, it would be the same as pulling WHERE !~ /.*/ which should also work, no?

[jsternberg]

I think I might have a misunderstanding of what a missing tag counts as. For the series cpu,host=server01 should the region be null or ""? If it's null, is there and should there be a way to have a tag value that is the empty string? If it's "", does that mean every series has a near-infinite number of tags with the empty string?

There seems to be a bit of inconsistency in regards to how a missing tag key is treated. Some places use the empty string and some places treat it as null.