Add fill(linear) to query language #7403

desa · 2016-10-03T18:34:00Z

Required for all non-trivial PRs

Rebased/mergable
Tests pass
CHANGELOG.md updated
Sign CLA (if not already signed)

Required only if applicable

You can erase any checkboxes below this note if they are not applicable to your Pull Request.

InfluxQL Spec updated
Provide example syntax
Update man page when modifying a command

This PR introduces fill(linear) #3273 which will fill empty group by intervals linearly between successive values.

There are two interesting cases.

1 missing point

Suppose that I wrote the following points into the database

cpu,host=A value=2 10s
cpu,host=A value=4 30s

Then the query SELECT mean(value) FROM cpu WHERE time >= '1970-01-01T00:00:00Z' AND time <= '1970-01-01T00:01:00Z' GROUP BY time(10s) fill(linear) should return

name: cpu
time    mean
----    ----
0
10  2
20  3
30  4
40
50

> 1 missing point

Suppose I insert the following points

cpu,host=A value=2 10s
cpu,host=A value=7 60s

Then the query SELECT mean(value) FROM cpu WHERE time >= '1970-01-01T00:00:00Z' AND time <= '1970-01-01T00:01:00Z' GROUP BY time(10s) fill(linear) should return

name: cpu
time    mean
----    ----
0
10  2
20  3
30  4
40  5
50  6
60  7

jwilder · 2016-10-03T19:01:12Z

@jsternberg

jsternberg · 2016-10-03T19:05:02Z

influxql/average.go

@@ -0,0 +1,20 @@
+package influxql
+
+func averageFloat(wt, lt, nt int64, prev, next float64) float64 {


Can you add a comment describing this and the variables?

Added comments and changed code format so things should be more clear now. Let me know if theres anything else I can add.

jsternberg · 2016-10-03T19:09:23Z

influxql/iterator.gen.go.tmpl

@@ -673,6 +673,23 @@ func (itr *{{$k.name}}FillIterator) Next() (*{{$k.Name}}Point, error) {
 		switch itr.opt.Fill {
 		case NullFill:
 			p.Nil = true
+		case AverageFill:


A less conventional way to do this, but probably better, is to just move AverageFill above NullFill and do this with the condition.

{{if or (eq $k.Name "Float") (eq $k.Name "Integer")}} ... code that you have ... {{else}} fallthrough {{end}}

That way you don't need averageString or averageBoolean.

should be addressed now

desa · 2016-10-03T22:15:27Z

When adding comments to this file, it became clear that this should be called fill(linear) not fill(average) since what is being done here is really linear interpolation.

jsternberg · 2016-10-03T22:17:17Z

Haha, that's why I asked. It looked like it was linear interpolation and I was confused because the code was hard to understand. Now, I don't expect math code to be easy to understand, but I did want a comment so I don't have to try reading math 😄

desa · 2016-10-03T22:21:39Z

:)

Let me know if theres anything that I missed.

jsternberg · 2016-10-03T22:27:45Z

influxql/linear.go

+// and returns the value of the point on the line with time windowTime
+// y = mx + b
+func linearInteger(windowTime, previousTime, nextTime int64, previousValue, nextValue int64) int64 {
+	m := (nextValue - previousValue) / (nextTime - previousTime) // the slope of the line


It seems to me like this won't give the most accurate answer. Maybe we should use a float as the slope and then round after we get the result instead of making this into an integer. For example, take the following values:

previousTime: 0 previousValue: 1 nextTime: 10 nextValue: 5 windowTime: 5

Now I would say that the value at 5 should be 3 here because the value halfway in between 1 and 5 is 3. But if I perform the math here, it seems I get a slope of zero and so I get a value of 1. If I use floats, I get the value 3.

yup. I'll fix that now.

jsternberg · 2016-10-03T22:28:14Z

influxql/select_test.go

@@ -1133,6 +1133,62 @@ func TestSelect_Fill_Previous_Float(t *testing.T) {
 	}
 }

+// Ensure a SELECT query with a fill(linear) statement can be executed.
+func TestSelect_Fill_Linear_Float_One(t *testing.T) {


Can you include some tests with integers?

Good call. This actually caught an issue

desa · 2016-10-04T18:39:00Z

@jsternberg Should be good to go. Let me know if theres anything I missed.

jsternberg

Approved, but please squash the commits before merging.

jwilder

This needs a changelog entry under features.

Clean up template for fill average Change fill(average) to fill(linear) Update average to linear in infuxql spec Add Integer Tests and associated fixes Update CHANGELOG for fill(linear)

desa · 2016-10-04T23:58:27Z

@jwilder changelog updated.

desa force-pushed the md-fill-average branch from f96db74 to cd5fc82 Compare October 3, 2016 18:36

jsternberg suggested changes Oct 3, 2016

View reviewed changes

desa changed the title ~~Add fill(average) to query language~~ Add fill(linear) to query language Oct 3, 2016

jsternberg suggested changes Oct 3, 2016

View reviewed changes

jsternberg approved these changes Oct 4, 2016

View reviewed changes

jwilder suggested changes Oct 4, 2016

View reviewed changes

Add fill(linear) to query language

966e550

Clean up template for fill average Change fill(average) to fill(linear) Update average to linear in infuxql spec Add Integer Tests and associated fixes Update CHANGELOG for fill(linear)

desa force-pushed the md-fill-average branch from b346221 to 966e550 Compare October 4, 2016 21:27

jwilder approved these changes Oct 5, 2016

View reviewed changes

desa merged commit fc57c0f into master Oct 5, 2016

desa deleted the md-fill-average branch October 5, 2016 17:40

desa mentioned this pull request Oct 5, 2016

Add documentation for fill(linear) influxdata/docs.influxdata.com-ARCHIVE#764

Closed

timhallinflux added this to the 1.1.0 milestone Dec 19, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add fill(linear) to query language #7403

Add fill(linear) to query language #7403

desa commented Oct 3, 2016 •

edited

Loading

jwilder commented Oct 3, 2016

jsternberg Oct 3, 2016

desa Oct 3, 2016

jsternberg Oct 3, 2016

desa Oct 3, 2016

desa commented Oct 3, 2016

jsternberg commented Oct 3, 2016 •

edited

Loading

desa commented Oct 3, 2016

jsternberg Oct 3, 2016

desa Oct 3, 2016

jsternberg Oct 3, 2016

desa Oct 4, 2016

desa commented Oct 4, 2016

jsternberg left a comment

jwilder left a comment

desa commented Oct 4, 2016

		@@ -0,0 +1,20 @@
		package influxql

		func averageFloat(wt, lt, nt int64, prev, next float64) float64 {

Add fill(linear) to query language #7403

Add fill(linear) to query language #7403

Conversation

desa commented Oct 3, 2016 • edited Loading

Required for all non-trivial PRs

Required only if applicable

1 missing point

> 1 missing point

jwilder commented Oct 3, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

desa commented Oct 3, 2016

jsternberg commented Oct 3, 2016 • edited Loading

desa commented Oct 3, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

desa commented Oct 4, 2016

jsternberg left a comment

Choose a reason for hiding this comment

jwilder left a comment

Choose a reason for hiding this comment

desa commented Oct 4, 2016

desa commented Oct 3, 2016 •

edited

Loading

jsternberg commented Oct 3, 2016 •

edited

Loading