ESQL: extend BUCKET with spans. Turn it into a grouping function #107272

bpintea · 2024-04-09T14:04:40Z

This extends BUCKET function to accept a two-parameters-only
invocation: the first parameter remains as is, while the second is a
span. It can be a numeric (floating point) span, if the first argument
is numeric, or a date period or time duration, if the first argument is
a date.

Also, the function can now be invoked with the alias BIN.

Additionally, the function has been turned into a grouping-only function
and thus can only be used within a STATS command.

This renames the function AUTO_BUCKET to just BUCKET.

This extends `BUCKET` function to accept a two-parameters-only invocation: the first parameter remains as is, while the second is a span. It can be a numeric (floating point) span, if the first argument is numeric, or a date period or time duration, if the first argument is a date. Also, the function can now be invoked with the alias `BIN`.

github-actions · 2024-04-09T14:04:52Z

Documentation preview:

✨ Changed pages

elasticsearchmachine · 2024-04-09T14:05:04Z

Hi @bpintea, I've created a changelog YAML for you.

bpintea · 2024-04-10T14:03:30Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/io/stream/PlanNamedTypes.java

-        out.writeExpression(bucket.from());
-        out.writeExpression(bucket.to());
+        out.writeExpression(bucket.bucketsOrSpan());
+        out.writeOptionalExpression(bucket.from());


BUCKET is "newly" introduced, so these aren't posing a bwc-issue.

elasticsearchmachine · 2024-04-10T15:01:17Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

bpintea · 2024-04-10T15:04:04Z

.../esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/math/Bucket.java

+                double t = ((Number) to.fold()).doubleValue();
+                r = pickRounding(b, f, t);
+            } else {
+                r = ((Number) bucketsOrSpan.fold()).doubleValue();


A 0d will result in a NaN, but this isn't new and would address it subsequently.

alex-spies

Gave it a first round, will do another after lunch.

Looks good so far but I have a couple remarks.

.../esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/math/Bucket.java

x-pack/plugin/esql/qa/testFixtures/src/main/resources/ints.csv-spec

.../esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/math/Bucket.java

x-pack/plugin/esql/qa/testFixtures/src/main/resources/ints.csv-spec

alex-spies · 2024-04-11T08:44:37Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/date.csv-spec

+4              |2023-11-01T00:00:00.000Z
+3              |2025-10-01T00:00:00.000Z
+;
+


I think we should add test cases that use BIN, e.g. with queries that compare the output of BIN to that of BUCKET. Applies also to int.csv-spec.

I've added some BIN tests. Not really comparing to BUCKET (I'd leave the aliasing test aside), but they're copies of existing tests with s/BUCKET/BIN.

alex-spies · 2024-04-11T08:46:12Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/meta.csv-spec

@@ -229,6 +231,7 @@ asin          |Returns the {wikipedia}/Inverse_trigonometric_functions[arcsine]
 atan          |Returns the {wikipedia}/Inverse_trigonometric_functions[arctangent] of the input numeric expression as an angle, expressed in radians.
 atan2         |The {wikipedia}/Atan2[angle] between the positive x-axis and the ray from the origin to the point (x , y) in the Cartesian plane, expressed in radians.
 avg           |The average of a numeric field.
+bin           |Creates human-friendly buckets and returns a datetime value for each row that corresponds to the resulting bucket the row falls into.


Thought (out of scope): Hm, IMO it'd be better if this said "alias for bucket", but that'll require some plumbing to achieve.

Agreed, that'd be nice to have.

.../esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/math/Bucket.java

alex-spies · 2024-04-11T09:14:57Z

.../esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/math/Bucket.java

+                ? resolution.and(checkArgsCount(4))
+                    .and(() -> isStringOrDate(from, sourceText(), THIRD))
+                    .and(() -> isStringOrDate(to, sourceText(), FOURTH))


praise: that's neat with and allowing to use a supplier.

alex-spies

Alright, all done now. Close to LGTM; we should look into the tests for BIN though and double check the kibana docs, IMHO.

alex-spies · 2024-04-11T11:29:59Z

docs/reference/esql/functions/kibana/definition/bucket.json

-          "optional" : false,
+          "optional" : true,
          "description" : ""
        },
        {
          "name" : "to",
          "type" : "datetime",
-          "optional" : false,
+          "optional" : true,


I think this is not correct; in the integer case, neither from nor to are optional, no?

Applies below as well.

Yeh, not sure how to go about this file, it being generated: the optionality of the parameters can't be accurately specified for functions with more than one optional parameter (the two or four bit), or based on types of various non-optional parameters.

Oh, that's not ideal. That means we probably want to update our docs generation a bit. (out of scope, in this case)

CC @nik9000 , if you agree I'll open an issue so we don't forget.

.../esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/math/Bucket.java

alex-spies · 2024-04-11T11:42:47Z

.../src/test/java/org/elasticsearch/xpack/esql/expression/function/scalar/math/BucketTests.java

+        suppliers.add(new TestCaseSupplier(name, List.of(numberType, DataTypes.DOUBLE), () -> {
+            List<TestCaseSupplier.TypedData> args = new ArrayList<>();
+            args.add(new TestCaseSupplier.TypedData(number.get(), "field"));
+            args.add(new TestCaseSupplier.TypedData(50., DataTypes.DOUBLE, "span").forceLiteral());


Maybe we could randomize the span here.

It could be randomised, though the matcher (and the style of these tests) would then need to be updated. I've kept it as is for now, but can be convinced of otherwise. :)

Probably okay either way, I'll leave that to your discretion :)

x-pack/plugin/esql/qa/testFixtures/src/main/resources/bucket.csv-spec

alex-spies · 2024-04-11T15:26:36Z

.../src/test/java/org/elasticsearch/xpack/esql/expression/function/scalar/math/BucketTests.java

+        suppliers.add(new TestCaseSupplier(name, List.of(numberType, DataTypes.DOUBLE), () -> {
+            List<TestCaseSupplier.TypedData> args = new ArrayList<>();
+            args.add(new TestCaseSupplier.TypedData(number.get(), "field"));
+            args.add(new TestCaseSupplier.TypedData(50., DataTypes.DOUBLE, "span").forceLiteral());


Probably okay either way, I'll leave that to your discretion :)

alex-spies

LGTM, thanks @bpintea !

costin · 2024-04-12T00:07:23Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/meta.csv-spec

@@ -229,6 +231,7 @@ asin          |Returns the {wikipedia}/Inverse_trigonometric_functions[arcsine]
 atan          |Returns the {wikipedia}/Inverse_trigonometric_functions[arctangent] of the input numeric expression as an angle, expressed in radians.
 atan2         |The {wikipedia}/Atan2[angle] between the positive x-axis and the ray from the origin to the point (x , y) in the Cartesian plane, expressed in radians.
 avg           |The average of a numeric field.
+bin           |Creates human-friendly buckets and returns a datetime value for each row that corresponds to the resulting bucket the row falls into.


What about numeric bucketing?

Good point, this was due an update for a while now -- I've updated it.

costin · 2024-04-12T00:07:36Z

x-pack/plugin/ql/src/main/java/org/elasticsearch/xpack/ql/expression/Expression.java

+        public TypeResolution and(Supplier<TypeResolution> other) {
+            return failed ? this : other.get();
+        }


Not a fan since it touches QL.

costin · 2024-04-12T00:14:14Z

.../esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/math/Bucket.java

 */
-public class Bucket extends EsqlScalarFunction implements Validatable {
+public class Bucket extends EsqlScalarFunction implements Validatable, TwoOptionalArguments {


Let's restrict bucketing to being a grouping function first - for evals/scalar function's there's date_trunc.
For that introduce a dedicated GroupingFunction base class, similar to the approach used in SQL and wire that in the verifier so this is allowed:

STATS c = count() BY bucket(...)

while this is not:

EVAL x = bucket(..) STATS c = count() BY x

We could allow in time the latter however conceptually bucketing should create buckets/groups/bins - if we treat them as scalar functions, we'd have to materialize the group which is tricky.
Note that if the keys are needed, one could do:

STATS by key = bucket()

which is the same as EVAL however it better preserves the context.

I had rethought this, as other implementations still offer it as a scalar. Though not sure if it's that useful as such and indeed one can get the keys with an aggregation, though reducing the scope.
But I followed the recommendations and turned BUCKET into a grouping function.

costin · 2024-04-12T00:20:52Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/date.csv-spec

-| WHERE hire_date >= "1985-01-01T00:00:00Z" AND hire_date < "1986-01-01T00:00:00Z"
-| EVAL month = BUCKET(hire_date, 20, "1985-01-01T00:00:00Z", "1986-01-01T00:00:00Z")


With the merged implicit conversion for literals, the tests above using strings should still work.

They're all still there, just moved to bucket.csv-spec to group them.

costin · 2024-04-12T00:21:32Z

...sql/src/main/java/org/elasticsearch/xpack/esql/expression/function/EsqlFunctionRegistry.java

@@ -193,7 +193,7 @@ private FunctionDefinition[][] functions() {
                def(Asin.class, Asin::new, "asin"),
                def(Atan.class, Atan::new, "atan"),
                def(Atan2.class, Atan2::new, "atan2"),
-                def(Bucket.class, Bucket::new, "bucket"),
+                def(Bucket.class, Bucket::new, "bucket", "bin"),


See my comment above on grouping - potentially extract this into a separate grouping section.

costin · 2024-04-12T00:23:46Z

.../esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/math/Bucket.java

    private final Expression field;
-    private final Expression buckets;
+    private final Expression bucketsOrSpan;


I don't think orSpan is necessary (when dealing with time, the size of the bucket is the time span).

costin

Left another small round of comments.

costin · 2024-04-15T05:20:43Z

...gin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/grouping/Bucket.java

    private final Expression field;
-    private final Expression buckets;
+    private final Expression bucket;


Inconsistent - the @Param and method refer to buckets instead - pick either bucket or buckets and use that everywhere.
Also - Bucket.bucket?

Reverted back to buckets, that was the intention, thx.

costin · 2024-04-15T05:24:05Z

docs/reference/esql/functions/parameters/bucket.asciidoc

@@ -5,7 +5,7 @@
 `field`::


-`buckets`::
+`bucketsOrSpan`::


This need to be updated.

costin · 2024-04-15T05:24:15Z

docs/reference/esql/functions/kibana/definition/bucket.json

@@ -13,20 +13,56 @@
          "description" : ""
        },
        {
-          "name" : "buckets",
+          "name" : "bucketsOrSpan",


costin · 2024-04-15T05:27:22Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Verifier.java

+                                    inner -> failures.add(
+                                        fail(
+                                            inner,
+                                            "cannot imbricate grouping functions; found [{}] inside [{}]",


imbicate -> nest

costin · 2024-04-15T05:29:19Z

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/VerifierTests.java

+    public void testGroupingInsideAggs() {
+        assertEquals(
+            "1:22: can only use grouping function [bucket(emp_no, 5.)] part of the BY clause",
+            error("from test| stats 3 + bucket(emp_no, 5.) by bucket(emp_no, 6.)")


Please add a test that shows the following works:
stats 3 + bucket(emp_no, 5) by bucket(emp_no, 5) - namely repeating the grouping function in the by clauses, as an aggregation.
Also a test with bucket only inside the agg but not inside BY, plus another one without a BY clause.

Added all these tests and a couple more. Updating the verifier was needed too.

astefan

LGTM

astefan · 2024-04-12T15:59:10Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Verifier.java

+                                    inner -> failures.add(
+                                        fail(
+                                            inner,
+                                            "cannot imbricate grouping functions; found [{}] inside [{}]",


"imbricate" -> maybe "nest"?

## Summary Wraps in the changes from elastic/elasticsearch#107272 <img width="491" alt="Screenshot 2024-04-25 at 4 46 31 PM" src="https://github.com/elastic/kibana/assets/315764/4fb3db49-7702-466b-b1fd-b22ca3ed7a0d"> ### Checklist - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios (still needs to be done... waiting on elastic/elasticsearch#107918)

## Summary Wraps in the changes from elastic/elasticsearch#107272 <img width="491" alt="Screenshot 2024-04-25 at 4 46 31 PM" src="https://github.com/elastic/kibana/assets/315764/4fb3db49-7702-466b-b1fd-b22ca3ed7a0d"> ### Checklist - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios (still needs to be done... waiting on elastic/elasticsearch#107918) (cherry picked from commit 5c69e1f)

# Backport This will backport the following commits from `main` to `8.14`: - [[ES|QL] Update `bucket` signature (#181787)](#181787)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Drew Tate <drew.tate@elastic.co>

## Summary Wraps in the changes from elastic/elasticsearch#107272 <img width="491" alt="Screenshot 2024-04-25 at 4 46 31 PM" src="https://github.com/elastic/kibana/assets/315764/4fb3db49-7702-466b-b1fd-b22ca3ed7a0d"> ### Checklist - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios (still needs to be done... waiting on elastic/elasticsearch#107918)

There were some last minute changes to ES|QL that will ship in 8.14 that we need to take into account: - [BUCKET is now an aggregation function](elastic/elasticsearch#107272) - [index names can no longer be escaped with backticks ](elastic/elasticsearch#108431) I'm also including a change that translates `=` to `==` in WHERE commands, and more useful error messages (map a syntax error to the command where it occurred). As BUCKET is often used for timeseries data and we replaced single and double quotes around index names with backticks, this introduces a high chance of generating syntactically invalid queries. This PR updates the docs and examples and removes the correction from `"` and `'` to "`"

) There were some last minute changes to ES|QL that will ship in 8.14 that we need to take into account: - [BUCKET is now an aggregation function](elastic/elasticsearch#107272) - [index names can no longer be escaped with backticks ](elastic/elasticsearch#108431) I'm also including a change that translates `=` to `==` in WHERE commands, and more useful error messages (map a syntax error to the command where it occurred). As BUCKET is often used for timeseries data and we replaced single and double quotes around index names with backticks, this introduces a high chance of generating syntactically invalid queries. This PR updates the docs and examples and removes the correction from `"` and `'` to "`" (cherry picked from commit 6004cad)

) (#183037) # Backport This will backport the following commits from `main` to `8.14`: - [[Obs AI Assistant] Remove ES|QL escaping for index names (#183028)](#183028)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Dario Gieselaar <dario.gieselaar@elastic.co>

bpintea added 6 commits April 4, 2024 12:57

Rename AUTO_BUCKET to just BUCKET

be31f9b

This renames the function AUTO_BUCKET to just BUCKET.

more renamings

cf41939

Further renamings

99b1b04

Merge branch 'main' into esql/rename_auto_bucket

06e8cb7

Disable some bwc tests

8c1ed8b

bpintea added >enhancement :Analytics/ES|QL AKA ESQL v8.14.0 labels Apr 9, 2024

Update docs/changelog/107272.yaml

8497615

bpintea mentioned this pull request Apr 9, 2024

ESQL: Better tests to AUTO_BUCKET #107228

Merged

bpintea added 3 commits April 9, 2024 16:12

Spotless

2642fd9

Merge branch 'main' into esql/extend_bucket_with_spans

2417727

Readded wrongly merged file

c4b2bf4

bpintea commented Apr 10, 2024

View reviewed changes

bpintea marked this pull request as ready for review April 10, 2024 15:00

bpintea requested review from nik9000, luigidellaquila and costin April 10, 2024 15:01

elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Apr 10, 2024

bpintea requested review from astefan and alex-spies April 10, 2024 15:01

bpintea commented Apr 10, 2024

View reviewed changes

alex-spies reviewed Apr 11, 2024

View reviewed changes

alex-spies self-requested a review April 11, 2024 09:19

alex-spies reviewed Apr 11, 2024

View reviewed changes

Address reviews

e850a2f

alex-spies reviewed Apr 11, 2024

View reviewed changes

alex-spies approved these changes Apr 11, 2024

View reviewed changes

costin reviewed Apr 12, 2024

View reviewed changes

bpintea added 4 commits April 12, 2024 17:21

Make BUCKET a grouping function only

0a63d47

Merge branch 'main' into esql/extend_bucket_with_spans

3609d59

Merge branch 'main' into esql/extend_bucket_with_spans

276c91c

Make tests deterministic

772cee0

bpintea changed the title ~~ESQL: extend BUCKET with spans~~ ESQL: extend BUCKET with spans. Turn it into a grouping function Apr 12, 2024

Spotless

178e35a

costin approved these changes Apr 15, 2024

View reviewed changes

astefan approved these changes Apr 15, 2024

View reviewed changes

bpintea added 5 commits April 15, 2024 17:51

Review comments

1cc02ff

Merge branch 'main' into esql/extend_bucket_with_spans

30eea3e

Merge branch 'main' into esql/extend_bucket_with_spans

eab2cc5

Merge branch 'main' into esql/extend_bucket_with_spans

94305b5

Disable bwc for new test

200618c

bpintea added the ES|QL-ui Impacts ES|QL UI label Apr 16, 2024

bpintea merged commit a2c2e8f into elastic:main Apr 16, 2024
14 checks passed

bpintea deleted the esql/extend_bucket_with_spans branch April 16, 2024 10:57

drewdaemon mentioned this pull request Apr 18, 2024

[ES|QL] Client side validation misconfiguration elastic/kibana#177699

Open

27 tasks

drewdaemon mentioned this pull request Apr 25, 2024

[ES|QL] Update bucket signature elastic/kibana#181787

Merged

1 task

dgieselaar mentioned this pull request May 9, 2024

[Obs AI Assistant] Remove ES|QL escaping for index names elastic/kibana#183028

Merged

luigidellaquila mentioned this pull request Jul 2, 2024

ESQL: AUTO_BUCKET() can return NaN #105166

Closed

		\| WHERE hire_date >= "1985-01-01T00:00:00Z" AND hire_date < "1986-01-01T00:00:00Z"
		\| EVAL month = BUCKET(hire_date, 20, "1985-01-01T00:00:00Z", "1986-01-01T00:00:00Z")

ESQL: extend BUCKET with spans. Turn it into a grouping function #107272

ESQL: extend BUCKET with spans. Turn it into a grouping function #107272

Conversation

bpintea commented Apr 9, 2024 • edited Loading

github-actions bot commented Apr 9, 2024

elasticsearchmachine commented Apr 9, 2024

Choose a reason for hiding this comment

elasticsearchmachine commented Apr 10, 2024

Choose a reason for hiding this comment

alex-spies left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alex-spies left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alex-spies left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

costin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

astefan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bpintea commented Apr 9, 2024 •

edited

Loading