window functions docs by techdocsmith · Pull Request #14739 · apache/druid

techdocsmith · 2023-08-02T20:15:05Z

Adds docs for SQL Window Functions

This PR has:

[x ] been self-reviewed.
- using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)
added documentation for new or modified features or behaviors.
a release note entry in the PR description.
added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
added or updated version, license, or notice information in licenses.yaml
added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
added integration tests.
[ x] been tested in a test Druid cluster.

vtlim

Great examples! Left comments on style and areas that could use more explanation.

docs/querying/sql-window-functions.md

Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>

docs/querying/sql-window-functions.md

Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>

vtlim

Docs LGTM!

2bethere

Overall I think this looks good.
I'd like to suggest adding a section around known issues and the limitations that we have here. For example, if the user sees a Null point exception during a query, they might be using an unsupported query shape.
Similar to how you called out Group by queries are required.

docs/querying/sql-window-functions.md

Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>

docs/querying/sql-window-functions.md

Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>

docs/querying/sql-window-functions.md

imply-cheddar · 2023-08-08T14:44:16Z

docs/querying/sql-window-functions.md

+The following are known issues with window functions:
+
+- Descending order, DESC, in the ORDER BY clause in the window definition causes a Null Pointer Exception.
+- Druid throws an exception if you do not include the offset and default for LEAD() and LAG() functions.


What would the alternative be? That is, if the offset is not defined, what should be returned? The default can be null, so that makes sense, but I'm unclear what a meaning default value for offset in LEAD/LAG would be?

I only know by comparison to Postgres which sets the default offset to 1 if not included:

If omitted, offset defaults to 1 and default to NULL.

I have also observed that the default value is not respected when set. For example for the first row in the window a LEAD(dim, 1, default) should apply the default value and that does not appear to be the case.

Also, it would be better to catch the exception throw a user-readable message so they know to set the offset.

this default offset stuff was fixed recently

rm unwanted file

docs/querying/sql-window-functions.md

kgyrtkirk · 2023-10-13T10:11:44Z

docs/querying/sql-window-functions.md

+
+There are known issues where ORDER BY only works on ascending order and certain options may cause errors.
+
+Set the context parameter `windowsAreForClosers: true` to use window functions.


we should rename this to something else :)

kgyrtkirk · 2023-10-13T10:13:56Z

docs/querying/sql-window-functions.md

+You can use the OVER clause to treat other Druid aggregation functions as window functions. For example, the sum of a value for rows within a window.
+
+When working with window functions, consider the following:
+- Window functions only work on GROUP BY queries.


I don't think we need this anymore - it should work

I tried a couple of queries like

SELECT RANK() OVER (PARTITION BY m1 ORDER BY m1 DESC) AS ranking, m1 FROM foo

Does not give any results but if I add a GROUP BY m1 it does give results. As for this I would like to include GROUP BY in the docs as the window functions are still experimental and once we are confident that the cases run without the group by we can update it in the next release

kgyrtkirk · 2023-10-13T10:22:08Z

docs/querying/sql-window-functions.md

+|`PERCENT_RANK()`| Returns the rank of the row calculated as a percentage according to the formula: `(rank - 1) / (total window rows - 1)` |
+|`CUME_DIST()`| Returns the cumulative distribution of the current row within the window calculated as `number of window rows at the same rank or higher than current row` / `total window rows` |
+|`NTILE(tiles)`| Divides the rows within a window as evenly as possible into the number of tiles, also called buckets, and returns the value of the tile that the row falls into  |None |
+|`LAG(expr, offset, default)`| Returns the value evaluated at the row that precedes the current row by the offset number within the window; if there is no such row, returns the given default value |


I don't think we have LAG(expr, offset, default) ; to my best knowledge we have this

for which we could write:

LAG(expr, [offset])

offset defaults to 1 if omitted

remove the part about default value

similar commment notes for LEAD

kgyrtkirk · 2023-10-13T10:27:05Z

docs/querying/sql-window-functions.md

+    PERCENT_RANK() OVER w AS pct_rank,
+    CUME_DIST() OVER w AS cumulative_dist,
+    NTILE(4) OVER w AS ntile_val,
+    LAG(ABS(delta), 1, 0) OVER w AS lag_val,


I don't think this default work - I see null in the output; and not 0
I assume this query can be run so I think we accept it ; but doesn't live up to working accordingly....

@kgyrtkirk , don't know if this is a bug in the function or if there is not an intention to be a default value (I was basing my docs off pg)

It's kinda odd to me that the function signature accepts the value, but maybe it is doing something else?

its a bug in the LEAD/LAG implementation ; it doesn't take the default into consideration; however Calcite accepts it - we could either reject these ; or implement it - either way I've opened this ticket for it

kgyrtkirk · 2023-10-13T10:40:39Z

docs/querying/sql-window-functions.md

+|`2016-06-27T04:20:00.000Z`|`#kk.wikipedia`|56|56|
+|`2016-06-27T04:35:00.000Z`|`#kk.wikipedia`|2440|2496|


I might have made a mistake I can't see right now - but I'm pretty confident that this should be 2496 for both of these rows

I think we have:

OVER(... ORDER BY FLOOR( __time TO HOUR) => should be ordered by the floored value

running aggregate of SUM() should be calculated for up-to-all current values of orderby value...

these 2 rows should floor to 4am but they have different running-sum values

I was playing with pg in this sqlfiddle

@kgyrtkirk

The query floors to MINUTE not hour in line 177:
FLOOR(__time TO MINUTE)

For consistency I changed line 180 to floor to MINUTE too -- it keeps the same results as I have here.

I wonder if there is a bug with Druid in the case that you pointed out. Where it looks like it honored the MINUTE and not the HOUR?

for the resultset line 180 mattered more :)

Thank you for changing it now the resultset seem to be more accurate! 🗡️

kgyrtkirk · 2023-10-13T10:43:17Z

docs/querying/sql-window-functions.md

+The following are known issues with window functions:
+
+- Descending order, DESC, in the ORDER BY clause in the window definition causes a Null Pointer Exception.
+- Druid throws an exception if you do not include the offset and default for LEAD() and LAG() functions.


this default offset stuff was fixed recently

docs/querying/sql-window-functions.md

kgyrtkirk · 2023-10-13T10:57:06Z

website/sidebars.json

    ]
  }
-}
+}


nit: newline removal?

abhishekagarwal87 · 2023-10-27T11:51:23Z

docs/querying/sql-window-functions.md

+Apache Druid supports two query languages: [Druid SQL](sql.md) and [native queries](querying.md).
+This document describes the SQL language.
+
+Window functions are an [experimental](../development/experimental.md) feature. Development and testing are still at early stage. Feel free to try window functions and provide your feedback.


Suggested change

Window functions are an [experimental](../development/experimental.md) feature. Development and testing are still at early stage. Feel free to try window functions and provide your feedback.

Window functions are an [experimental](../development/experimental.md) feature. Development and testing are still at early stage. Feel free to try window functions and provide your feedback. Windows functions are not supported by multi-stage-query engine and thus cannot be used in SQL-based Ingestion as of now.

kgyrtkirk · 2023-10-27T11:53:49Z

docs/querying/sql-window-functions.md

+
+The OVER clause defines the query windows for window functions as follows:
+- PARTITION BY indicates the dimension that defines the rows within the window
+- ORDER BY specifies the order of the rows within the windows. Currently only ascending order, ASC, works.


we now accept DESC as well now (and nulls X in case it aligns with the current behaviour)
so Currently only ascending order, ASC, works. could be removed

kgyrtkirk · 2023-10-27T12:07:26Z

docs/querying/sql-window-functions.md

+    PERCENT_RANK() OVER w AS pct_rank,
+    CUME_DIST() OVER w AS cumulative_dist,
+    NTILE(4) OVER w AS ntile_val,
+    LAG(ABS(delta), 1, 0) OVER w AS lag_val,


its a bug in the LEAD/LAG implementation ; it doesn't take the default into consideration; however Calcite accepts it - we could either reject these ; or implement it - either way I've opened this ticket for it

kgyrtkirk · 2023-10-27T12:10:45Z

docs/querying/sql-window-functions.md

+|`2016-06-27T04:20:00.000Z`|`#kk.wikipedia`|56|56|
+|`2016-06-27T04:35:00.000Z`|`#kk.wikipedia`|2440|2496|


for the resultset line 180 mattered more :)

Thank you for changing it now the resultset seem to be more accurate! 🗡️

kgyrtkirk · 2023-10-27T12:35:11Z

docs/querying/sql-window-functions.md

+
+The following are known issues with window functions:
+
+- Descending order, DESC, in the ORDER BY clause in the window definition causes a Null Pointer Exception.


no; this is not there anymore - however the following is known:

aggregates which have ORDER BY specified are processed in the window: ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW

note: other database seem to use the default of RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW in this case (not sure if it is specified or not in the standard - have to dig it out)

note: in case the order column is unique there is no difference between RANGE / ROWS

windows with RANGE specifications are handled as ROWS

LEAD/LAG ignores the default value

LAST_VALUE returns the last value of the window even in the presence of an order by

here is an example query which could illustrate these (not sure if needed)

SELECT FLOOR(__time TO DAY) t, cnt, FLOOR(m1/3), COUNT(1) OVER (ORDER BY FLOOR(m1/3) ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW), --ok COUNT(1) OVER (ORDER BY FLOOR(m1/3) RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW), -- bad LAST_VALUE(cnt) OVER (ORDER BY __time), -- last value of the window LAG(cnt) OVER (), -- ok LAG(cnt,1,999) OVER () -- default value ignored FROM foo

soumyava · 2023-10-31T15:40:17Z

docs/querying/sql-window-functions.md

+
+Window functions are an [experimental](../development/experimental.md) feature. Development and testing are still at early stage. Feel free to try window functions and provide your feedback.
+
+There are known issues where ORDER BY only works on ascending order and certain options may cause errors.


This has been fixed recently. Now both the orders work fine. The known issue mentioned here has been resolved

docs/querying/sql-window-functions.md

soumyava

Some nits. Mostly LGTM !

docs/querying/sql-window-functions.md

soumyava · 2023-11-03T05:03:33Z

docs/querying/sql-window-functions.md

+
+## Examples
+
+The following example illustrates all of the built-in window functions to compare the number of characters changed per event for a channel in the Wikipedia data set.


Probably sometime later when these are not experimental we could run an example with a use case and show how an empty over() works followed by introducing partition by and order by similar to how Drill does in https://drill.apache.org/docs/sql-window-functions-introduction/. Not now, but some food for thought for future

docs/querying/sql-window-functions.md

somu-imply

Thanks @techdocsmith for the docs. LGTM

* draft window functions * Apply suggestions from code review Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com> * address comments * remove default column * Update docs/querying/sql-window-functions.md Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com> * Update docs/querying/sql-window-functions.md Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com> * fix ntile * remove default header column * code tics to remove spelling errors * add known issues, add SUM example * Apply suggestions from code review Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com> * address spelling * remove extra chars * add to sidebar, fix admonition * Update sql-window-functions.md accept suggestion, change admonition style * update sidebar * Delete Untitled.ipynb rm unwanted file * Update docs/querying/sql-window-functions.md * Update docs/querying/sql-window-functions.md * update context param, accept suggestions * accept suggestions * Apply suggestions from code review * Fix known issues * require GROUP BY, explain order of operation * accept suggestions * fix spelling --------- Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>

draft window functions

e0447b1

github-actions bot added the Area - Documentation label Aug 2, 2023

vtlim reviewed Aug 2, 2023

View reviewed changes

Apply suggestions from code review

c827e9f

Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>

vtlim reviewed Aug 2, 2023

View reviewed changes

docs/querying/sql-window-functions.md Outdated Show resolved Hide resolved

techdocsmith and others added 5 commits August 2, 2023 14:01

address comments

30daa20

remove default column

06f5052

Update docs/querying/sql-window-functions.md

947d3be

Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>

Update docs/querying/sql-window-functions.md

2d482f3

Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>

fix ntile

18f2c56

vtlim approved these changes Aug 2, 2023

View reviewed changes

remove default header column

e89a3fc

techdocsmith force-pushed the wf-docs branch from 6af8583 to 0a226c3 Compare August 2, 2023 23:30

2bethere reviewed Aug 2, 2023

View reviewed changes

docs/querying/sql-window-functions.md Show resolved Hide resolved

code tics to remove spelling errors

0afe246

techdocsmith force-pushed the wf-docs branch 2 times, most recently from e026fdc to 093a1c7 Compare August 3, 2023 20:25

add known issues, add SUM example

e2e0cd1

techdocsmith force-pushed the wf-docs branch from 093a1c7 to e2e0cd1 Compare August 3, 2023 20:26

vtlim reviewed Aug 3, 2023

View reviewed changes

Apply suggestions from code review

2773073

Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>

techdocsmith commented Aug 3, 2023

View reviewed changes

docs/querying/sql-window-functions.md Show resolved Hide resolved

docs/querying/sql-window-functions.md Outdated Show resolved Hide resolved

techdocsmith and others added 3 commits August 3, 2023 15:29

Apply suggestions from code review

3191038

Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>

address spelling

b3766cf

Merge branch 'wf-docs' of github.com:techdocsmith/druid into wf-docs

3f5d801

techdocsmith changed the title ~~draft window functions~~ window functions docs Aug 4, 2023

remove extra chars

27cabcd

imply-cheddar reviewed Aug 8, 2023

View reviewed changes

techdocsmith added 2 commits August 16, 2023 15:21

Merge branch 'master' into wf-docs

ad0726b

add to sidebar, fix admonition

100fe85

Delete Untitled.ipynb

35f615f

rm unwanted file

vtlim reviewed Aug 16, 2023

View reviewed changes

docs/querying/sql-window-functions.md Outdated Show resolved Hide resolved

Update docs/querying/sql-window-functions.md

8b85803

techdocsmith commented Aug 16, 2023

View reviewed changes

docs/querying/sql-window-functions.md Outdated Show resolved Hide resolved

Update docs/querying/sql-window-functions.md

32fd8b0

vtlim approved these changes Aug 16, 2023

View reviewed changes

kgyrtkirk reviewed Oct 13, 2023

View reviewed changes

merge master sidebars

211aae3

techdocsmith mentioned this pull request Oct 17, 2023

Rename windowing option #15184

Merged

update context param, accept suggestions

5f3740c

LakshSingla added this to the 28.0 milestone Oct 17, 2023

accept suggestions

d0be3ff

abhishekagarwal87 reviewed Oct 27, 2023

View reviewed changes

kgyrtkirk reviewed Oct 27, 2023

View reviewed changes

soumyava reviewed Oct 31, 2023

View reviewed changes

techdocsmith commented Nov 1, 2023

View reviewed changes

docs/querying/sql-window-functions.md Outdated Show resolved Hide resolved

techdocsmith commented Nov 1, 2023

View reviewed changes

docs/querying/sql-window-functions.md Outdated Show resolved Hide resolved

techdocsmith commented Nov 1, 2023

View reviewed changes

docs/querying/sql-window-functions.md Outdated Show resolved Hide resolved

techdocsmith commented Nov 1, 2023

View reviewed changes

docs/querying/sql-window-functions.md Outdated Show resolved Hide resolved

techdocsmith added 3 commits November 1, 2023 10:54

Apply suggestions from code review

8daf3e1

Fix known issues

c1257ca

require GROUP BY, explain order of operation

91a0025

soumyava reviewed Nov 3, 2023

View reviewed changes

accept suggestions

fa4b895

somu-imply approved these changes Nov 6, 2023

View reviewed changes

soumyava approved these changes Nov 6, 2023

View reviewed changes

fix spelling

1d22f81

soumyava merged commit 0403e48 into apache:master Nov 6, 2023


		There are known issues where ORDER BY only works on ascending order and certain options may cause errors.

		Set the context parameter `windowsAreForClosers: true` to use window functions.

		\|`2016-06-27T04:20:00.000Z`\|`#kk.wikipedia`\|56\|56\|
		\|`2016-06-27T04:35:00.000Z`\|`#kk.wikipedia`\|2440\|2496\|

                   ]
                 }
-              }
+              }

                
                    No newline at end of file

	Window functions are an [experimental](../development/experimental.md) feature. Development and testing are still at early stage. Feel free to try window functions and provide your feedback.
	Window functions are an [experimental](../development/experimental.md) feature. Development and testing are still at early stage. Feel free to try window functions and provide your feedback. Windows functions are not supported by multi-stage-query engine and thus cannot be used in SQL-based Ingestion as of now.


		The following are known issues with window functions:

		- Descending order, DESC, in the ORDER BY clause in the window definition causes a Null Pointer Exception. No newline at end of file


		## Examples

		The following example illustrates all of the built-in window functions to compare the number of characters changed per event for a channel in the Wikipedia data set.

Comments

Conversation

techdocsmith commented Aug 2, 2023

Uh oh!

vtlim left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vtlim left a comment

Choose a reason for hiding this comment

Uh oh!

2bethere left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

techdocsmith Aug 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

techdocsmith Oct 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

techdocsmith Aug 8, 2023 •

edited

Loading

techdocsmith Oct 18, 2023 •

edited

Loading