feat: Window func #5401

doki23 · 2022-05-16T08:08:28Z

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

impl it partially including

aggr function
const rows frame
const range frame
only works in the old scheduler framework now

Changelog

New Feature
aggregate window function

Related Issues

Closes #4653

vercel · 2022-05-16T08:08:32Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Updated
databend	⬜️ Ignored (Inspect)		Jun 9, 2022 at 1:49AM (UTC)

mergify · 2022-05-16T08:08:58Z

Thanks for the contribution!
I have applied any labels matching special text in your PR Changelog.

Please review the labels and make any necessary changes.

Xuanwo · 2022-05-16T11:30:36Z

@mergify update

mergify · 2022-05-16T11:31:10Z

update

✅ Branch has been successfully updated

BohuTANG · 2022-05-16T11:38:15Z

common/functions/src/window/mod.rs

@@ -0,0 +1,5 @@
+mod window_frame;


Hmm, need a license header to make the lint happy :)

Thanks, I'll fix it then. And this pr's far away from ready for reviews 😂.

A great start! Welcome asking at discussions if you have any questions 💌

doki23 · 2022-06-07T07:24:16Z

Sorry for the late update, because I'm too busy recently.
It's ready for review now.

doki23 · 2022-06-07T07:31:08Z

common/functions/src/window/window_frame.rs

+#[derive(
+    Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Hash, serde::Serialize, serde::Deserialize,
+)]
+pub struct WindowFrame {


Code in this file is mainly copied from another open source project -- apache/arrow-datafusion. And I'm not sure whether it's proper or if I should add some description on the top of it.

I think it's ok. Do we have stateless tests for this pr? Like: https://github.com/datafuselabs/databend/blob/49252766894f3bb5f5d4767037142cb1b10d92bc/tests/suites/0_stateless/03_dml/03_0003_select_group_by.sql

Please attach their license header in this file with a clear statement, for example:

https://github.com/apache/arrow-datafusion/blob/master/datafusion/core/src/prelude.rs#L1-L16

And add descriptions include:

Original Project

Link to commits

I think it's ok. Do we have stateless tests for this pr? Like: https://github.com/datafuselabs/databend/blob/49252766894f3bb5f5d4767037142cb1b10d92bc/tests/suites/0_stateless/03_dml/03_0003_select_group_by.sql

Thanks for the remind and I'll add some stateless tests then.

sundy-li · 2022-06-08T03:41:05Z

Impressive pr, I'll take time to review it.

zhang2014

👍 Look good to me. BTW: need implement window function in new processor.

BohuTANG · 2022-06-08T14:08:51Z

@mergify update

mergify · 2022-06-08T14:09:25Z

update

✅ Branch has been successfully updated

BohuTANG

👍

doki23 · 2022-06-08T14:50:58Z

👍 Look good to me. BTW: need implement window function in new processor.

Yes, I'll create a follow-up issue and also make some optimizations about performance.

sundy-li

Left some comments, these can be improved in another pr.

sundy-li · 2022-06-09T03:54:19Z

query/src/pipelines/transforms/transform_window_func.rs

+            arguments.push(ColumnWithField::new(Arc::clone(arg_column), arg_field));
+        }
+
+        let function = if !AggregateFunctionFactory::instance().check(op) {


we can have a function field in this transformer.

sundy-li · 2022-06-09T03:54:51Z

query/src/pipelines/transforms/transform_window_func.rs

+            })
+            .collect::<Vec<_>>();
+
+        let window_col = Series::concat(&window_col_per_tuple).unwrap();


unwrap could be replaced by Result.

sundy-li · 2022-06-09T03:57:05Z

query/src/pipelines/transforms/transform_window_func.rs

+                    .collect::<Vec<ColumnRef>>()
+            })
+            .map(|args| {
+                let place = arena.alloc_layout(func.state_layout());


we should call drop_states to destroy the inner allocation as PartialAggregator did.

BohuTANG · 2022-06-24T00:47:10Z

Ping @soyeric128 for documentation.

soyeric128 · 2022-06-27T14:34:16Z

Hi @BohuTANG @doki23

Could you pls help confirm?

Do we support ALL of these aggregate functions for window functions: https://databend.rs/doc/reference/functions/aggregate-functions
Do we now support any of non-aggr window functions?

Thanks.

doki23 · 2022-06-28T02:08:19Z

Hi @BohuTANG @doki23

Could you pls help confirm?

Do we support ALL of these aggregate functions for window functions: https://databend.rs/doc/reference/functions/aggregate-functions

Do we now support any of non-aggr window functions?

Thanks.

We support all aggregate functions(any function that impl trait AggregateFunction) but have not yet implemented any non-aggr functions.

soyeric128 · 2022-07-04T18:38:44Z

Hi @BohuTANG @doki23
Could you pls help confirm?

Do we support ALL of these aggregate functions for window functions: https://databend.rs/doc/reference/functions/aggregate-functions

Do we now support any of non-aggr window functions?

Thanks.

We support all aggregate functions(any function that impl trait AggregateFunction) but have not yet implemented any non-aggr functions.

Thanks for reply.
Pls review this PR for doc updates: #6445

doki23 added 11 commits April 22, 2022 12:06

window func expression

7304180

update structure of window function expression

0d57622

window plan

962f50f

local commit

e180ef1

Merge branch 'main' of github.com:datafuselabs/databend into window_func

9e1ba01

window logical plan done

f455a6a

Merge branch 'main' of github.com:datafuselabs/databend into window_func

4dee1d1

fix

a121073

Merge branch 'main' of github.com:datafuselabs/databend into window_func

3027eef

temporary commit

89a707a

basic func

65c3563

databend-bot added the need-review label May 16, 2022

mergify bot added the pr-feature this PR introduces a new feature to the codebase label May 16, 2022

Merge branch 'main' into window_func

f350820

vercel bot deployed to Preview May 16, 2022 11:32 View deployment

BohuTANG reviewed May 16, 2022

View reviewed changes

doki23 and others added 9 commits May 24, 2022 10:08

Merge branch 'main' of github.com:datafuselabs/databend into window_func

6ca5006

Merge branch 'main' of github.com:datafuselabs/databend into window_func

2a284c9

naive evaluation and rows frame

e032aa7

Merge branch 'datafuselabs:main' into window_func

1e60ccb

stage but with a mock hash function

9701d7c

support multi partition by

ebd377f

support range frame

f6fd723

evaluation with segment tree and clean the code

70b2ed5

Merge branch 'main' of github.com:datafuselabs/databend into window_func

be48007

vercel bot deployed to Preview June 6, 2022 16:52 View deployment

doki23 marked this pull request as ready for review June 7, 2022 07:22

doki23 changed the title ~~(WIP)Window func~~ feat: Window func Jun 7, 2022

doki23 commented Jun 7, 2022

View reviewed changes

clippy

dd15974

doki23 requested review from BohuTANG and Xuanwo June 7, 2022 07:36

doki23 added 2 commits June 8, 2022 11:22

add stateless tests and fix some bugs of range frame current row bound

ce0e752

licenses

f358099

sundy-li self-requested a review June 8, 2022 03:41

BohuTANG requested review from zhang2014 and removed request for BohuTANG June 8, 2022 09:52

remove error case

5532bc8

zhang2014 approved these changes Jun 8, 2022

View reviewed changes

Merge branch 'main' into window_func

e5c8a8d

BohuTANG mentioned this pull request Jun 8, 2022

need implement window function in new processor. #5855

Closed

BohuTANG approved these changes Jun 8, 2022

View reviewed changes

vercel bot deployed to Preview June 8, 2022 14:11 View deployment

mergify bot added 2 commits June 8, 2022 22:33

Merge branch 'main' into window_func

0b06551

Merge branch 'main' into window_func

acaf070

mergify bot merged commit f7e7b1c into datafuselabs:main Jun 9, 2022

sundy-li reviewed Jun 9, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Window func #5401

feat: Window func #5401

doki23 commented May 16, 2022 •

edited

vercel bot commented May 16, 2022 •

edited

mergify bot commented May 16, 2022

Xuanwo commented May 16, 2022

mergify bot commented May 16, 2022

BohuTANG May 16, 2022 •

edited

doki23 May 16, 2022

Xuanwo May 16, 2022

doki23 commented Jun 7, 2022

doki23 Jun 7, 2022

sundy-li Jun 7, 2022 •

edited

Xuanwo Jun 7, 2022

doki23 Jun 7, 2022

sundy-li commented Jun 8, 2022

zhang2014 left a comment

BohuTANG commented Jun 8, 2022

mergify bot commented Jun 8, 2022

BohuTANG left a comment

doki23 commented Jun 8, 2022 •

edited

sundy-li left a comment

sundy-li Jun 9, 2022

sundy-li Jun 9, 2022

sundy-li Jun 9, 2022

BohuTANG commented Jun 24, 2022

soyeric128 commented Jun 27, 2022

doki23 commented Jun 28, 2022 •

edited

soyeric128 commented Jul 4, 2022

feat: Window func #5401

feat: Window func #5401

Conversation

doki23 commented May 16, 2022 • edited

Summary

Changelog

Related Issues

vercel bot commented May 16, 2022 • edited

mergify bot commented May 16, 2022

Xuanwo commented May 16, 2022

mergify bot commented May 16, 2022

✅ Branch has been successfully updated

BohuTANG May 16, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

doki23 commented Jun 7, 2022

Choose a reason for hiding this comment

sundy-li Jun 7, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sundy-li commented Jun 8, 2022

zhang2014 left a comment

Choose a reason for hiding this comment

BohuTANG commented Jun 8, 2022

mergify bot commented Jun 8, 2022

✅ Branch has been successfully updated

BohuTANG left a comment

Choose a reason for hiding this comment

doki23 commented Jun 8, 2022 • edited

sundy-li left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BohuTANG commented Jun 24, 2022

soyeric128 commented Jun 27, 2022

doki23 commented Jun 28, 2022 • edited

soyeric128 commented Jul 4, 2022

doki23 commented May 16, 2022 •

edited

vercel bot commented May 16, 2022 •

edited

BohuTANG May 16, 2022 •

edited

sundy-li Jun 7, 2022 •

edited

doki23 commented Jun 8, 2022 •

edited

doki23 commented Jun 28, 2022 •

edited