New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a UDF to extract the window start time from the message key #1674

Closed
apurvam opened this Issue Jul 31, 2018 · 11 comments

Comments

Projects
None yet
8 participants
@apurvam
Copy link
Contributor

apurvam commented Jul 31, 2018

We should add a WINDOWSTARTTIME() UDF which returns the start time of the window if the underlying topic is windowed, or else return some sentinel value (like LONG_MAX) if it isn't.

Getting access to the window start time is generally useful, and we can't depend on ROWTIME for this piece of information. See #1497

@zlex7

This comment has been minimized.

Copy link

zlex7 commented Aug 9, 2018

Hey, I'm new to contributing. Can I try adding this?

@rodesai

This comment has been minimized.

Copy link
Contributor

rodesai commented Aug 9, 2018

@zlex7 go for it!

@apurvam

This comment has been minimized.

Copy link
Contributor

apurvam commented Aug 9, 2018

yes. thanks @zlex7 . I am assigning this to you.

@apurvam

This comment has been minimized.

Copy link
Contributor

apurvam commented Aug 9, 2018

Actually, I can't assign it to you for some reason. In any case, feel free to take a stab at it. You can follow the example from here to add a new UDF: https://github.com/confluentinc/ksql/pull/1611/files

@miguno

This comment has been minimized.

Copy link
Member

miguno commented Aug 14, 2018

Thanks for taking a stab at this, @zlex7 !

@rmoff

This comment has been minimized.

Copy link
Contributor

rmoff commented Sep 3, 2018

would be good to include WINDOWENDTIME for completeness :)

@blueedgenick

This comment has been minimized.

Copy link
Contributor

blueedgenick commented Sep 3, 2018

Thinking aloud, maybe a udf is not the best solution for this - how about we instead provide an additional pseudo-column that's automatically created on the output table, say WINDOWTIME, just like we do for the message key and timestamp already. I wonder if this would feel more natural?

@mjsax

This comment has been minimized.

Copy link
Member

mjsax commented Sep 3, 2018

I would recommend to return NULL if the UDF is applied to a non-windowed column. A sentinel value as LONG_MAX might not make much sense IMHO and could be miss-leading downstream.

@rmoff

This comment has been minimized.

Copy link
Contributor

rmoff commented Sep 4, 2018

@blueedgenick IMO a UDF would be a useful stop-gap solution for something that has in effect caused a regression in functionality. I think the discussion around UDF vs system column did take place, and TBH so long as we get it built sooner than later either would be fine :)

@apurvam

This comment has been minimized.

Copy link
Contributor

apurvam commented Sep 26, 2018

@big-andy-coates said he would take this on.

@big-andy-coates big-andy-coates referenced this issue Oct 3, 2018

Merged

Add WindowStart and WindowEnd UDFs #1993

1 of 5 tasks complete
@big-andy-coates

This comment has been minimized.

Copy link
Member

big-andy-coates commented Oct 8, 2018

Yep, I'm working on this... it's more involved than it might look.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment