You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Arroyo supports window functions, but only under very specific circumstances:
Only ROW_NUMBER() is supported.
The input data can't be updating.
The first term of the partition clause has to be a window struct, from either hop() or tumble().
The first two constraints are simply a matter of missing implementations. There's no semantic issue with supporting window functions such as LAG or ntile(), just a matter of correctly generating the code.
Similarly, updating inputs just require being able to remove or modify records from state, which is doable.
For the final constraint, it's a little trickier. Many of the window functions, such as ROW_NUMBER(), are sensitive to the total order of the partition. A single record coming in can change the value for an arbitrary number of rows. Supporting this style of update may be asking for trouble. That being said, there are some other time-sensitive windowing functions we could support. For instance, if your ordering is by event time and your window function only needs to know preceding terms, you could compute the streaming result by using the watermarks.
This is a tracking issue just to start the discussion. Individual improvements should reference it as we make progress.
The text was updated successfully, but these errors were encountered:
Arroyo supports window functions, but only under very specific circumstances:
hop()
ortumble()
.The first two constraints are simply a matter of missing implementations. There's no semantic issue with supporting window functions such as
LAG
orntile()
, just a matter of correctly generating the code.Similarly, updating inputs just require being able to remove or modify records from state, which is doable.
For the final constraint, it's a little trickier. Many of the window functions, such as ROW_NUMBER(), are sensitive to the total order of the partition. A single record coming in can change the value for an arbitrary number of rows. Supporting this style of update may be asking for trouble. That being said, there are some other time-sensitive windowing functions we could support. For instance, if your ordering is by event time and your window function only needs to know preceding terms, you could compute the streaming result by using the watermarks.
This is a tracking issue just to start the discussion. Individual improvements should reference it as we make progress.
The text was updated successfully, but these errors were encountered: