Group By of windowed sources results in UnsupportedOperationException #4396
Comments
agavra
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
{{ message }}
agavra
changed the title
QTT test case:
{ "name": "windowed stream source using window bounds columns", "statements": [ "CREATE STREAM TEST (ID bigint KEY, ignored bigint) WITH (kafka_topic='test_topic', value_format='DELIMITED', WINDOW_TYPE='Tumbling', WINDOW_SIZE='30 seconds');", "CREATE TABLE S2 as SELECT count(1) as count, min(windowstart) as wstart, min(windowend) as wend FROM test GROUP BY ID;" ], "inputs": [ {"topic": "test_topic", "key": 0, "value": "0", "timestamp": 0, "window": {"start": 0, "end": 30000, "type": "time"}}, {"topic": "test_topic", "key": 0, "value": "100", "timestamp": 0, "window": {"start": 0, "end": 30000, "type": "time"}}, {"topic": "test_topic", "key": 0, "value": "10", "timestamp": 0, "window": {"start": 0, "end": 30000, "type": "time"}}, {"topic": "test_topic", "key": 1, "value": "50", "timestamp": 30000, "window": {"start": 30000, "end": 60000, "type": "time"}} ], "outputs": [ {"topic": "S2", "key": 0, "value": "1,0,30000", "timestamp": 0}, {"topic": "S2", "key": 0, "value": "2,0,30000", "timestamp": 0}, {"topic": "S2", "key": 0, "value": "3,0,30000", "timestamp": 0}, {"topic": "S2", "key": 1, "value": "1,30000,60000", "timestamp": 30000} ], "post": { "sources": [ { "name": "S2", "type": "table", "keyFormat": {"format": "KAFKA", "windowType": null}, "schema": "ROWKEY INT KEY, COUNT BIGINT, WSTART BIGINT, WEND BIGINT" } ] } }Results in error:
This error message is really unhelpful.
The query is taking a windowed stream with a schema, (including the window bounds columns of):
And then is performing a GROUP BY on the ID column, counting the number of rows, and capturing the minimum window start and end times. As it stands, it's a pretty nonsensical query. Though there may be useful variants that utilise different udafs... just can't think of any!
Really, such a query should probably just work. So solution 1 is 'make it work'. Alternatively, we can punt fixing this to some later time (maybe after structured keys work). In which case we should split this into two tasks. Task 1: improve the error message, i.e. replace the
UnsupportedOperationExceptionwith a single KsqlException stating "GROUP BYon a windowed source is not supported", and then a second task to actually make it work.The text was updated successfully, but these errors were encountered: