docs: add klip-24: key column semantics in queries.

[big-andy-coates]

This klip looks to address some of the shortcomings found during the recent work to remove the restriction that all key columns must be named ROWKEY.

Please have a read and let me know your thoughts.

Importantly, the 'any key name' work is not yet enabled, (See #5093). So we have a choice... do we fix these semantics before or after we enable this feature?

Before means less disruption for users, but may mean the feature slips the milestone.

After means we'll hit the milestone, but users will be asked to change their existing queries one way, only to be asked to change them back on the next release ... annoying! For example:

-- existing valid GROUP BY persistent query:
CREATE TABLE OUTPUT AS 
   SELECT V0, COUNT() AS COUNT 
     FROM INPUT GROUP BY V0;

-- with 'any key name' merged the above fails with a duplicate column error on V0.
-- the user needs to change the query to:
CREATE TABLE OUTPUT AS 
   SELECT COUNT() AS COUNT 
     FROM INPUT GROUP BY V0;

-- with this klip the first statement is valid, but the second is not, hence we'd be asking users to change their query back again.

[big-andy-coates]

cc: @MichaelDrogalis @derekjn @agavra @apurvam @blueedgenick @rmoff

[agavra]

Thanks @big-andy-coates! I like all of the "main" changes proposed in this KLIP, my comments are mostly on the "secondary" and TBD parts.

Before means less disruption for users, but may mean the feature slips the milestone.

This is my vote, I think the current behavior for any keys would turn off some users - and the confusion for users jumping between models would be really frustrating.

[big-andy-coates]

@mjsax & @agavra : I've updated this inline with discussions and also discovered another edge case: grouping by multiple expressions. Can you please take a look at this section and provide any feedback? Thanks!

[rmoff]

left a few comments but in general this all looks good 👍

[agavra]

LGTM!

[derekjn]

Great proposal @big-andy-coates, I think this makes things much cleaner and more explicit. The only question I have has been asked by @rmoff: #5115 (comment)

Otherwise LGTM 👍

[big-andy-coates]

Great proposal @big-andy-coates, I think this makes things much cleaner and more explicit. The only question I have has been asked by @rmoff: #5115 (comment)

Otherwise LGTM 👍

@derekjn, replied to Robin's comment: it's actually a editing mistake in the KLIP.

[big-andy-coates]

I've updated the KLIP with a new edge case around outer joins and joins on non-column-refs. The proposed short to medium term work around is not pretty, but is practical.

Would appreciate peoples views. cc @rmoff, @blueedgenick, @agavra, @derekjn, @apurvam

[agavra]

Still LGTM

[mjsax]

As the proposal is to use ROWKEY column for outer joins, it seems the same pattern could be applied for this case to allow people to rename the PK?

[big-andy-coates]

Hummmm.... you've got a point.

However, I think there's a subtle difference we may want to consider.

With a grouping statement on multiple things, the current implementation combines all the result columns into a single STRING value. So the Kafka message's key contains all the grouping data, just in a nasty format. No additional column is being synthesised.

Conversely, for outer joins the Kafka message's key contains the result of COALESCE(leftJoinExp, rightJoinExp), i.e. unlike the grouping statement, it does NOT contain all the joining data: it loses the data about any side being null.

Because of this subtle difference we know that the upcoming structured keys work will allow users to alias the multiple grouping expressions in the projection. However, the issue with joins can not be fixed with structured key support alone.

If we go the same route for groupings as we've proposed for joins, then we end up with:

CREATE TABLE OUTPUT AS
   SELECT ROWKEY AS K, COUNT() FROM INPUT GROUP BY V0, V1;

-- vs --

CREATE TABLE OUTPUT AS
   SELECT V0, V1, COUNT() FROM INPUT GROUP BY V0, V1;

There's certainly arguments for going either way. Personally, I'm happy with what's been proposed for groupings in the KLIP.

cc @derekjn for a product view.

[mjsax]

I guess we agree that the end-state should be the last query you showed:

CREATE TABLE OUTPUT AS
   SELECT V0, V1, COUNT() FROM INPUT GROUP BY V0, V1;

(With a proper structured key <V0,V1> stored in the message key).

I guess the question is what intermediate state we want to be in. The query from above is a valid query now however, it does not really expose the PK that is stored in the ROWKEY.

From the KLIP:

we propose that the projection should still accept the individual columns, and recognise them as key columns

Not sure if I can follow. If both columns V0 and V1 are store in the value, how can this be done?

As this KLIP seems to try to expose the actual message key in the schema, it seems consequent to add ROWKEY for this case as an intermediate step. I understand, that it might look like a step backward for a language POV as the above query that is valid now, would not be valid until we reach the end-state and it becomes valid again...

[big-andy-coates]

To be clear the KLIP is proposing that the following will be valid even before structured keys:

CREATE TABLE OUTPUT AS
   SELECT V0, V1, COUNT() FROM INPUT GROUP BY V0, V1;

And both V0 and V1 will be stored in the key in a munged together column called KSQL_COL_0, not in the value.

Yes, we could support:

CREATE TABLE OUTPUT AS
   SELECT KSQL_COL_0, COUNT() FROM INPUT GROUP BY V0, V1;

But sures will be left wondering where KSQL_COL_0 came from.

Given than we're adding support for multiple key columns next, product are happy with not supporting aliasing of the resulting key column name.

[mjsax]

Thanks for the KLIP!

[S-makes]

design-proposals/klip-24-key-column-semantics-in-queries.md