Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Joined streams: pair with the most recent update only #4611

Open
ShahOdin opened this issue Feb 23, 2020 · 1 comment
Open

Joined streams: pair with the most recent update only #4611

ShahOdin opened this issue Feb 23, 2020 · 1 comment
Labels

Comments

@ShahOdin
Copy link

ShahOdin commented Feb 23, 2020

I have a stream join as below:

CREATE STREAM FOO AS
  SELECT c.title, s.fromInstant, s.toInstant
  FROM CATALOGUE_STREAM c
  FULL OUTER JOIN SCHEDULE_STREAM s WITHIN 48 HOURS
  ON s.ROWKEY = c.ROWKEY
  EMIT CHANGES;

where SCHEDULE_STREAM and CATALOGUE_STREAM are streams based on my kafka topics.
and since I'd potentially get multiple messages from SCHEDULE_STREAM and CATALOGUE_STREAM for a given key, I tried the following to match it only with the latest update:

CREATE STREAM BAR AS
  SELECT A1.*
  FROM FOO AS F1 LEFT JOIN FOO AS F2
  ON (F1.ROWKEY = F2.ROWKEY AND F1.ROWTIME < F2.ROWTIME)    
  WHERE F2.ROWTIME IS NULL
  EMIT CHANGES;

see here for an explanation of the idea used here or see the following gist

but I get:

io.confluent.ksql.execution.expression.tree.LogicalBinaryExpression cannot be cast to io.confluent.ksql.execution.expression.tree.ComparisonExpression

  • Is this a conscious decision on the part of ksqldb?
  • What are the alternative ways of achieving this?
  • I further tried:
CREATE STREAM BAZ AS
  SELECT c.title, s.fromInstant, s.toInstant
  FROM CATALOGUE_TABLE c
  FULL OUTER JOIN SCHEDULE_TABLE s
  ON s.ROWKEY = c.ROWKEY
  EMIT CHANGES;

where SCHEDULE_TABLE and CATALOGUE_TABLE are tables based on my kafka topics. Unfortunately I got:

Invalid result type. Your SELECT query produces a TABLE. Please use CREATE TABLE AS SELECT statement instead.

@ShahOdin
Copy link
Author

ShahOdin commented Feb 23, 2020

just came across #3985 which covers a more general discussion of this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant