Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

push queries should start from beginning #3579

Open
big-andy-coates opened this issue Oct 15, 2019 · 1 comment
Open

push queries should start from beginning #3579

big-andy-coates opened this issue Oct 15, 2019 · 1 comment

Comments

@big-andy-coates
Copy link
Contributor

big-andy-coates commented Oct 15, 2019

As decided in KLIP-8, continuous queries on streams should start from the beginning of the stream, not the current offsets. This causes a lot of confusion for people.

As part of this work we should disable/ignore anyone issuing SET 'auto.offset.reset'='earliest' or maybe just output a warning that this is no longer needed / heeded? We should also remove above from any docs

We should look to provide some construct to allow users to query the stream 'from now', i.e. old functionality. Some kind of WHERE offsets > current_offsets() or similar.

@big-andy-coates big-andy-coates mentioned this issue Oct 15, 2019
27 tasks
@big-andy-coates big-andy-coates added this to To do in Pull Queries via automation Oct 25, 2019
@mjsax
Copy link
Member

mjsax commented Nov 14, 2019

I am +1 to this proposal. A query over a stream should always process the whole stream -- auto.offset.reset is very Kafka specific and it would be good to not use it in KSQL.

KLIP-11 includes a similar proposal. Note, that KLIP-11 proposes to handle streams and tables differently, because table records conceptually don't have an offset (the offset of records in a changelog topic are an implementation detail that should not leak into the query language).

@big-andy-coates big-andy-coates changed the title Pull Queries: continuous queries should start from beginning Pull Queries: push queries should start from beginning Nov 21, 2019
@big-andy-coates big-andy-coates moved this from To do to Future Work? in Pull Queries Nov 25, 2019
@big-andy-coates big-andy-coates changed the title Pull Queries: push queries should start from beginning push queries should start from beginning Apr 20, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Pull Queries
  
Backlog
Development

No branches or pull requests

2 participants