Headless KSQL to support schema inference (Avro)

[miguno]

Schema inference doesn't work when using headless KSQL. For example, when your input data is in Avro format, you must still enter column names/types manually (when KSQL is run in headless config).

The problem with headless (non-interactive mode) is that the engine parses all the queries in the set up front, and fails if any of the fields are not found in the meta store. The schema inference is done during execution, which happens after parsing. So if a subsequent query refers to an inferred field, it will fail with the parse exception.

• The workaround is to manually specify the columns.
• The solution is to parse and execute queries one by one in all modes.

Related Issues

We also have other issues because of this dichotomy between the way our interactive and non-interactive modes operate.

• KSQL in headless mode doesn't do the same validation on input queries as the interactive mode #1320 (KSQL in headless mode doesn't do the same validation on input queries as the interactive mode)
• Avro limitation for commands in RUN SCRIPT vs. interactive KSQL usage #1031 (Avro limitation for commands in run script versus interactive)

[miguno]

cc @ybyzek @rmoff

[rmoff]

This is an important thing for us to fix to enable people to painlessly move from dev through to production.

[big-andy-coates]

+1 from me - this is a biggie for production.

[yunhappy]

1. +1 from me ,
2. offset set to earliest can not work ?

[miguno]

@yunhappy : Please upvote (+1 / thumbs-up) the first message/comment in this thread, this way we can track the number of upvotes more easily.

[big-andy-coates]

This is in direct conflict with issue #2205.

The issue with supporting avro field inference in headless mode is that the schema could change over time. So we might start KSQL only to find the schema has changed, with columns added / removed. How should KSQL handle such a change?

Currently, in interactive mode, the schema is queried once and then baked into the command topic. We could support something similar for headless. I know @rodesai is looking to store other system metadata in a topic in headless mode. Maybe this could be too.

Alternatively, we can look at supporting schema evolution in both headless and interactive. But this is a larger piece of work and probably a more long term goal / feature.

As for #2205, I see that as just a quick 'make KSQL fail hard and fast' until we get around to adding true support for this, i.e. fix the fact it currently parses OK, but doesn't actually work.

[dhanasgit]

+1

[miguno]

Closing this issue. If there are future requests similar to this, please reopen or create a new issue.