KSQL - the Streaming SQL Engine for Apache Kafka
Switch branches/tags
v6.0.0-beta181011024526 v6.0.0-beta181009071136 v6.0.0-beta181009071126 v6.0.0-beta181009070836 v5.1.0-beta201806200051 v5.1.0-beta201806191956 v5.1.0-beta39 v5.1.0-beta38 v5.1.0-beta37 v5.1.0-beta36 v5.1.0-beta35 v5.1.0-beta34 v5.1.0-beta180929024526 v5.1.0-beta180928000756 v5.1.0-beta180919183606 v5.1.0-beta180917172706 v5.1.0-beta180912202326 v5.1.0-beta180911213206 v5.1.0-beta180905054157 v5.1.0-beta180829024526 v5.1.0-beta180828173516 v5.1.0-beta180828022857 v5.1.0-beta180824214446 v5.1.0-beta180820223106 v5.1.0-beta180812233046 v5.1.0-beta180730185716 v5.1.0-beta180724024536 v5.1.0-beta180723173636 v5.1.0-beta180723023347 v5.1.0-beta180722215127 v5.1.0-beta180718203536 v5.1.0-beta180707004950 v5.1.0-beta180706202701 v5.1.0-beta180703024529 v5.1.0-beta180702220040 v5.1.0-beta180702214311 v5.1.0-beta180702063440 v5.1.0-beta180702063039 v5.1.0-beta180701175749 v5.1.0-beta180701010040 v5.1.0-beta180630224439 v5.1.0-beta180628184841 v5.1.0-beta180628064520 v5.1.0-beta180627203509 v5.1.0-beta180626014959 v5.1.0-beta180622181348 v5.1.0-beta180620183559 v5.1.0-beta180620180739 v5.1.0-beta180620180431 v5.1.0-beta180619025141 v5.1.0-beta180618225004 v5.1.0-beta180618223247 v5.1.0-beta180618214711 v5.1.0-beta180618191747 v5.1.0-beta180615005408 v5.1.0-beta180614233101 v5.1.0-beta180613013021 v5.1.0-beta180612224009 v5.1.0-beta180612043613 v5.1.0-beta180611231144 v5.0.1-rc1 v5.0.1-beta181018200736 v5.0.1-beta181008233543 v5.0.1-beta180914024526 v5.0.1-beta180913003126 v5.0.1-beta180911213156 v5.0.1-beta180909000436 v5.0.1-beta180909000146 v5.0.1-beta180905054336 v5.0.1-beta180902210116 v5.0.1-beta180830182727 v5.0.1-beta180828173436 v5.0.1-beta180826190446 v5.0.1-beta180824214627 v5.0.1-beta180812233236 v5.0.1-beta180802235906 v5.0.0 v5.0.0-rc4 v5.0.0-rc3 v5.0.0-rc1 v5.0.0-beta33 v5.0.0-beta32 v5.0.0-beta31 v5.0.0-beta30 v5.0.0-beta29 v5.0.0-beta28 v5.0.0-beta27 v5.0.0-beta26 v5.0.0-beta25 v5.0.0-beta24 v5.0.0-beta23 v5.0.0-beta22 v5.0.0-beta21 v5.0.0-beta20 v5.0.0-beta19 v5.0.0-beta18 v5.0.0-beta17 v5.0.0-beta16 v5.0.0-beta15 v5.0.0-beta14
Nothing to show
Clone or download
rodesai Rm ksql-examples, cp-ksql-server, and cp-ksql-cli from ksql (#2059)
Remove the ksql-examples, cp-ksql-server, and cp-ksql-cli docker image
builds from the ksql repo.
Latest commit 035ed00 Oct 20, 2018
Permalink
Failed to load latest commit information.
.github Also add ksql team for docs (#1827) Sep 4, 2018
bin MINOR: Cause 'ksql help' and 'ksql -help' to behave the same as 'ksql… Oct 12, 2018
build-tools Bump Confluent to 5.2.0-SNAPSHOT, Kafka to 2.2.0-SNAPSHOT Oct 9, 2018
checkstyle Add query status event to jmx (#1747) Sep 17, 2018
config MINOR: Clean up some unused / unnecessary props (#1763) Sep 10, 2018
debian Add WantedBy dependency to ensure startup order (#893) Mar 9, 2018
design-proposals Add ability to submit design proposals aka (KLIPS) on github (#1235) Jun 20, 2018
docker Set the default server port to 8088. (#930) Mar 14, 2018
docs Merge branch '5.1.x' Oct 19, 2018
ext Reduce log level when loading blacklist to INFO from ERROR (#1522) Jul 4, 2018
findbugs Have findbugs run as part of the build (#1498) Jun 27, 2018
ksql-cli Added stringtodate and datetostring UDFs (#1890) Oct 15, 2018
ksql-clickstream-demo Bump Confluent to 5.2.0-SNAPSHOT, Kafka to 2.2.0-SNAPSHOT Oct 9, 2018
ksql-common Fix session windows (#2024) Oct 17, 2018
ksql-console-scripts Add ksql-etc module (#2047) Oct 13, 2018
ksql-engine Re-enable fail-on-deprecation (#2034) Oct 19, 2018
ksql-etc Add ksql-etc module (#2047) Oct 13, 2018
ksql-examples Rm ksql-examples, cp-ksql-server, and cp-ksql-cli from ksql (#2059) Oct 20, 2018
ksql-metastore Fix session windows (#2024) Oct 17, 2018
ksql-package Bump Confluent to 5.2.0-SNAPSHOT, Kafka to 2.2.0-SNAPSHOT Oct 9, 2018
ksql-parser Fix session windows (#2024) Oct 17, 2018
ksql-rest-app Fix session windows (#2024) Oct 17, 2018
ksql-serde Bump Confluent to 5.2.0-SNAPSHOT, Kafka to 2.2.0-SNAPSHOT Oct 9, 2018
ksql-test-util Fix session windows (#2024) Oct 17, 2018
ksql-tools Bump Confluent to 5.2.0-SNAPSHOT, Kafka to 2.2.0-SNAPSHOT Oct 9, 2018
ksql-udf Bump Confluent to 5.2.0-SNAPSHOT, Kafka to 2.2.0-SNAPSHOT Oct 9, 2018
ksql-version-metrics-client Re-enable fail-on-deprecation (#2034) Oct 19, 2018
licenses Bump Confluent to 5.2.0-SNAPSHOT, Kafka to 2.2.0-SNAPSHOT Oct 9, 2018
notices Starting a clean ksql repo by removing all the previous commit history. Aug 16, 2017
.github_changelog_generator Add ability to autogenerate change logs (#692) Feb 3, 2018
.gitignore extract ksql experimental ui (#863) Mar 7, 2018
CHANGELOG.md Update CHANGELOG with 4.1.0 information (#1274) May 4, 2018
CONTRIBUTING.md Document code style and static checks. (#1655) Jul 27, 2018
Jenkinsfile Rm ksql-examples, cp-ksql-server, and cp-ksql-cli from ksql (#2059) Oct 20, 2018
LICENSE Starting a clean ksql repo by removing all the previous commit history. Aug 16, 2017
PULL_REQUEST_TEMPLATE.md Fix typo Sep 25, 2018
README.md Merge branch '5.0.x' Aug 30, 2018
ksq-lrocket.png Resize rocket Aug 24, 2017
pom.xml Rm ksql-examples, cp-ksql-server, and cp-ksql-cli from ksql (#2059) Oct 20, 2018
screencast.jpg KSQL screencast image Aug 30, 2017

README.md

KSQL rocket KSQL - Streaming SQL for Apache Kafka

KSQL is now GA and officially supported by Confluent Inc. Get started with KSQL today.

KSQL is the streaming SQL engine for Apache Kafka.

KSQL is an open source streaming SQL engine for Apache Kafka. It provides a simple and completely interactive SQL interface for stream processing on Kafka; no need to write code in a programming language such as Java or Python. KSQL is open-source (Apache 2.0 licensed), distributed, scalable, reliable, and real-time. It supports a wide range of powerful stream processing operations including aggregations, joins, windowing, sessionization, and much more.

Click here to watch a screencast of the KSQL demo on YouTube. KSQL screencast

Getting Started and Download

Stable Releases

Stable releases are published every four months and are officially supported by Confluent.

  1. Download latest stable KSQL, which is included in the Enterprise and Open Source editions of Confluent Platform.
  2. Follow the Quick Start.
  3. Read the KSQL Documentation, notably the KSQL Tutorials and Examples, which include Docker-based variants.

Preview Releases

In addition to supported stable KSQL releases, we also provide monthly preview releases. We encourage you to try them in development and testing environments and to take advantage of Confluent Community resources to get help and share feedback.

Documentation

See KSQL documentation for the latest stable release.

Use Cases and Examples

Streaming ETL

Apache Kafka is a popular choice for powering data pipelines. KSQL makes it simple to transform data within the pipeline, readying messages to cleanly land in another system.

CREATE STREAM vip_actions AS
  SELECT userid, page, action
  FROM clickstream c
  LEFT JOIN users u ON c.userid = u.user_id
  WHERE u.level = 'Platinum';

Anomaly Detection

KSQL is a good fit for identifying patterns or anomalies on real-time data. By processing the stream as data arrives you can identify and properly surface out of the ordinary events with millisecond latency.

CREATE TABLE possible_fraud AS
  SELECT card_number, count(*)
  FROM authorization_attempts
  WINDOW TUMBLING (SIZE 5 SECONDS)
  GROUP BY card_number
  HAVING count(*) > 3;

Monitoring

Kafka's ability to provide scalable ordered messages with stream processing make it a common solution for log data monitoring and alerting. KSQL lends a familiar syntax for tracking, understanding, and managing alerts.

CREATE TABLE error_counts AS
  SELECT error_code, count(*)
  FROM monitoring_stream
  WINDOW TUMBLING (SIZE 1 MINUTE)
  WHERE  type = 'ERROR'
  GROUP BY error_code;

Latest News

Join the Community

You can get help, learn how to contribute to KSQL, and find the latest news by connecting with the Confluent community.

Contributing

Contributions to the code, examples, documentation, etc. are very much appreciated.

License

The project is licensed under the Apache License, version 2.0.

Apache, Apache Kafka, Kafka, and associated open source project names are trademarks of the Apache Software Foundation.