Feature: Support KSqlDb driver

[andreashimin]

Description

Support KSqlDb driver to connect KSqldb data source

Issue ticket number

closes N/A

How To Test

Follow the official site quick start step below:
https://ksqldb.io/quickstart.html

or run the cmd line in the root directory:

• start the docker containers in docker-compose.yml

docker-compose -f ./packages/extension-driver-ksqldb/test/docker/docker-compose.yml up -d

• open db terminal

docker exec -it ksqldb-cli ksql http://ksqldb-server:8088

• create & insert reference data

CREATE STREAM riderLocations (profileId VARCHAR, latitude DOUBLE, longitude DOUBLE)
  WITH (kafka_topic='locations', value_format='json', partitions=1);

CREATE TABLE currentLocation AS
  SELECT profileId,
         LATEST_BY_OFFSET(latitude) AS la,
         LATEST_BY_OFFSET(longitude) AS lo
  FROM riderlocations
  GROUP BY profileId
  EMIT CHANGES;

CREATE TABLE ridersNearMountainView AS
  SELECT ROUND(GEO_DISTANCE(la, lo, 37.4133, -122.1162), -1) AS distanceInMiles,
         COLLECT_LIST(profileId) AS riders,
         COUNT(*) AS count
  FROM currentLocation
  GROUP BY ROUND(GEO_DISTANCE(la, lo, 37.4133, -122.1162), -1);

INSERT INTO riderLocations (profileId, latitude, longitude) VALUES ('c2309eec', 37.7877, -122.4205);
INSERT INTO riderLocations (profileId, latitude, longitude) VALUES ('18f4ea86', 37.3903, -122.0643);
INSERT INTO riderLocations (profileId, latitude, longitude) VALUES ('4ab5cbad', 37.3952, -122.0813);
INSERT INTO riderLocations (profileId, latitude, longitude) VALUES ('8b6eae59', 37.3944, -122.0813);
INSERT INTO riderLocations (profileId, latitude, longitude) VALUES ('4a7c7b41', 37.4049, -122.0822);
INSERT INTO riderLocations (profileId, latitude, longitude) VALUES ('4ddad000', 37.7857, -122.4011);

• In vulcan.yaml, add the extension below

extensions:
  ksqldb: '@vulcan-sql/extension-driver-ksqldb'

• In profile.yaml, add the configuration below

- name: ksql
  type: ksqldb
  connection:
    host: http://localhost:8088
    username: <username>
    password: <password>
    timeout: 25000
  allow: '*'

test.sql

create stream outputEventStream as select *, rowtime from riderLocations where latitude > {{context.params.latitude}};

test.yaml

urlPath: /test
request:
  - fieldName: latitude
    fieldIn: query
    description: lang scope
sample:
  profile: ksql
  parameters:
    latitude: 38
profile: ksql

Additional Context

The KsqlDb currently has some limitations.

• Not support sub query Support for Subqueries confluentinc/ksql#745, pagination may not supported
• Not support offset Limit and Offset and in PULL queries confluentinc/ksql#7790

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
vulcan-sql-document	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jul 31, 2023 2:16am

[codecov-commenter]

Codecov Report

Patch coverage: 89.53% and project coverage change: -0.04% ⚠️

Comparison is base (bf4b1a2) 90.25% compared to head (03a2ffa) 90.21%.
Report is 23 commits behind head on develop.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #255      +/-   ##
===========================================
- Coverage    90.25%   90.21%   -0.04%     
===========================================
  Files          340      346       +6     
  Lines         5694     5930     +236     
  Branches       762      793      +31     
===========================================
+ Hits          5139     5350     +211     
- Misses         403      423      +20     
- Partials       152      157       +5     

Flag	Coverage Δ
extension-driver-ksqldb	89.49% <89.49%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
...xtension-driver-ksqldb/src/lib/ksqldbDataSource.ts	80.85% <80.85%> (ø)
...s/extension-driver-ksqldb/src/lib/restfulClient.ts	86.25% <86.25%> (ø)
...kages/extension-driver-ksqldb/test/ksqlDbServer.ts	87.50% <87.50%> (ø)
...ages/extension-driver-ksqldb/src/lib/sqlBuilder.ts	100.00% <100.00%> (ø)
...ages/extension-driver-ksqldb/src/lib/typeMapper.ts	100.00% <100.00%> (ø)
...ges/extension-driver-ksqldb/test/docker/compose.ts	100.00% <100.00%> (ø)
...gingface/src/lib/filters/tableQuestionAnswering.ts	80.55% <100.00%> (-1.03%)	⬇️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[kokokuo]

Beside some comment suggestion, others LGTM 👍