Merge pull request #1 from mmolimar/develop

Release version 0.2.0

.circleci/config.yml

@@ -1,4 +1,4 @@
-version: 2
+version: 2.1
 jobs:
   build:
     docker:
@@ -16,7 +16,15 @@ jobs:
           keys:
             - v1-dependencies-{{ checksum "build.sbt" }}
             - v1-dependencies-
-      - run: cat /dev/null | sbt clean kukulcan publishLocal
+      - run:
+          name: Compile and test
+          command: sbt clean compile test
+      - run:
+          name: Code coverage
+          command: sbt jacocoAggregate jacocoAggregateReport
+      - run:
+          name: Build and publish
+          command: sbt kukulcan publishLocal
       - save_cache:
           paths:
             - ~/.m2

.gitignore

@@ -25,10 +25,12 @@ build
 libs
 venv
 null
+out
 tmp
 temp
 test-output
 build.log
+sbt.json
 python/*.egg-info
 python/setup.cfg
 __pycache__

README.md

@@ -1,10 +1,11 @@
-# kukulcan
+# kukulcan [![Build Status](https://circleci.com/gh/mmolimar/kukulcan.svg?style=shield)](https://circleci.com/gh/mmolimar/kukulcan)
 
-*K'uk'ulkan* ("Feathered Serpent") is the name of a deity which was workshipped by the Yucatec maya people. You can
+*K'uk'ulkan* ("Feathered Serpent") is the name of a deity which was worshipped by the Yucatec maya people. You can
 read a lot more in books or on the Internet about it and will see that, in someways, is related to the wind and water.
 
-Besides the origin of this name I reused to name this project, Kukulcan provides an API and different
-sort of [REPLs](https://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop) to interact with your streams or administer your [Apache Kafka](https://kafka.apache.org) deployment.
+Beyond the origin of this name I reused to name this project, Kukulcan provides an API and different
+sort of [REPLs](https://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop) to interact with your streams 
+or administer your [Apache Kafka](https://kafka.apache.org) deployment.
 
 It supports POSIX and Windows OS and Scala, Java and Python programming languages.
 
@@ -25,7 +26,7 @@ of its rich API plus some additional utils included and make your interaction be
 ### Requirements
 
 Before starting, you'll need to have installed JDK 11 and [SBT](https://www.scala-sbt.org/). Additionally,
-if you want to use the PyKukulcan REPL, you'll also need Python (supported 2.7, 3.4, 3.5, 3.6, 3.7 and 3.8 versions)
+if you want to use the PyKukulcan REPL, you'll also need Python (tested 2.7, 3.4, 3.5, 3.6, and 3.7 versions)
 and [pip](https://pypi.org/project/pip) installed.
 
 ### Building from source
@@ -39,7 +40,7 @@ Just clone the ``kukulcan`` repo and *"kukulcan-it"*:
 ### Configuration
 
 In the ``config`` directory, you'll find some files with the configurations for each of the APIs Kukulcan provides.
-All of the possible configs for each API are in these files with their description so, if you need to set some
+All the possible configs for each API are in these files with their description so, if you need to set some
 specific configs for your environment, you should set them there before starting.
 
 Also, there is an important environment variable named **``KUKULCAN_HOME``**. If not set, its default value will be
@@ -57,17 +58,19 @@ Contains all the Scala and Java classes to interact with Kafka and extending the
 functionalities. This API contains:
 
 * **KAdmin**: grouped utils for administrative operations for topics, configs, ACLs and metrics.
-* **KConnect**: methods to execute request agains Kafka Connect.
+* **KConnect**: methods to execute requests against Kafka Connect REST API.
 * **KConsumer** and **KProducer**: Kafka consumer/producer with some extra features.
 * **KStreams**: extends Kafka Streams to see how your topology is (printing it in a graph).
+* **KSchemaRegistry**: to interact and manage schemas in [Confluent Schema Registry](https://github.com/confluentinc/schema-registry).
+* **KKsql**: client for querying [Confluent KSQL](https://github.com/confluentinc/ksql) server and integrated with the KSQL-CLI.
 
 ### kukulcan-repl
 
 Enables two sort of entry points for the REPLs: one based on the [Scala REPL](https://docs.scala-lang.org/overviews/repl/overview.html)
 and the other based on [JShell](https://docs.oracle.com/javase/9/jshell).
 
 Additionally, it includes the logic to read and ``reload`` your configurations stored the ``config`` directory
-in a transparent way.
+transparently.
 
 ### pykukulcan
 
@@ -90,6 +93,14 @@ you can do with Kukulcan:
 
   ![](/docs/img/kconnect.png)
 
+- Managing schemas in Schema Registry:
+
+  ![](/docs/img/kschema-registry.png)
+
+- Interacting with KSQL server and using the KSQL-CLI:
+
+  ![](/docs/img/kksql.png)
+
 > **NOTE**: The REPLs have already the Kukulcan imports: ``com.github.mmolimar.kukulcan`` in case of the
 Scala and Ammonite REPLs and ``com.github.mmolimar.kukulcan.Kukulcan`` in case of the JShell REPL. So you just
 have to start typing ``kukulcan.<option>.`` or ``Kukulcan.<option>.`` respectively.
@@ -100,7 +111,7 @@ have to start typing ``kukulcan.<option>.`` or ``Kukulcan.<option>.`` respective
 [Ammonite](https://ammonite.io) is an improved Scala REPL with a lot of interesting features you can find very
 useful to load scripts or even code in an easier way. Obviously, you must install it previously.
 
-It already includes all dependencies and the Kukulcan Scala API to interact with Kafka.
+It already includes all dependencies, and the Kukulcan Scala API to interact with Kafka.
 
 > **NOTE**: If you're going to use Kukulcan with Ammonite, you'll have to publish the project in local, executing
   ``sbt publishLocal``.
@@ -138,7 +149,7 @@ For Windows OS:
 
 ### PyKukulcan REPL
 
-A Python shell including the needed bindings with Kukulcan Scala API via [Py4J](https://www.py4j.or).
+A Python shell including the needed bindings with Kukulcan Scala API via [Py4J](https://www.py4j.org).
 
 > **NOTE**: If you're going to use the PyKukulcan REPL, you must have Python and pip installed. Then, install
 ``pykukulcan`` in this way: ``pip install python/ -r python/requirements.txt``
@@ -151,12 +162,10 @@ For Windows OS:
 
 ``bin\pykukulcan.cmd``
 
-## TODO's
+## TODOs
 
-- [ ] Tools in the Admin API.
-- [ ] Integration with Schema Registry.
+- [ ] Tools in the KAdmin API.
 - [ ] Integration with REST Proxy.
-- [ ] Integration with ksqlDB.
 - [ ] API for Python.
 
 ## Contribute

bin/jkukulcan

@@ -14,6 +14,6 @@ done
 
 eval exec "\"$JAVA_EXEC\"" ${KUKULCAN_OPTS} -cp "\"${KUKULCAN_CLASSPATH}\"" \
   com.github.mmolimar.kukulcan.repl.JKukulcanRepl \
-  --class-path "\"$(find "$(echo $KUKULCAN_CLASSPATH  | sed 's/\/\*//g')" | tr '\n' ':')\""
+  --class-path "\"$(find "$(echo "$KUKULCAN_CLASSPATH"  | sed 's/\/\*//g')" | tr '\n' ':')\""
 
 exit $?

bin/kukulcan-amm

@@ -1,6 +1,6 @@
 #!/usr/bin/env amm
 
-val kukulcanVersion = "0.1.0"
+val kukulcanVersion = "0.2.0"
 val predefCode =
   s"""
      |import coursierapi.MavenRepository
@@ -9,7 +9,8 @@ val predefCode =
      |  List(
      |  coursierapi.Repository.ivy2Local,
      |  coursierapi.Repository.central,
-     |  MavenRepository.of("https://packages.confluent.io/maven/"))
+     |  MavenRepository.of("https://packages.confluent.io/maven/"),
+     |  MavenRepository.of("https://jitpack.io/"))
      |)
      |@
      |

build.sbt

@@ -1,33 +1,52 @@
-val projectVersion = "0.1.0"
+val projectVersion = "0.2.0"
 
 val repos = Seq(
   "Confluent Maven Repo" at "https://packages.confluent.io/maven/",
+  "jitpack" at "https://jitpack.io",
   Resolver.mavenLocal
 )
 
 lazy val settings = new {
-  val projectScalaVersion = "2.12.11"
+  val projectScalaVersion = "2.12.13"
 
   val dependencies = new {
-    val kafkaVersion = "2.5.0"
-    val kafkaConnectClientVersion = "3.1.0"
-    val circeVersion = "0.12.3"
     val asciiGraphsVersion = "0.0.6"
+    val circeVersion = "0.13.0"
+    val confluentVersion = "6.1.0"
+    val kafkaVersion = "2.7.0"
+    val kafkaConnectClientVersion = "3.1.0"
+
+    val scalaTestVersion = "3.2.5"
 
+    val exclusions = ExclusionRule(organization = "org.apache.kafka")
     val api = Seq(
+      "org.scala-lang" % "scala-compiler" % projectScalaVersion,
       "org.apache.kafka" %% "kafka" % kafkaVersion,
       "org.apache.kafka" % "kafka-clients" % kafkaVersion,
       "org.apache.kafka" % "kafka-tools" % kafkaVersion,
       "org.apache.kafka" %% "kafka-streams-scala" % kafkaVersion,
       "org.apache.kafka" % "kafka-streams-test-utils" % kafkaVersion,
-      "org.scala-lang" % "scala-compiler" % projectScalaVersion,
       "org.sourcelab" % "kafka-connect-client" % kafkaConnectClientVersion,
       "io.circe" %% "circe-core" % circeVersion,
       "io.circe" %% "circe-generic" % circeVersion,
       "io.circe" %% "circe-parser" % circeVersion,
-      "com.github.mutcianm" %% "ascii-graphs" % asciiGraphsVersion
+      "io.confluent" % "kafka-schema-registry-client" % confluentVersion excludeAll exclusions,
+      "io.confluent" % "kafka-json-schema-provider" % confluentVersion excludeAll exclusions,
+      "io.confluent" % "kafka-protobuf-provider" % confluentVersion excludeAll exclusions,
+      "io.confluent.ksql" % "ksqldb-cli" % confluentVersion excludeAll exclusions,
+      "com.github.mutcianm" %% "ascii-graphs" % asciiGraphsVersion,
+
+      "org.scalatest" %% "scalatest-wordspec" % scalaTestVersion % Test,
+      "org.scalatest" %% "scalatest-shouldmatchers" % scalaTestVersion % Test,
+      "org.apache.kafka" % "connect-runtime" % kafkaVersion % Test,
+      "org.apache.kafka" % "connect-file" % kafkaVersion % Test,
+      "io.github.embeddedkafka" %% "embedded-kafka-schema-registry" % confluentVersion % Test excludeAll exclusions,
+      "io.confluent.ksql" % "ksqldb-rest-app" % confluentVersion % Test excludeAll exclusions
+    )
+    val repl = Seq(
+      "org.scalatest" %% "scalatest-wordspec" % scalaTestVersion % Test,
+      "org.scalatest" %% "scalatest-shouldmatchers" % scalaTestVersion % Test
     )
-    val repl = Seq.empty
     val root = Seq.empty
   }
   val common = Seq(
@@ -54,11 +73,18 @@ lazy val settings = new {
     packExpandedClasspath := true,
     packGenerateMakefile := false,
     publish / skip := true,
-    libraryDependencies ++= dependencies.root
+    libraryDependencies ++= dependencies.root,
+    jacocoAggregateReportSettings := JacocoReportSettings(
+      title = "Kukulcan Project Code Coverage"
+    )
   )
   val api = Seq(
     name := "kukulcan-api",
-    libraryDependencies ++= dependencies.api
+    libraryDependencies ++= dependencies.api,
+    parallelExecution in Test := false,
+    jacocoAggregateReportSettings := JacocoReportSettings(
+      title = "Kukulcan API Module Code Coverage"
+    )
   )
   val repl = Seq(
     name := "kukulcan-repl",
@@ -78,7 +104,10 @@ lazy val settings = new {
     javacOptions ++= Seq(
       "--add-exports=jdk.jshell/jdk.internal.jshell.tool=ALL-UNNAMED"
     ),
-    libraryDependencies ++= dependencies.repl
+    libraryDependencies ++= dependencies.repl,
+    jacocoAggregateReportSettings := JacocoReportSettings(
+      title = "Kukulcan REPL Module Code Coverage"
+    )
   )
   val pykukulcan = Seq(
     name := "pykukulcan"
@@ -108,6 +137,7 @@ lazy val root = project
   .dependsOn(apiProject, replProject)
   .enablePlugins(PackPlugin)
   .enablePlugins(KukulcanPackPlugin)
+  .enablePlugins(JacocoPlugin)
   .settings(
     settings.common,
     settings.root