Another tool in the xkite library to give developers flexibility and control over their Kafka ecosystems.
- Easy to use Command Line Interface (CLI) to interact with the xkite-core library. It’s a simple alternative to the full xkite GUI, meant to provide users with the ability to automate their testing using scripts.
- Configure a custom docker ecosystem using command line inputs or a
config.jsonfile, deploy, pause/unpause, shutdown and remove containers and volumes. - Export the docker ecosystem configuration to a package.zip file containing the docker-compose.yml file and dependencies needed to deploy your kafka ecosystem to any cloud instance or remote server. Simply unzip and run
docker-compose up. - Interface with the xkite GUI server as a remote client. You can perform all the aforementioned functions provided remotely on the xkite GUI server by providing the URL as a configuration input.
- More granular control over which docker images you choose to deploy (currently unconfigurable with the xkite GUI).
The latest stable versions of:
-
Clone this Repository
git clone https://github.com/oslabs-beta/xkite-cli.git -
Install Dependencies
cdinto the cloned repository and runnpm install
To install the xkite-cli into your environment just simply run the following command:
$ npx create-xkite-cli <directory-name>After the installation is complete, you will be able to run xkite-cli as follows:
$ xkite-cliThe xkite-cli interfaces with the xkite-core library allowing the user to configure a YAML file, managing docker containers (configure, run, pause, and shutdown), interfacing with remote xkite servers, and providing configuration settings for users to easily connect to their Kafka instances for development purposes.
To learn more about xkite-core please follow the link: xkite-core library
Click to expand details.
-h | --help
Displays all the valid options for xkite-cli utility.
$ xkite-cli --help
_ _ _
__ _| | _(_) |_ ___
\ \/ / |/ / | __/ _ \
> <| <| | || __/
/_/\_\_|\_\_|\__\___|
Usage: xkite-cli [options]
CLI for xkite, an Apache Kafka Prototype and Test Tool
Options:
-V, --version output the version number
-s, --server <server:port> connect to an xkite server: i.e xkite-cli -s http://localhost:3000
-i, --input <value> Input configuration file for xkite
-b, --broker <# of brokers> <# of replicas> <port1,...,portn> <1 = enable jmx> Kafka broker setup (default to 1 if not chosen)
-z, --zookeeper <# of zookeepers> <port1,...,portn> Zookeeper setup (default to 1 if not chosen)
-db, --database <type: ksql | postgresql> <port> Creates Source Database (default none if not chosen)
-sk, --sink <type: jupyter | spark> <port> Creates Sink at specified port (default none if not chosen)
-gr, --grafana <port> Creates Grafana instance at specified port (default none if not chosen
-pr, --prometheus <port> <scrape_interval> <evaluation_interval> Creates Prometheus instance at specified port with settings for scrape and eval interval (in seconds)
-r --run Runs configured docker instances
-p --pause Pauses active docker instances
-q --quit Shuts down all docker instances and removes associated container/volumes
-h, --help display help for command-V | --version
Displays the version of xkite-cli utility.
$ xkite-cli -V
_ _ _
__ _| | _(_) |_ ___
\ \/ / |/ / | __/ _ \
> <| <| | || __/
/_/\_\_|\_\_|\__\___|
1.0.5-s | --server server:port
Connect to a remote xkite instance such as xkite-GUI.
$ xkite-cli -s http://localhost:3000
_ _ _
__ _| | _(_) |_ ___
\ \/ / |/ / | __/ _ \
> <| <| | || __/
/_/\_\_|\_\_|\__\___|
Kite remote server state = Connected-i | --input < cfg.json >
Configure a local instance of Kite using the input cfg.json. Note: an example file can be found under the reference directory.
Click to expand details.
cfg.json Format
Not all fields are required, those that are not are tagged as optional below. The user may choose to not inlude the optional fields for their application's needs. For more details on the KiteConfig format please see the xkite-core library.{
"kafka": {
"brokers": {
"size": 2,
"id": [101, 102], //optional - broker id
"replicas": 2, //optional - number of replicas
"ports": { //optional
"brokers": [9092, 9093], // optional - external broker ports
"metrics": 29092, // optional - confluent metric interface on docker net
"jmx": [9991, 9992]; // optional - broker interface with jmx on docker net
};
};
"zookeepers": {
"size": 1;
"ports": { //optional
"peer": { //optional
//does not need to be configurable, docker net only
"internal": 2888, //optional
"external": 3888, //optional
};
"client": [2181], //optional - external interface with zookeeper
};
};
"jmx": { //optional
"ports": [5566, 5567], //optional - external host port to interface with port
};
"spring": { //optional
"port": 8080; //optional - external host port to interface with 8080
};
},
"db": { // optional
"name": "ksql", // either "ksql" or "postgres"
"port": 8088, //optional - external port
"postgresql": { // optional
"username": "ADMIN",
"password": "ADMIN",
"dbname": "xkiteDB",
};
"ksql": { // optional
"schema_port": 8180, //optional
};
"kafkaconnect": { //optional
"port": 9000,
};
},
"sink": { //optional
"name": "spark", // either 'jupyter' | 'spark';
"port": 9191, //optional - external webgui interface port
"rpc_port": 7077, //optional
"kafkaconnect": { //optional
"port": 9001, // optional
};
},
"grafana": { //optional
"port": 3050, //optional
},
"prometheus": { //optional
"port": 9090, //optional
"scrape_interval": 10, //optional - seconds
"evaluation_interval": 5, //optional - seconds
},
}
Example:
$ xkite-cli -i reference/cfg.json
_ _ _
__ _| | _(_) |_ ___
\ \/ / |/ / | __/ _ \
> <| <| | || __/
/_/\_\_|\_\_|\__\___|
Configuring docker instances...
creating Kite Config yml...
creating zookeepers...
creating brokers...
yaml configuration complete...-r | --run
Deploys or unpauses the configured xkite instance depending on the Kite state. If the configuration is pointing to a remote server instance of xkite, it will be run on the remote server.
Example of deploying -->
$ xkite-cli -r
_ _ _
__ _| | _(_) |_ ___
\ \/ / |/ / | __/ _ \
> <| <| | || __/
/_/\_\_|\_\_|\__\___|
Deploying docker instances...
deploying docker containers...
Building with native build. Learn about native build in Compose here: https://docs.docker.com/go/compose-native-build/
Creating network "download_default" with the default driver
Creating volume "download_jupyterhub_data" with local driver
Creating volume "download_dashboards" with local driver
Creating volume "download_provisioning" with local driver
Creating volume "download_postgresql" with local driver
Creating postgresql ...
Creating prometheus ...
Creating zookeeper2 ...
Creating spark ...
Creating zookeeper1 ...
Creating spark ... done
Creating prometheus ... done
Creating grafana ...
Creating postgresql ... done
Creating zookeeper2 ... done
Creating zookeeper1 ... done
Creating kafka2 ...
Creating kafka1 ...
Creating grafana ... done
Creating kafka1 ... done
Creating jmx-kafka1 ...
Creating kafka2 ... done
Creating spring ...
Creating kafka_connect_sink ...
Creating kafka_connect_src ...
Creating jmx-kafka2 ...
Creating jmx-kafka2 ... done
Creating spring ... done
Creating jmx-kafka1 ... done
Creating kafka_connect_sink ... done
Creating kafka_connect_src ... done
$Example of unpausing -->
$ xkite-cli -r
_ _ _
__ _| | _(_) |_ ___
\ \/ / |/ / | __/ _ \
> <| <| | || __/
/_/\_\_|\_\_|\__\___|
Unpausing all docker instances...
Unpausing postgresql ...
Unpausing postgresql ... done
Unpausing kafka_connect_src ...
Unpausing kafka_connect_src ... done
Unpausing kafka_connect_sink ...
Unpausing kafka_connect_sink ... done
Unpausing spark ...
Unpausing spark ... done
Unpausing prometheus ...
Unpausing prometheus ... done
Unpausing grafana ...
Unpausing grafana ... done
Unpausing zookeeper1 ...
Unpausing zookeeper1 ... done
Unpausing zookeeper2 ...
Unpausing zookeeper2 ... done
Unpausing kafka1 ...
Unpausing kafka1 ... done
Unpausing jmx-kafka1 ...
Unpausing jmx-kafka1 ... done
Unpausing kafka2 ...
Unpausing kafka2 ... done
Unpausing jmx-kafka2 ...
Unpausing jmx-kafka2 ... done
Unpausing spring ...
Unpausing spring ... done
$-p | --pause
Deploys or unpauses the configured xkite instance depending on the Kite state. If the configuration is pointing to a remote server instance of xkite, it will be run on the remote server.
Example of pausing
$ xkite-cli -p
_ _ _
__ _| | _(_) |_ ___
\ \/ / |/ / | __/ _ \
> <| <| | || __/
/_/\_\_|\_\_|\__\___|
Pausing all docker instances...
Pausing postgresql ...
Pausing postgresql ... done
Pausing kafka_connect_src ...
Pausing kafka_connect_src ... done
Pausing kafka_connect_sink ...
Pausing kafka_connect_sink ... done
Pausing spark ...
Pausing spark ... done
Pausing prometheus ...
Pausing prometheus ... done
Pausing grafana ...
Pausing grafana ... done
Pausing zookeeper1 ...
Pausing zookeeper1 ... done
Pausing zookeeper2 ...
Pausing zookeeper2 ... done
Pausing kafka1 ...
Pausing kafka1 ... done
Pausing jmx-kafka1 ...
Pausing jmx-kafka1 ... done
Pausing kafka2 ...
Pausing kafka2 ... done
Pausing jmx-kafka2 ...
Pausing jmx-kafka2 ... done
Pausing spring ...
Pausing spring ... done
$-q | --quit
Shuts down any running xkite instances and removes all the associated container and volumes. If the configuration is pointing to a remote server instance of xkite, it will be run on the remote server.
$ xkite-cli -q
_ _ _
__ _| | _(_) |_ ___
\ \/ / |/ / | __/ _ \
> <| <| | || __/
/_/\_\_|\_\_|\__\___|
Shutting down docker instances and removing volumes...
Stopping spring ...
Stopping jmx-kafka2 ...
Stopping kafka_connect_src ...
Stopping kafka_connect_sink ...
Stopping jmx-kafka1 ...
Stopping kafka2 ...
Stopping kafka1 ...
Stopping grafana ...
Stopping prometheus ...
Stopping zookeeper1 ...
Stopping spark ...
Stopping zookeeper2 ...
Stopping postgresql ...
Stopping grafana ... done
Stopping postgresql ... done
Stopping spring ... done
Stopping prometheus ... done
Stopping jmx-kafka2 ... done
Stopping jmx-kafka1 ... done
Stopping kafka_connect_sink ... done
Stopping kafka_connect_src ... done
Stopping kafka2 ... done
Stopping spark ... done
Stopping kafka1 ... done
Stopping zookeeper2 ... done
Stopping zookeeper1 ... done
Removing spring ...
Removing jmx-kafka2 ...
Removing kafka_connect_src ...
Removing kafka_connect_sink ...
Removing jmx-kafka1 ...
Removing kafka2 ...
Removing kafka1 ...
Removing grafana ...
Removing prometheus ...
Removing zookeeper1 ...
Removing spark ...
Removing zookeeper2 ...
Removing postgresql ...
Removing jmx-kafka1 ... done
Removing jmx-kafka2 ... done
Removing spark ... done
Removing grafana ... done
Removing spring ... done
Removing kafka2 ... done
Removing prometheus ... done
Removing kafka_connect_src ... done
Removing kafka1 ... done
Removing kafka_connect_sink ... done
Removing postgresql ... done
Removing zookeeper2 ... done
Removing zookeeper1 ... done
Removing network download_default
Removing volume download_jupyterhub_data
Removing volume download_dashboards
Removing volume download_provisioning
Removing volume download_postgresql
$-o | --output < directory_path >
Provides the package.zip file containing the configuration needed to deploy the docker ecosystem on a cloud or external server instance. If the configuration is pointing to a remote server instance of xkite, it will be retrieved from the remote server.
$ xkite-cli -o .
_ _ _
__ _| | _(_) |_ ___
\ \/ / |/ / | __/ _ \
> <| <| | || __/
/_/\_\_|\_\_|\__\___|
Getting package build zip...
$ ls package.zip -lth
-rw-rw-r-- 1 user user 69M Mar 15 13:47 package.zip
$ unzip -l package.zip
Archive: package.zip
Length Date Time Name
--------- ---------- ----- ----
0 2023-03-15 13:47 config/
10605 2023-03-15 13:47 config/cfg.json
8736 2023-03-15 13:47 docker-compose.yml
0 2023-03-15 13:47 grafana/
0 2023-03-15 13:47 grafana/dashboards/
117425 2023-03-15 13:47 grafana/dashboards/kafka-metrics_rev4.json
0 2023-03-15 13:47 grafana/provisioning/
0 2023-03-15 13:47 grafana/provisioning/dashboards/
869 2023-03-15 13:47 grafana/provisioning/dashboards/metrics.yaml
0 2023-03-15 13:47 grafana/provisioning/datasources/
1532 2023-03-15 13:47 grafana/provisioning/datasources/datasource.yaml
0 2023-03-15 13:47 jmx/
0 2023-03-15 13:47 jmx/exporter/
24808 2023-03-15 13:47 jmx/exporter/template.yml
15914 2023-03-15 13:47 jmx/jmxConfigKafka1.yml
15914 2023-03-15 13:47 jmx/jmxConfigKafka2.yml
0 2023-03-15 13:47 kafkaconnect/
152 2023-03-15 13:47 kafkaconnect/Dockerfile
540 2023-03-15 13:47 kafkaconnect/README.md
0 2023-03-15 13:47 ksql/
2534 2023-03-15 13:47 ksql/testscript.sql
0 2023-03-15 13:47 postgresql/
1338 2023-03-15 13:47 postgresql/init.sql
0 2023-03-15 13:47 prometheus/
227 2023-03-15 13:47 prometheus/prometheus.yml
0 2023-03-15 13:47 spring/
75383987 2023-03-15 13:47 spring/app.jar
755 2023-03-15 13:47 spring/application.yml
--------- -------
75585336 28 files-b | --broker <# of brokers> <# of replicas> < port1,...,portn > < 1 = enable jmx >
Creates a configuration of Kafka brokers as defined in the the input arguments. If no zookeeper configuration is added, it will default to only one zookeeper container. See example below:
$ xkite-cli -b 2 2 9092,9093
_ _ _
__ _| | _(_) |_ ___
\ \/ / |/ / | __/ _ \
> <| <| | || __/
/_/\_\_|\_\_|\__\___|
Configuring docker instances...
creating Kite Config yml...
creating zookeepers...
creating brokers...
yaml configuration complete...
$ xkite-cli -r
_ _ _
__ _| | _(_) |_ ___
\ \/ / |/ / | __/ _ \
> <| <| | || __/
/_/\_\_|\_\_|\__\___|
Deploying docker instances...
deploying docker containers...
Building with native build. Learn about native build in Compose here: https://docs.docker.com/go/compose-native-build/
Creating network "download_default" with the default driver
Creating volume "download_jupyterhub_data" with local driver
Creating volume "download_dashboards" with local driver
Creating volume "download_provisioning" with local driver
Creating zookeeper1 ...
Creating zookeeper1 ... done
Creating kafka2 ...
Creating kafka1 ...
Creating kafka2 ... done
Creating kafka1 ... done
docker deployment successful
-z | --zookeeper <# of zookeepers> < port1,...,portn >
Creates a configuration of Kafka Zookeepers as defined in the the input arguments. If no broker configuration is added, it will default to using two broker containers. See example below:
$ xkite-cli -z 2 2181,2182
_ _ _
__ _| | _(_) |_ ___
\ \/ / |/ / | __/ _ \
> <| <| | || __/
/_/\_\_|\_\_|\__\___|
Configuring docker instances...
creating Kite Config yml...
creating zookeepers...
creating brokers...
yaml configuration complete...
$ xkite-cli -r
_ _ _
__ _| | _(_) |_ ___
\ \/ / |/ / | __/ _ \
> <| <| | || __/
/_/\_\_|\_\_|\__\___|
Deploying docker instances...
deploying docker containers...
Building with native build. Learn about native build in Compose here: https://docs.docker.com/go/compose-native-build/
Creating network "download_default" with the default driver
Creating volume "download_jupyterhub_data" with local driver
Creating volume "download_dashboards" with local driver
Creating volume "download_provisioning" with local driver
Creating zookeeper1 ...
Creating zookeeper1 ... done
Creating zookeeper1 ... done
Creating zookeeper2 ... done
Creating kafka2 ...
Creating kafka1 ...
Creating kafka2 ... done
Creating kafka1 ... done
docker deployment successful
-z | --zookeeper <# of zookeepers> < port1,...,portn >
Creates a configuration of Kafka Zookeepers as defined in the the input arguments. If no broker configuration is added, it will default to using two broker containers. See example below:
$ xkite-cli -z 2 2181,2182
_ _ _
__ _| | _(_) |_ ___
\ \/ / |/ / | __/ _ \
> <| <| | || __/
/_/\_\_|\_\_|\__\___|
Configuring docker instances...
creating Kite Config yml...
creating zookeepers...
creating brokers...
yaml configuration complete...
$ xkite-cli -r
_ _ _
__ _| | _(_) |_ ___
\ \/ / |/ / | __/ _ \
> <| <| | || __/
/_/\_\_|\_\_|\__\___|
Deploying docker instances...
deploying docker containers...
Building with native build. Learn about native build in Compose here: https://docs.docker.com/go/compose-native-build/
Creating network "download_default" with the default driver
Creating volume "download_jupyterhub_data" with local driver
Creating volume "download_dashboards" with local driver
Creating volume "download_provisioning" with local driver
Creating zookeeper1 ...
Creating zookeeper1 ... done
Creating zookeeper1 ... done
Creating zookeeper2 ... done
Creating kafka2 ...
Creating kafka1 ...
Creating kafka2 ... done
Creating kafka1 ... done
docker deployment successful
-sk | --sink < type: jupyter | spark > < port >
Creates a configuration with the sink provided. The sink can be either 'jupyter' or 'spark' at this time. If no Broker or Zookeepers are defined in the the input arguments. A default or two brokers and one zookeeper containers will be used. See example below:
$ xkite-cli -sk jupyter
_ _ _
__ _| | _(_) |_ ___
\ \/ / |/ / | __/ _ \
> <| <| | || __/
/_/\_\_|\_\_|\__\___|
Configuring docker instances...
creating Kite Config yml...
creating zookeepers...
creating brokers...
yaml configuration complete...
$ xkite-cli -r
_ _ _
__ _| | _(_) |_ ___
\ \/ / |/ / | __/ _ \
> <| <| | || __/
/_/\_\_|\_\_|\__\___|
Deploying docker instances...
deploying docker containers...
Building with native build. Learn about native build in Compose here: https://docs.docker.com/go/compose-native-build/
Creating network "download_default" with the default driver
Creating volume "download_jupyterhub_data" with local driver
Creating volume "download_dashboards" with local driver
Creating volume "download_provisioning" with local driver
Pulling jupyter (jupyterhub/jupyterhub:)...
latest: Pulling from jupyterhub/jupyterhub
Digest: sha256:0d20df5083b3bc200c143fa1cef61fd11a103f1e8f5324a6c34bc77b97a22bba
Status: Downloaded newer image for jupyterhub/jupyterhub:latest
Creating zookeeper1 ...
Creating jupyter ...
Creating jupyter ... done
Creating zookeeper1 ... done
Creating kafka1 ...
Creating kafka2 ...
Creating kafka2 ... done
Creating kafka1 ... done
Creating kafka_connect_sink ...
Creating kafka_connect_sink ... done
docker deployment successful-db | --database < type: ksql | postgresql > < port >
Creates a configuration with the database source provided. The souce can be either 'ksql' or 'postgresql' at this time. If no Broker or Zookeepers are defined in the the input arguments. A default or two brokers and one zookeeper containers will be used. See example below:
$ xkite-cli -db ksql
_ _ _
__ _| | _(_) |_ ___
\ \/ / |/ / | __/ _ \
> <| <| | || __/
/_/\_\_|\_\_|\__\___|
Configuring docker instances...
creating Kite Config yml...
creating zookeepers...
creating brokers...
yaml configuration complete...
$ xkite-cli -r
_ _ _
__ _| | _(_) |_ ___
\ \/ / |/ / | __/ _ \
> <| <| | || __/
/_/\_\_|\_\_|\__\___|
Deploying docker instances...
deploying docker containers...
Building with native build. Learn about native build in Compose here: https://docs.docker.com/go/compose-native-build/
Creating network "download_default" with the default driver
Creating volume "download_jupyterhub_data" with local driver
Creating volume "download_dashboards" with local driver
Creating volume "download_provisioning" with local driver
Pulling ksql (confluentinc/ksqldb-server:)...
latest: Pulling from confluentinc/ksqldb-server
-gr | --grafana < port >
Creates a configuration with a grafana container at external port provided. If no Broker or Zookeepers are defined in the the input arguments it will use two brokers, one zookeeper, a prometheus, a spring and a JMX instances for the configuration. See example below:
$ xkite-cli -gr 3050
_ _ _
__ _| | _(_) |_ ___
\ \/ / |/ / | __/ _ \
> <| <| | || __/
/_/\_\_|\_\_|\__\___|
Configuring docker instances...
creating Kite Config yml...
creating zookeepers...
creating brokers...
yaml configuration complete...
$ xkite-cli -r
_ _ _
__ _| | _(_) |_ ___
\ \/ / |/ / | __/ _ \
> <| <| | || __/
/_/\_\_|\_\_|\__\___|
Deploying docker instances...
deploying docker containers...
Building with native build. Learn about native build in Compose here: https://docs.docker.com/go/compose-native-build/
Creating network "download_default" with the default driver
Creating volume "download_jupyterhub_data" with local driver
Creating volume "download_dashboards" with local driver
Creating volume "download_provisioning" with local driver
Creating prometheus ...
Creating zookeeper1 ...
Creating prometheus ... done
Creating grafana ...
Creating zookeeper1 ... done
Creating kafka1 ...
Creating kafka2 ...
Creating grafana ... done
Creating kafka2 ... done
Creating kafka1 ... done
Creating jmx-kafka2 ...
Creating jmx-kafka1 ...
Creating spring ...
Creating jmx-kafka2 ... done
Creating spring ... done
Creating jmx-kafka1 ... done
docker deployment successful-pr | --prometheus < port > < scrape_interval > < evaluation_interval >
Creates a configuration with a prometheus container at external port provided and with scrape/evaluation interval settings. If no interval settings are provide, then defaults will be used. If no Broker or Zookeepers are defined in the the input arguments it will use two brokers, one zookeeper, a spring and a JMX instances for the configuration. See example below:
$ xkite-cli -pr 9099 20 10
_ _ _
__ _| | _(_) |_ ___
\ \/ / |/ / | __/ _ \
> <| <| | || __/
/_/\_\_|\_\_|\__\___|
Configuring docker instances...
creating Kite Config yml...
creating zookeepers...
creating brokers...
yaml configuration complete...
$ xkite-cli -r
_ _ _
__ _| | _(_) |_ ___
\ \/ / |/ / | __/ _ \
> <| <| | || __/
/_/\_\_|\_\_|\__\___|
Deploying docker instances...
deploying docker containers...
Building with native build. Learn about native build in Compose here: https://docs.docker.com/go/compose-native-build/
Creating network "download_default" with the default driver
Creating volume "download_jupyterhub_data" with local driver
Creating volume "download_dashboards" with local driver
Creating volume "download_provisioning" with local driver
Creating zookeeper1 ...
Creating prometheus ...
Creating prometheus ... done
Creating zookeeper1 ... done
Creating kafka2 ...
Creating kafka1 ...
Creating kafka2 ... done
Creating kafka1 ... done
Creating jmx-kafka2 ...
Creating spring ...
Creating jmx-kafka1 ...
Creating jmx-kafka2 ... done
Creating jmx-kafka1 ... done
Creating spring ... done