In my organization, several of us had varying versions of scripts (bash, python, ruby, etc...) to create and submit Kafka Connector Configurations. We often shared these via Slack or Teams and NOBODY seemed to be sure what the correct version of which connector was deployed where. (Having typed this into this document, the embarrassment I feel is indescribable.)
The first step of solving any problem is admitting there is one. So let's move on...
The goal of this plugin is to give users a way to manage Apache Kafka Connector configurations and submit the configurations to a given cluster via the Connect API.
-
Gradle 6.7+
-
Java 8+
-
But why JSONNET? Because I preferred the templating engine of jsonnet to other tools to merge json documents. At the time of writing this plugin, I can find no adequate Java/Groovy API for jsonnet (with good reason, I suppose). There is a Scala API, but I did not feel like addressing the typical interop issues I have seen in my experience with Java calling Scala.
Think of the things that are environment-specific with regards to a Kafka Connector - number of tasks, database names, hostnames, URLs, credentials (please don't commit these to your repo), etc... These belong in *.jsonnet files.
The rest is boilerplate - such as the name of the connector, the key and value converters, tranforms and their params, etc... These would go in the *.libsonnet files that are referenced in *.jsonnet files using the import functionality.
It's pretty simple to install on MOST platforms (brew, apt-get, yum, etc...).
As for those concerned with "performance," this isn't some high-volume document processing system - it's a BUILD TOOL! However, I will an issue for in Github in the hopes to refactor this functionality. Stay tuned.
Add to your project's build.gradle with a valid version number:
// ...some gradle stuffs...
plugins {
// ...other gradle stuffs...
id 'com.github.sandonjacobs.gradle-kafka-connect' version '*.*.*'
// ...more gradle stuffs...
}
// ...even more gradle stuffs...
kafkaConnect {
// BTW: using the defaults here to illustrate this closure...
connectEndpoint = 'http://localhost:8083'
sourceBase = 'src/main'
connectorSourceName = 'connectors'
// if you wanna ignore sourceBase and connectorSourceName with a directory at the root of the project...
// connectorSourcePath = 'some-connectors'
}
// enough with the gradle stuffs...
| Property Name | Description | Default Value |
|---|---|---|
| connectEndpoint | HTTP url (with port) to the root of the Kafka Connect API. | http://localhost:8083 |
| connectorSourceName | Where are the connectors to be loaded from. | connectors |
| sourceBase | From root of project, the base of the connectorSourceName directory. |
src/main |
| connectorSourcePath | This will override the use of connectorSourceName and sourceBase if set. |
List the connectors that are running on a given cluster, using the Connectors API endpoint.
| Parameter | Description |
|---|---|
| connectorEndpoint | override the connector configuration endpoint url. |
> ./gradlew help --task listConnectors> ./gradlew listConnectors> ./gradlew listConnectors --connector-endpoint http://myconnect:8888Parse, (jsonnet) merge, and (conditionally) submit the connectors from a project directory to a given cluster, using the Connectors API endpoint.
| Parameter | Description |
|---|---|
| connectorEndpoint | override the connector configuration endpoint url. |
| connectorSubDir | (optional) specify a subdirectory to run with. If provided, is appended to the connectorSourcePath, as defined in the getConnectorsPath() function. |
| jsonnet-tla-str-args | Support for additional jsonnet args, maps to --tla-str <var>[=<val>]. |
| jsonnet-ext-code-file | Support for additional jsonnet ext code file inclusion, maps to --ext-code-file <var>[=<val>]. |
| connector | Optionally, specify the name of a SINGLE jsonnet file (minus the extension) to submit from the connectorSubDir. |
> ./gradlew help --task submitConnectors> ./gradlew submitConnectors> ./gradlew submitConnectors --connector-endpoint http://myconnect:8888> ./gradlew submitConnectors --print-only> ./gradlew submitConnectors --connector-subdir prod> ./gradlew submitConnectors --connector-subdir prod --connector fooNote: If the file
foo.jsonnetdoes not exist in theprodsubdirectory, the task will fail!
> ./gradlew submitConnectors --print-only --connector-subdir prodGiven a jsonnet file named foo.jsonnet in the prod subdirectory.
And that jsonnet file references a jsonnet extVar named env as follows:
local environmentVars = std.extVar('env');Use this command to generate the connector foo with the environment info from qa.libsonnet
> ./gradlew submitConnectors --connector-subdir prod --connector foo --jsonnet-ext-code-file env=qa.libsonnetNote: If the file
qa.libsonnetdoes not exist in theprodsubdirectory, the task will fail!
Suppose we have a connector that contains some value we want to bind late in the lifecycle - perhaps something only known by the CI/CD environment or an environment variable.
Utilize the tla-str functionality when defining the jsonnet templates and then use the jsonnet-tla-str-args parameter
of this task to pass the values of those tla-str args to the template. If there are multiple key=value pairs (mapping to
multiple arguments in the jsonnet tla function), add them as pipe-delimited pairs. Here's the example:
Suppose we have a template such as:
local lib = import '../base-tla.libsonnet';
function(value1, value2) {
"name": "my name is",
"config": lib.Config("sample-with-tla", value1, value2),
}Add the top-level function lives in the base-tla.libsonnet file referenced above, as such:
{
Config(name, value1, value2): {
"name": name,
"s3.region": "us-east-1",
"something.of.value1": value1,
"something.of.value2": value2,
},
}
Executing the task below will pass these values to the templates and functions, replacing the variables in the resulting json:
> ./gradlew submitConnectors --print-only --connector-subdir prod --jsonnet-tla-str-args='value1=the value|value2=something else'