Home

Create a Kafka Connector

Connectors in the Learning Dashboard are based on Kafka Connect. Here we provide a short example of a Source Connector for Github.

This connector reads commits and issues (currently not being used) from the API of Github. The information is accessed by HTTP request and the treatment of that information afterwards. This is done from the poll() method in GithubSourceTask.java, once for each team we have configured.

In summary, to make a connector there are five steps:

1. Get the data from your data source, you'll find an example GithubApi.java.

2. Think about a schema for your data (how data ist loaded into Kafka), for example GithubSchema.java.

3. Create a Connector class, that is responsible for managing the copying of data between Kafka and another system, like in GithubSourceConnector.java.

4. Create a task, that does the actual work of copying the data, like in GithubSourceTask.java.

5. Provide configuration to your Connector and your Worker tasks, e.g. GithubSourceConfig.java.

For extensive documentation on how to create Kafka connectors, check out the Connector Developer Guide.