Skip to content
A Kafka Source connector to receive data from REST APIs and publish them to Kafka. It has an extended version to support FitBit APIs.
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
docker
gradle/wrapper
kafka-connect-fitbit-source
kafka-connect-rest-source
scripts/REDCAP-FITBIT-AUTH-AUTO
.dockerignore
.editorconfig
.gitignore
Dockerfile
LICENSE
README.md
build.gradle
docker-compose.yml
gradlew
gradlew.bat
settings.gradle

README.md

Kafka Connect REST Source and Fitbit Source

This project contains a Kafka Connect source connector for a general REST API, and one for Fitbit in particular. The documentation of the Kafka Connect REST source still needs to be done.

Fitbit source connector

Installation

This repository relies on a recent version of docker and docker-compose as well as an installation of Java 8 or later.

Usage

First, register a Fitbit App with Fitbit. It should be either a server app, for multiple users, or a personal app for a single user. With the server app, you need to request access to intraday API data.

For every Fitbit user you want access to, copy docker/fitbit-user.yml.template to a file in docker/users/. Get an access token and refresh token for the user using for example the Fitbit OAuth 2.0 tutorial page.

For automatic configuration for multiple users, please take a look at scripts/REDCAP-FITBIT-AUTH-AUTO/README.md.

Copy docker/source-fitbit.properties.template to docker/source-fitbit.properties and enter your Fitbit App client ID and client secret. The following tables shows the possible properties.

Name Description Type Default Valid Values Importance
rest.source.poll.interval.msHow often to poll the source URL.long60000low
rest.source.base.urlBase URL for REST source connector.stringhigh
rest.source.destination.topicsThe list of destination topics for the REST source connector.list""high
rest.source.topic.selectorThe topic selector class for REST source connector.classorg.radarbase.connect.rest.selector.SimpleTopicSelectorClass extending org.radarbase.connect.rest.selector.TopicSelectorhigh
rest.source.payload.converter.classClass to be used to convert messages from REST calls to SourceRecordsclassorg.radarbase.connect.rest.converter.StringPayloadConverterClass extending org.radarbase.connect.rest.converter.PayloadToSourceRecordConverterlow
rest.source.request.generator.classClass to be used to generate REST requestsclassorg.radarbase.connect.rest.single.SingleRequestGeneratorClass extending org.radarbase.connect.rest.request.RequestGeneratorlow
fitbit.usersThe user ID of Fitbit users to include in polling, separated by commas. Non existing user names will be ignored. If empty, all users in the user directory will be used.list""high
fitbit.api.clientClient ID for the Fitbit APIstringnon-empty stringhigh
fitbit.api.secretSecret for the Fitbit API client set in fitbit.api.client.passwordhigh
fitbit.api.intradaySet to true if the client has permissions to Fitbit Intraday API, false otherwise.booleantruemedium
fitbit.user.repository.classClass for managing users and authentication.classorg.radarbase.connect.rest.fitbit.user.YamlUserRepositoryClass extending org.radarbase.connect.rest.fitbit.user.UserRepositorymedium
fitbit.user.dirDirectory containing Fitbit user information and credentials. Only used if a file-based user repository is configured.string/var/lib/kafka-connect-fitbit-source/userslow
fitbit.user.repository.urlURL for webservice containing user credentials. Only used if a webservice-based user repository is configured.string""low
fitbit.max.users.per.pollMaximum number of users to query in a single poll operation. Decrease this if memory constrains are pressing.int100[1,...]low
fitbit.intraday.steps.topicTopic for Fitbit intraday stepsstringconnect_fitbit_intraday_stepsnon-empty string without control characterslow
fitbit.intraday.heart.rate.topicTopic for Fitbit intraday heart_ratestringconnect_fitbit_intraday_heart_ratenon-empty string without control characterslow
fitbit.sleep.stages.topicTopic for Fitbit sleep stagesstringconnect_fitbit_sleep_stagesnon-empty string without control characterslow
fitbit.sleep.classic.topicTopic for Fitbit sleep classic datastringconnect_fitbit_sleep_classicnon-empty string without control characterslow
fitbit.time.zone.topicTopic for Fitbit profile time zonestringconnect_fitbit_time_zonenon-empty string without control characterslow
fitbit.activity.log.topicTopic for Fitbit activity log.stringconnect_fitbit_activity_lognon-empty string without control characterslow

Now you can run a full Kafka stack using

docker-compose up -d --build

Inspect the progress with docker-compose logs -f radar-fitbit-connector. To inspect data that is coming out of the requests, run

docker-compose exec schema-registry-1 kafka-avro-console-consumer \
  --bootstrap-server kafka-1:9092,kafka-2:9092,kafka-3:9092 \
  --from-beginning \
  --topic connect_fitbit_intraday_heart_rate

Contributing

Code should be formatted using the Google Java Code Style Guide. If you want to contribute a feature or fix browse our issues, and please make a pull request.

You can’t perform that action at this time.