This repository contains the implementation for the event broker used internally by dojot's event processing services.
Data Broker provides two services: Kafka's topic management and real-time data. These are required by most of the services that compose dojot, being essential to maintain the system working.
The topic management service handles Kafka's topics, used for information dissemination (e.g. device creation events), handling the creation of runtime Kafka topics that segregate information on a tenant context basis, restricting the set of events a given service is exposed to only the ones it is actually allowed to process.
The real-time service deals with the routing of data received via sockets that is directed to one or more devices. This data is shown in the device information screen in the GUI.
As we are dealing with Kafka queues, it is up to the consumers of the returned topics to keep track of where the head of its processed queue is at. That allows consumers to process events at their own pace, thus avoiding unwanted data loss in the process. Another important characteristic of the configured topics is that, by default, they are single-partitioned without replications. This can be changed by using endpoints. Check Data Broker's API reference for more information.
Data Broker depends on some dojot modules to run. The minimum set of modules it needs to run is:
- Auth
- Kafka
- Redis
These are the environment variables used by Data Broker code.
| Key | Purpose | Default Value |
|---|---|---|
| DATABROKER_CACHE_HOST | Redis cache host address | "data-broker-redis" |
| DATABROKER_KAFKA_ADDRESS | Kafka address | "kafka" |
| DATABROKER_KAFKA_PORT | Kafka port | 9092 |
| HC_CPU_USAGE_TIMEOUT | Healthcheck CPU usage timeout (ms) | 300000 |
| HC_KAFKA_TIMEOUT | Healthcheck Kafka timeout (ms) | 30000 |
| HC_MEMORY_USAGE_TIMEOUT | Healthcheck Memory usage timeout (ms) | 300000 |
| HC_UPTIME_TIMEOUT | Healthcheck Uptime timeout (ms) | 300000 |
| KAFKA_NUM_PARTITIONS | Default number of partitions in Kafka topics | 1 |
| KAFKA_REPLICATION_FACTOR | Default replication factor in Kafka topics | 1 |
| LOG_LEVEL | Logger level (error, warn, info, debug) | "info" |
| SERVICE_PORT | Data Broker service port | 80 |
Some environment variables are used by dojot libraries that Data Broker uses.
| Key | Purpose | Default Value |
|---|---|---|
| AUTH_URL | Auth host address | "http://auth:5000" |
| DATA_BROKER_URL | Data Broker host address | "http://data-broker" |
| DEVICE_MANAGER_URL | Device Manager host address | "http://device-manager:5000" |
| DOJOT_MANAGEMENT_TENANT | Management tenant | "dojot-management" |
| DOJOT_MANAGEMENT_USER | Management user | "dojot-management" |
| DOJOT_SUBJECT_DEVICES | Subject for device management messages | "dojot.device-manager.device" |
| DOJOT_SUBJECT_DEVICE_DATA | Subject for device data messages | "device-data" |
| DOJOT_SUBSCRIPTION_HOLDOFF | Time (ms) before attempting to subscribe to a set of topics | 2500 |
| DOJOT_SUBJECT_TENANCY | Subject for tenancy messages | "dojot.tenancy" |
| KAFKA_GROUP_ID | Kafka group ID for consumers | "kafka" |
| KAFKA_HOSTS | List of Kafka instances | "kafka:9092" |
Beforehand, you need an already running dojot instance in your machine. See dojot documentation for more information on installation methods.
Generate a Docker image:
docker build -t <username>/data-broker:<tag> -f docker/Dockerfile .Then an image tagged as <username>/data-broker:<tag> will be made available. You can send it to
your DockerHub registry to made it available for non-local dojot installations:
docker push <username>/data-broker:<tag>NOTE THAT an official image is provided at dojot's DockerHub.
As of version v0.5.0, the subscription service is being removed from Data Broker because of its lack
of use. As a consequence, the endpoint /subscription has being removed.
If you found a problem or need help, leave an issue in the main dojot repository and we will help you!