Taskbroker provides a Kafka consumer, RPC interface, and inflight task storage that form the core engine of asynchronous task execution at Sentry.
flowchart LR
Sentry -- produce activation --> k[(Kafka)]
k -- consume messages --> Taskbroker
Taskbroker -- store --> sql[(SQLite)]
Worker -- grpc GetTask --> Taskbroker
Worker -- grpc SetTaskStatus --> Taskbroker
Sentry (and other applications in the future) produce task activations. Taskbroker consumes those messages and stores them in a SQLite database to avoid head-of-line blocking, enable out-of-order execution and per-task acknowledgements.
Workers communicate with Taskbroker via gRPC. There are two primary gRPC methods:
GetTask
Workers can request a task from Taskbroker.SetTaskStatus
Once a worker has completed a task,SetTaskStatus
is used to report the task's outcome.
Taskbroker is written in Rust and requires the latest stable Rust compiler. To compile, run:
cargo build --release
The resulting binary ends up in target/release/
along with debug information files.
To build taskbroker
we require the latest stable Rust and protoc
. The root of the repository
contains a Makefile
with frequently run development commands:
make setup
Run setup tasks to create and configure your development environment.make test
Run rust tests.make integration-test
Run end to end tests.make format
Format rust code withcargo fmt
andcargo clippy
.
The simplest way to run taskbroker is with cargo
. You'll also need
kafka
running. Kafka is typically managed with devservices:
cargo run -- -c ./config/config-sentry-dev.yaml
Taskbroker uses YAML files for configuration, and all of the available configuration options can be found in Config
All configuration options can also be defined as environment variables using
the TASKBROKER_
prefix.
The test suite is composed of unit and integration tests in Rust, and end-to-end tests scripted with python.
# Run unit/integration tests
make test
# Run end-to-end tests
make integration-test
We use rustfmt
and clippy
from the latest stable channel for code formatting and linting:
# Check for formatting issues
make style
# Fix formatting and lint issues
make format