Skip to content
Airbnb's Open-source Distributed Delayed Job Queueing System
Java
Branch: master
Clone or download
Latest commit 2b3164a Jan 7, 2020
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
conveyor
dynein-api initial commit Dec 10, 2019
dynein-common
dynein
example initial commit Dec 10, 2019
gradle
.gitignore initial commit Dec 10, 2019
LICENSE initial commit Dec 10, 2019
README.md Fix typo in README.md Jan 7, 2020
build.gradle initial commit Dec 10, 2019
gradlew initial commit Dec 10, 2019
gradlew.bat initial commit Dec 10, 2019
lombok.config initial commit Dec 10, 2019
settings.gradle

README.md

Dynein

Dynein is Airbnb's Open-source Distributed Delayed Job Queueing System. Learn more about Dynein in this blog post: Dynein: Building an Open-source Distributed Delayed Job Queueing System.

Example

Dynein's components are structured as Managed objects in Dropwizard. We have prepared an example application to demonstrate how to integrate Dynein in your service in ./example.

To run this example, follow the following steps:

  1. Setup localstack. The following steps will be using the default ports of localstack.

  2. Run this command to create the necessary SQS queues:

    aws sqs create-queue --endpoint-url http://localhost:4576 --queue-name example --region us-east-1
    aws sqs create-queue --endpoint-url http://localhost:4576 --queue-name inbound --region us-east-1
  3. Run this command to create the DynamoDB table:

    aws dynamodb create-table --table-name dynein_schedules \
      --attribute-definitions AttributeName=shard_id,AttributeType=S AttributeName=date_token,AttributeType=S \
      --key-schema AttributeName=shard_id,KeyType=HASH AttributeName=date_token,KeyType=RANGE \
      --billing-mode PAY_PER_REQUEST --endpoint-url http://localhost:4569 --region us-east-1
  4. cd example && gradle run --args='server example.yml'

  5. In another terminal, use curl to schedule jobs. The following endpoints are available, which maps to scheduled, immediate, and sqs-delayed jobs:

    1. curl 'localhost:28080/schedule/scheduled'
    2. curl 'localhost:28080/schedule/immediate'
    3. curl 'localhost:28080/schedule/sqs-delayed'

    Each endpoint supports optional parameters delaySeconds and name. For example: curl 'localhost:28080/schedule/scheduled?name=testing-name&delaySeconds=15'

  6. In the original terminal where the example is started, at the scheduled time, the following log lines will be printed:

    INFO  [2019-12-10 01:59:04,785] com.airbnb.dynein.example.ExampleJob: Running job with name: "testing-name"
    INFO  [2019-12-10 01:59:04,800] com.airbnb.dynein.example.worker.ManagedWorker: Successfully consumed job from queue
    

Configuration

Here's an example from ./example/example.yml. We will use inline comments to explain what each option does.

scheduler:
  sqs:
    inboundQueueName: inbound # inboud queue name
    endpoint: "http://localhost:4576" # SQS endpoint. normally this line can be omitted, but it's useful for using localstack
  dynamoDb:
    schedulesTableName: dynein_schedules # dynamodb table name
    queryLimit: 10 # max number of results per query
    endpoint: "http://localhost:4569" # DynamoDB endpoint. normally this line can be omitted, but it's useful for localstack
  workers:
    numberOfWorkers: 1 # number of workers to start in this scheduler
    recoverStuckJobsLookAheadMs: 60000 # look ahead time for recovering stuck jobs
    threadPoolSize: 2 # must be at least 1 more than the number of workers
    partitionPolicy: STATIC # can also be `K8S` to use dynamic partitioning
    staticPartitionList: [0] # only needed for static partition policy
  heartbeat:
    monitorInterval: 60000 # the interval for heartbeat monitoring
    stallTolerance: 300000 # the minimum time for a worker to be considered stopped
  maxPartitions: 1 # max number of partitions. can only be increased without reassigning partitions for jobs already scheduled
You can’t perform that action at this time.