Skip to content

Commit

Permalink
Document project
Browse files Browse the repository at this point in the history
Closes #6
  • Loading branch information
tbobm committed Dec 24, 2020
1 parent 7f7d887 commit 4e14f99
Show file tree
Hide file tree
Showing 2 changed files with 64 additions and 3 deletions.
65 changes: 63 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,70 @@
# Payload Spooler
# Payl[oad Spoo]ler

_Send your payload now, treat it later._

## What is this?

Payler is an asyncio-based Python application intended to provide a way of delaying message execution. The goal of this program is to reduce the workload on your existing message broker solution (Only RabbitMQ is currently supported, but other message-brokers can be easily implemented) by putting the payloads in a storage backend which will then be polled to re-inject payloads in the corresponding destination.

## How to use this

Using the command line:

1. Specify the input and output URLs for your drivers (see [configuration](#configuration))
2. (optional) Customize the configuration to suit your needs _currently the example configuration is the only valid one_
3. Run payler `payler --config-file configuration.yaml`

Using the docker image:

1. Pull the docker image `docker pull ghcr.io/tbobm/payler:latest`
2. (optional) Customize the configuration to suit your needs _currently the example configuration is the only valid one_ (mount the configuration file into the volume at `/configuration.yaml`)
3. Run the docker image and provide environment variables `docker run -d --name payler -e BROKER_URL="amqp://payler:secret@my-broker/" -e MONGODB_URL="mongodb://payler:secret@my-mongo/payler" ghcr.io/tbobm/payler`

## Configuration

In order to configure the different workflows, payler uses a configuration file (see [configuration.yml](./configuration.yml)).

Example config file:

```yaml
---
workflows:
- name: "Fetch payloads from RabbitMQ and store them in MongoDB"
location: "payler"
callable: "client.process_queue"
- name: "Re-injects payloads to RabbitMQ"
callable: "client.watch_storage"
```

The `workflows[].name` attribute is currently unused, but will offer a more human-friendly way of getting informed about a workflow's state.
The `workflows[].location` corresponds to the package where the `workflows[].callable` can be found. It defaults to `payler`, but can this is a way of offering a dumb and simple plugin mechanism by creating function matching the following signature:

```python
async def my_workflow(loop: asyncio.AbstractEventLoop) -> None:
"""My user-defined workflow."""
# configure your driver(s)
input_driver.serve()
```

## Features

- Listen to a Broker Queue
- Store messages with a duration or date as metadata
- Re-inject the messages after the duration or the specified date in the queue
- Re-inject the messages after the duration in the default Exchange

## Testing

This project has unittests with [pytest](https://docs.pytest.org/en/latest/). A wrapper script is available at [run-tests.sh](./run-tests.sh).

## Contributing

Feel free to open new issues for feature requests and bug reports in the [issue page](github.com/tbobm/payler/issues/new) and even create PRs if you feel like it.

This project is linted with `pylint` with some minor adjustments (see the [setup.cfg](./setup.cfg)).

## Note

This side-project is born from the following:
- I wanted to experiment with Python's `asyncio`
- A friend of mine had issues with delaying lots of messages using RabbitMQ's [delayed exchange plugin](https://github.com/rabbitmq/rabbitmq-delayed-message-exchange)
- I was looking for a concrete use-case to work with Github Actions.
2 changes: 1 addition & 1 deletion configuration.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
workflows:
- name: 'Consume broker payloads and store'
# location: "payler"
location: "payler"
callable: "client.process_queue"
- name: "Poll storage and re-inject in RabbitMQ"
callable: "client.watch_storage"

0 comments on commit 4e14f99

Please sign in to comment.