Skip to content

Commit

Permalink
📝 Update task-queue docs in contributing
Browse files Browse the repository at this point in the history
  • Loading branch information
evanroyrees committed Jul 26, 2023
1 parent a51efff commit 7e1ccb9
Show file tree
Hide file tree
Showing 4 changed files with 51 additions and 10 deletions.
2 changes: 1 addition & 1 deletion automappa/pages/home/tasks/__init__.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from .task_status_badge import set_badge_color
from .status_badge import set_badge_color
from .sample_cards import (
create_metagenome_model,
initialize_refinement,
Expand Down
File renamed without changes.
3 changes: 2 additions & 1 deletion docs/CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
- [Postgres](#postgres)
- [RabbitMQ](#rabbitmq)
- [Celery](#celery-task-queue)
- [task-queue docs](task-queue.md)
- [Redis](#redis)
- [Flower](#flower)
- [Dev Resources](#development-resources)
Expand Down Expand Up @@ -1117,7 +1118,7 @@ Automappa services (`redis`, `flower`, `web`, `celery`, `rabbitmq`)
as tasks are registered with celery at instantiation and will not be
'hot-reloaded' like other parts of the app.

For more information on implementing new tasks see the [task docs](../automappa/pages/home/tasks/README.md)
For more information on implementing new tasks see the [task-queue docs](task-queue.md)

<details>

Expand Down
56 changes: 48 additions & 8 deletions docs/task-queue.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,13 @@
# Tasks

Simple steps to adding a task:

1. [Add page's `tasks` module](#adding-a-new-task-module-discovery-path)
2. [Add page tasks module in `celeryconfig.py`](#add-in-tasks-module-in-celeryconfigpy)
3. [Define task in `tasks` module](#define-task-in-tasks-module)
4. [Import task in `__init__.py` w.r.t `tasks` module](#import-task-in-__init__py-within-respective-tasks-module)


## Adding a new task module discovery path

To have the Celery task-queue register a page's tasks, they must be
Expand All @@ -10,31 +18,63 @@ We have our page's tasks module:

```console
automappa/pages/home/tasks
├── README.md
|...
├── __init__.py
└── task_status_badge.py
└── status_badge.py
```

## Add in `tasks` module in `celeryconfig.py`

We configure celery for this module in the `celeryconfig.py` file.

```python
# contents of celeryconfig.py
# contents of automappa/conf/celeryconfig.py
imports = ("automappa.pages.home.tasks", "automappa.tasks")
```

This ***almost*** takes care of celery checking for tasks in the module

Unfortunately, this is not all, we also need to update the
Unfortunately, this is not all...

## Define task in `tasks` module

Here is a simple example where we will simulate a long-running task with sleep and then set a badge color.

> Notice we are importing our celery task-queue application (`queue`) from the base automappa directory.
```python
# contents of /automappa/pages/home/tasks/status_badge.py
import time

from automappa.tasks import queue


@queue.task(bind=True)
def set_badge_color(self, color: str) -> str:
time.sleep(15)
return color


if __name__ == "__main__":
pass

```

Now with our task defined we can move on to importing it into our `tasks` module to be used by celery.

## Import task in `__init__.py` within respective `tasks` module

We also need to update the
`automappa/pages/home/tasks/__init__.py` with all of our implemented tasks in the
module for celery to recognize and register the tasks under the `automappa/pages/home/tasks` module.

For example we have a task `set_badge_color` defined in `task_status_badge.py`.
For example we have a task `set_badge_color` defined in `status_badge.py`.

We would need to explicitly add this task to `__init__.py` like so:

```python
# contents of `automappa/pages/home/tasks/__init__.py`
from .task_status_badge import set_badge_color
from .status_badge import set_badge_color # *Notice this is a relative import*

__all__ = ["set_badge_color"]
```
Expand Down Expand Up @@ -62,7 +102,7 @@ Voilá, now celery will recognize the task on startup. It should look like this:
.> celery exchange=celery(direct) key=celery
[tasks]
. automappa.pages.home.tasks.task_status_badge.set_badge_color
. automappa.pages.home.tasks.status_badge.set_badge_color
. automappa.tasks.aggregate_embeddings
. automappa.tasks.count_kmer
. automappa.tasks.embed_kmer
Expand Down Expand Up @@ -97,4 +137,4 @@ but each page has its own `tasks` module with an `__init__.py`
where any tasks implemented under this module must be imported into
`__init__.py` and specified in the `__all__` dunder variable.

You can avoid alot of headache by recalling this...
You can avoid alot of headache by recalling this...

0 comments on commit 7e1ccb9

Please sign in to comment.