Skip to content

Commit

Permalink
docs: add missing info
Browse files Browse the repository at this point in the history
  • Loading branch information
thearchitector committed Jul 15, 2023
1 parent 428e506 commit 2e84fa7
Show file tree
Hide file tree
Showing 2 changed files with 26 additions and 8 deletions.
17 changes: 13 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,13 +19,14 @@ $ pip install --user just-jobs

## Features

just-jobs doesn't aim to replace the invocations that arq provides, only wrap some of them to make job creation and execution easier and better. It lets you:
just-jobs doesn't aim to replace the invocations that arq provides, only wrap some of them to make job creation and execution better and easier. It lets you:

- Define and run non-async jobs. Passing a non-async `@job` function to arq will run properly. Non-async jobs can also be defined as either IO-bound or CPU-bound, which changes how the job will be executed to prevent blocking the asyncio event loop.
- The arq `Context` parameter now works a lot like [FastAPI's `Request`](https://fastapi.tiangolo.com/advanced/using-request-directly/). It's no longer a required parameter, but if it exists, it will get set. It doesn't have to be named `ctx` either, only have the type `Context`.
- Specify a single `RedisSettings` within your `WorkerSettings` from which you can create a pool using `Settings.create_pool()`.
- Run jobs either immediately or via normal arq enqueueing.
- Use non-pickable job arguments and kwargs (supported by the [dill](http://dill.rtfd.io/) library).
- Use non-picklable job arguments and kwargs (supported by the [dill](http://dill.rtfd.io/) library).
- Signed secure job serialization using `blake2b`.

## Usage

Expand All @@ -36,7 +37,7 @@ Using just-jobs is pretty straight forward:
If the job is synchronous, specify its job type so just-jobs knows how to optimally run it. If you don't, you'll get an error. This helps encourage thoughtful and intentional job design while ensuring that the event loop is never blocked.

```python
@job(job_type=JobType.CPU_BOUND)
@job(job_type=JobType.CPU_BOUND) # or JobType.IO_BOUND
def complex_math(i: int, j: int, k: int)
```

Expand All @@ -47,6 +48,10 @@ If it's a coroutine function, you don't need to specify a job type (and will get
async def poll_reddit(subr: str)
```

By default, just-jobs will utilize your Python version's default number of [thread](https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor) and [process](https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ProcessPoolExecutor) workers to handle IO-bound and CPU-bound tasks respectively. On 3.8+, that is `min(32, CPU_COUNT + 4)` for IO-bound jobs and `1 <= CPU_COUNT <= 61` for CPU-bound ones.

If you want to configure those max worker values, you can do so via the `MAX_THREAD_WORKERS` and `MAX_PROCESS_WORKERS` environment variables.

### Invoke a job normally if you want to run it immediately.

Invoking a job as a regular function allows you to run a job as if it were one. If you have logic that you only want to execute when enqueued, include a parameter with type `Context` and check if it exists at runtime (functions with a `Context` that are run immediately will have that argument set to `None`).
Expand Down Expand Up @@ -90,9 +95,13 @@ async with Settings.create_pool() as pool:
...
```

### Sign your job serializations with `blake2b`.

By default, using just-jobs `Settings` means all serialized jobs are prefixed with a signature which is then parsed and validated before job execution. This helps ensure that any jobs you serialize do not get tampered with while enqueued and waiting for execution. The default (and very insecure) secret used for signing is `thisisasecret`. In any production or public-facing deployment, you _should_ change this value to something private and secure. It can be changed via the `JOB_SERIALIZATION_SECRET` environment variable.

### Enqueue your job.

just-jobs doesn't change the way in which you enqueue your jobs. Just use `await pool.enqueue_job(...)`.
just-jobs doesn't change the way in which you enqueue your jobs. Just use `await pool.enqueue_job(...)`. Using just-jobs, you also don't have to worry as much about the type of arguments you supply; all Python objects supported by the [dill](http://dill.rtfd.io/) serialization library will work just fine.

```python
await pool.enqueue_job('complex_math', 2, 1, 3)
Expand Down
17 changes: 13 additions & 4 deletions docs/just_jobs.html
Original file line number Diff line number Diff line change
Expand Up @@ -112,14 +112,15 @@ <h1 class="modulename">

<h2 id="features">Features</h2>

<p>just-jobs doesn't aim to replace the invocations that arq provides, only wrap some of them to make job creation and execution easier and better. It lets you:</p>
<p>just-jobs doesn't aim to replace the invocations that arq provides, only wrap some of them to make job creation and execution better and easier. It lets you:</p>

<ul>
<li>Define and run non-async jobs. Passing a non-async <code>@job</code> function to arq will run properly. Non-async jobs can also be defined as either IO-bound or CPU-bound, which changes how the job will be executed to prevent blocking the asyncio event loop.</li>
<li>The arq <code><a href="#Context">Context</a></code> parameter now works a lot like <a href="https://fastapi.tiangolo.com/advanced/using-request-directly/">FastAPI's <code>Request</code></a>. It's no longer a required parameter, but if it exists, it will get set. It doesn't have to be named <code>ctx</code> either, only have the type <code><a href="#Context">Context</a></code>.</li>
<li>Specify a single <code>RedisSettings</code> within your <code>WorkerSettings</code> from which you can create a pool using <code>Settings.create_pool()</code>.</li>
<li>Run jobs either immediately or via normal arq enqueueing.</li>
<li>Use non-pickable job arguments and kwargs (supported by the <a href="http://dill.rtfd.io/">dill</a> library).</li>
<li>Use non-picklable job arguments and kwargs (supported by the <a href="http://dill.rtfd.io/">dill</a> library).</li>
<li>Signed secure job serialization using <code>blake2b</code>.</li>
</ul>

<h2 id="usage">Usage</h2>
Expand All @@ -131,7 +132,7 @@ <h3 id="add-job-to-any-function-to-make-it-a-delayable-job">Add <code>@job()</co
<p>If the job is synchronous, specify its job type so just-jobs knows how to optimally run it. If you don't, you'll get an error. This helps encourage thoughtful and intentional job design while ensuring that the event loop is never blocked.</p>

<div class="pdoc-code codehilite">
<pre><span></span><code><span class="nd">@job</span><span class="p">(</span><span class="n">job_type</span><span class="o">=</span><span class="n"><a href="#JobType.CPU_BOUND">JobType.CPU_BOUND</a></span><span class="p">)</span>
<pre><span></span><code><span class="nd">@job</span><span class="p">(</span><span class="n">job_type</span><span class="o">=</span><span class="n"><a href="#JobType.CPU_BOUND">JobType.CPU_BOUND</a></span><span class="p">)</span> <span class="c1"># or <a href="#JobType.IO_BOUND">JobType.IO_BOUND</a></span>
<span class="k">def</span> <span class="nf">complex_math</span><span class="p">(</span><span class="n">i</span><span class="p">:</span> <span class="nb">int</span><span class="p">,</span> <span class="n">j</span><span class="p">:</span> <span class="nb">int</span><span class="p">,</span> <span class="n">k</span><span class="p">:</span> <span class="nb">int</span><span class="p">)</span>
</code></pre>
</div>
Expand All @@ -144,6 +145,10 @@ <h3 id="add-job-to-any-function-to-make-it-a-delayable-job">Add <code>@job()</co
</code></pre>
</div>

<p>By default, just-jobs will utilize your Python version's default number of <a href="https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor">thread</a> and <a href="https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ProcessPoolExecutor">process</a> workers to handle IO-bound and CPU-bound tasks respectively. On 3.8+, that is <code>min(32, CPU_COUNT + 4)</code> for IO-bound jobs and <code>1 &lt;= CPU_COUNT &lt;= 61</code> for CPU-bound ones.</p>

<p>If you want to configure those max worker values, you can do so via the <code>MAX_THREAD_WORKERS</code> and <code>MAX_PROCESS_WORKERS</code> environment variables.</p>

<h3 id="invoke-a-job-normally-if-you-want-to-run-it-immediately">Invoke a job normally if you want to run it immediately.</h3>

<p>Invoking a job as a regular function allows you to run a job as if it were one. If you have logic that you only want to execute when enqueued, include a parameter with type <code><a href="#Context">Context</a></code> and check if it exists at runtime (functions with a <code><a href="#Context">Context</a></code> that are run immediately will have that argument set to <code>None</code>).</p>
Expand Down Expand Up @@ -190,9 +195,13 @@ <h3 id="use-settingscreate_pool">Use <code>Settings.create_pool()</code>.</h3>
</code></pre>
</div>

<h3 id="sign-your-job-serializations-with-blake2b">Sign your job serializations with <code>blake2b</code>.</h3>

<p>By default, using just-jobs <code>Settings</code> means all serialized jobs are prefixed with a signature which is then parsed and validated before job execution. This helps ensure that any jobs you serialize do not get tampered with while enqueued and waiting for execution. The default (and very insecure) secret used for signing is <code>thisisasecret</code>. In any production or public-facing deployment, you _should_ change this value to something private and secure. It can be changed via the <code>JOB_SERIALIZATION_SECRET</code> environment variable.</p>

<h3 id="enqueue-your-job">Enqueue your job.</h3>

<p>just-jobs doesn't change the way in which you enqueue your jobs. Just use <code>await pool.enqueue_job(...)</code>.</p>
<p>just-jobs doesn't change the way in which you enqueue your jobs. Just use <code>await pool.enqueue_job(...)</code>. Using just-jobs, you also don't have to worry as much about the type of arguments you supply; all Python objects supported by the <a href="http://dill.rtfd.io/">dill</a> serialization library will work just fine.</p>

<div class="pdoc-code codehilite">
<pre><span></span><code><span class="k">await</span> <span class="n">pool</span><span class="o">.</span><span class="n">enqueue_job</span><span class="p">(</span><span class="s1">&#39;complex_math&#39;</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span>
Expand Down

0 comments on commit 2e84fa7

Please sign in to comment.