Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

porting over task guide, mdx bugfixes #13754

Merged
merged 12 commits into from
Jun 4, 2024
Merged
3 changes: 2 additions & 1 deletion docs/3.0rc/concepts/artifacts.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -26,12 +26,13 @@ Creating artifacts allows you to publish data from task and flow runs or outside
There are five artifact types: links, Markdown, progress, images, and tables.

<Note>
Each artifact created within a task will be displayed individually in the Prefect UI. </Note>
Each artifact created within a task will be displayed individually in the Prefect UI.
This means that each call to `create_link_artifact()` or `create_markdown_artifact()` generates a distinct artifact.

Unlike the `print()` command, where you can concatenate multiple calls to include additional items in a report, within a task, these commands must be used multiple times if necessary.

To create artifacts like reports or summaries using `create_markdown_artifact()`, compile your message string separately and then pass it to `create_markdown_artifact()` to create the complete artifact.
</Note>

### Creating link artifacts

Expand Down
307 changes: 307 additions & 0 deletions docs/3.0rc/guides/deferred-tasks.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,307 @@
---
title: Deferred Tasks
description: Examples of using Prefect tasks and the task server.
---

## Why use deferred tasks?

Prefect tasks are a great way to quickly execute small, discrete units of work. _Deferred_ Prefect tasks run in a background process using a Prefect task server. Use deferred tasks to move work out of the foreground of your application and distribute concurrent execution across across multiple processes or machines.

For example, if you have a web app, you can use deferred tasks to offload processes such as sending emails, processing images, or inserting data into a database.

This document focuses on how to use deferred task execution. However, Prefect tasks support other advanced use cases, such as running tasks in parallel on Ray or Dask clusters, caching return values, configuring automatic retries, and building workflows with task dependencies.
WillRaphaelson marked this conversation as resolved.
Show resolved Hide resolved

## Using deferred tasks

Prefect tasks are Python functions that can be run immediately or deferred for background execution.

You define a task by adding the `@task` decorator to a Python function, after which you can use the `delay` method to run the task in the background.

If you schedule the task for background execution, you'll run a task server in a separate process or container to execute the task. This process is similar to how you would run a Celery worker or an arq worker to execute background tasks.
WillRaphaelson marked this conversation as resolved.
Show resolved Hide resolved

### Defining a task

Add the `@task` decorator to a Python function to define a Prefect task:

```python
from prefect import task

@task
def my_background_task(name: str):
# Task logic here
print(f"Hello, {name}!")
```

### Calling tasks

You can call a task to run it immediately, or you can defer the task by scheduling it for background execution with `Task.delay`.

**NOTE**: It is also possible to submit tasks to a _task runner_ such as Ray or Dask -- and to defer task execution -- within a workflow, which in Prefect is called a _flow_. However, this document will focus on deferring task execution outside of workflows. For example, by calling `my_task.delay()` within a web application.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This note is very similar to the paragraph beginning on line 12 above. I think we can do without it.


However you run a task, Prefect will use your task configuration to manage and control task execution.
The following example shows both methods mentioned earlier, calling a task and using `delay`:

```python
# Import the previously-defined task
from my_tasks import my_background_task

# Run the task immediately
my_background_task("Joaquim")

# Schedule the task for execution outside of this process
my_background_task.delay("Agrajag")
```

For documentation on the features available for tasks, refer to the [Prefect Tasks documentation](https://docs-3.prefect.io/3.0rc/tutorial/tasks).

### Executing deferred tasks with a task server

To run tasks in a separate process or container, start a task server.

The task server will continually receive instructions to execute deferred tasks from Prefect's API, execute them, and report the results back to the API.

**NOTE:** Task servers only run deferred tasks, not tasks you call directly.

You can run a task server by passing tasks into the `prefect.task_server.serve()` method:

```python tasks.py
from prefect import task
from prefect.task_server import serve


@task
def my_background_task(name: str):
# Task logic here
print(f"Hello, {name}!")


if __name__ == "__main__":
# NOTE: The serve() function accepts multiple tasks. The Task Server
# will listen for scheduled task runs for all tasks passed in.
serve(my_background_task)
```

Run this script to start the task server.

The task server should begin listening for scheduled tasks. If tasks were scheduled before the task server started, it will begin processing them.

You can also use the helper CLI command `prefect task serve` to start a task server.

```bash
prefect task serve my_task.py:my_background_task
```


## Guided exploration of deferred tasks and task servers in Prefect

Below we explore increasingly realistic examples of using deferred tasks and task servers in Prefect.

We'll start by running a Prefect task in the foreground by calling it.

Next we'll start a task server and defer tasks so that they run in the background. We'll see how we can use multiple task servers to run tasks in parallel.

Then we'll create a basic FastAPI application that defers tasks when you hit an endpoint.

Next we'll use Docker in two examples that mimic real use cases.
One example uses a FastAPI server with multiple microservices and simulates a new user signup workflow.
The other example uses a Flask server with [Marvin](https://www.askmarvin.ai/) to ask questions of an LLM from the CLI and get back answers.

The examples build on the ones that come before.

### Setup

<summary>Expand</summary>

Step 1: Activate a virtual environment

The following example uses [conda](https://conda.io/projects/conda/en/latest/user-guide/install/index.html), but any virtual environment manager will work.

```bash
conda deactivate
conda create -n python-tasks python=3.12
conda activate python-tasks
```

Step 2: Install Python dependencies

```bash
pip install -U prefect marvin fastapi==0.107
```

Step 3: Connect to Prefect Cloud or a local Prefect server instance (if not set already)

You can use either Prefect Cloud or a local Prefect server instance for these examples.

You need to have `PREFECT_API_URL` set to send tasks to task servers.

If you're using a local Prefect server instance with a SQLite backing database (the default database), you can save this value to your active Prefect Profile by running the following command in your terminal.

```bash
prefect config set PREFECT_API_URL=http://127.0.0.1:4200/api
```

If using Prefect Cloud, set the `PREFECT_API_URL` value to the Prefect Cloud API URL and add your [API key](https://docs.prefect.io/cloud/users/api-keys/).

The examples that use docker (examples 4 and 5) use a local Prefect server instance by default.
You can switch to Prefect Cloud by changing the `PREFECT_API_URL` and adding a variable for your API key in the `docker-compose.yaml`.
Or use a local server instance backed by a PostgreSQL database by setting the `PREFECT_API_DATABASE_CONNECTION_URL`.

If using a local Prefect server instance instead of Prefect Cloud, start your server by running the following command:

```bash
prefect server start
```

Step 4: Clone the repository (optional)

You can code from scratch or clone the repository to get the code files for the examples.

```bash
git clone https://github.com/PrefectHQ/prefect-background-task-examples.git
```

Move into the directory.

```bash
cd prefect-background-task-examples
```

Let's run some tasks!

### Example 1: Run a Prefect task in the foreground by calling it

<summary>Expand</summary>

Add the `@task` decorator to any Python function to define a Prefect task.

Step 1: Create a file named `greeter.py` and save the following code in it, or run the existing file in the basic-examples directory.

```python
from prefect import task

@task(log_prints=True)
def greet(name: str = "Marvin"):
print(f"Hello, {name}!")

if __name__ == "__main__":
greet()
```

Step 2: Run the script in the terminal.

```bash
python greeter.py
```

You should see the task run in the terminal. This task runs in the foreground. In other words, it is not deferred.

#### Optional

You can see the task run in the UI.
If you're using a self-hosted Prefect Server instance, you can also see the task runs in the database.

If you want to inspect the SQLite database, use your favorite interface.
We explain how to use *DB Browser for SQLite* below.

Download it [here](https://sqlitebrowser.org/dl/), if needed. Install it and open it.

Click *Connect*. Then navigate to your SQLite DB file. It will be in the `~/.prefect` directory by default.

Head to the `task_run` table and you should see all your task runs there.
You can scroll down to see your most recent task runs or filter for them.

Hit the refresh button for updates, if needed.


### Example 2: Start a task server and run deferred tasks in the background

<summary>Expand</summary>

In this example, we'll start a task server and run deferred tasks in the background.

To run tasks in a separate process or container, you'll need to start a task server, similar to how you would run a Celery worker or an arq worker.
The task server will continually receive scheduled tasks to execute from Prefect's API, execute them, and report the results back to the API.
You can run a task server by passing tasks into the `prefect.task_server.serve()` method.

Step 1: Define the task and task server in the file `task_server.py`

```python
from prefect import task
from prefect.task_server import serve


@task
def my_background_task(name: str):
print(f"Hello, {name}!")


if __name__ == "__main__":
serve(my_background_task)
```

Step 2: Start the task server by running the script in the terminal.

```bash
python task_server.py
```

The task server is now waiting for runs of the `my_background_task` task.
Let's give it some task runs.

Step 3: Create a file named `task_scheduler.py` and save the following code in it.

```python
from tasks import my_background_task

if __name__ == "__main__":
my_background_task.delay("Agrajag")
```

Step 4: Open another terminal and run the script.

```bash
python task_scheduler.py
```

Note that we return the a "future" from the `delay` method. You can use this object to wait for the task to complete with `wait()` and to retrieve its result with `result()`.
We can also see the task run's UUID and other information about the task run.

Step 5: See the task run in the UI.

Use the task run UUID to see the task run in the UI.
The URL will look like this:

`http://127.0.0.1:4200/task-runs/task-run/my_task_run_uuid_goes_here`

Substitute your UUID at the end of the URL.
Note that the UI navigation experience for task runs will be improved soon.

Step 6: You can use multiple task servers to run tasks in parallel.

Start another instance of the task server. In another terminal run:

```bash
python task_server.py
```

Step 7: Send multiple tasks to the task server.

Modify the `task_scheduler.py` file to send multiple tasks to the task server with different inputs:

```python
from tasks import my_background_task

if __name__ == "__main__":
my_background_task.delay("Ford")
my_background_task.delay("Prefect")
my_background_task.delay("Slartibartfast")
```

Run the file and watch the work get distributed across both task servers!

Step 8: Shut down the task servers with *control* + *c*.

Alright, you're able to send tasks to multiple Prefect task servers running in the background!
This is cool because we can observe these tasks executing in parallel and very quickly with web sockets - no polling required.

See additional examples in the [deferred tasks github repository](https://github.com/PrefectHQ/prefect-background-task-examples.git).
2 changes: 1 addition & 1 deletion docs/3.0rc/guides/host.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ prefect server start

![Viewing the dashboard in the Prefect UI.](/3.0rc/img/ui/self-hosted-server-dashboard.png)

3. Shut down the Prefect server with \<kdb> ctrl \</kbd> + \<kdb> c \</kbd> in the terminal.
3. Shut down the Prefect server with ctrl + c in the terminal.

### Comparing a self-hosted Prefect server instance and Prefect Cloud

Expand Down
1 change: 1 addition & 0 deletions docs/3.0rc/guides/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ description: Learn how to do common workflows with Prefect.
| [Specifying Upstream Dependencies](/3.0rc/guides/specifying-upstream-dependencies/) | Run tasks in a desired order. |
| [Third-party Secrets](/3.0rc/guides/secrets/) | Use credentials stored in a secrets manager in your workflows. |
| [Prefect Recipes](/3.0rc/recipes/recipes/) | Common, extensible examples for setting up Prefect. |
| [Using Deferred Tasks](/3.0rc/guides/deferred-tasks.mdx/) | Examples of using Prefect tasks and the task server. |

## Execution

Expand Down
3 changes: 2 additions & 1 deletion docs/mint.json
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,8 @@
"3.0rc/guides/ci-cd",
"3.0rc/guides/specifying-upstream-dependencies",
"3.0rc/guides/secrets",
"3.0rc/guides/recipes"
"3.0rc/guides/recipes",
"3.0rc/guides/deferred-tasks"
]
},
{
Expand Down