Skip to content

Commit

Permalink
Update integration links and content (#14096)
Browse files Browse the repository at this point in the history
  • Loading branch information
discdiver committed Jun 18, 2024
1 parent d615611 commit 9e48e20
Show file tree
Hide file tree
Showing 45 changed files with 248 additions and 369 deletions.
58 changes: 0 additions & 58 deletions docs/3.0rc/guides/index.mdx

This file was deleted.

File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes.
File renamed without changes
6 changes: 0 additions & 6 deletions docs/integrations/prefect-aws/deployments/steps.mdx

This file was deleted.

14 changes: 7 additions & 7 deletions docs/integrations/prefect-aws/ecs_guide.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ ECS (Elastic Container Service) tasks are a good option for executing Prefect fl

## ECS flow run execution

Prefect enables remote flow execution via [workers](https://docs.prefect.io/deploy/dynamic-infra/control-runs/#worker-overview) and [work pools](https://docs.prefect.io/deploy/dynamic-infra/control-runs/#work-pool-overview). To learn more about these concepts please see our [deployment tutorial](https://docs.prefect.io/deploy/serve-flows/).
Prefect enables remote flow execution via workers and work pools. To learn more about these concepts please see our [deployment docs](/3.0rc/deploy/work-pools/control-runs).

For details on how workers and work pools are implemented for ECS, see the diagram below.

Expand Down Expand Up @@ -69,7 +69,7 @@ graph TB
<Tip>
**ECS tasks != Prefect tasks**

An ECS task is **not** the same thing as a [Prefect task](https://docs.prefect.io/latest/develop/write-tasks/#tasks-overview).
An ECS task is **not** the same thing as a [Prefect task](3.0rc/develop/write-tasks/index).

ECS tasks are groupings of containers that run within an ECS Cluster. An ECS task's behavior is determined by its task definition.
</Tip>
Expand All @@ -78,9 +78,9 @@ An [*ECS task definition*](https://docs.aws.amazon.com/AmazonECS/latest/develope

ECS tasks are instances of a task definition. A Task Execution launches container(s) as defined in the task definition **until they are stopped or exit on their own**. This setup is ideal for ephemeral processes such as a Prefect flow run.

The ECS task running the Prefect [worker](https://docs.prefect.io/latest/deploy/dynamic-infra/control-runs/#worker-overview) should be an [**ECS Service**](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_services.html), given its long-running nature and need for **auto-recovery in case of failure**. An ECS service automatically replaces any task that fails, which is ideal for managing a long-running process such as a Prefect worker.
The ECS task running the Prefect worker should be an [**ECS Service**](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_services.html), given its long-running nature and need for **auto-recovery in case of failure**. An ECS service automatically replaces any task that fails, which is ideal for managing a long-running process such as a Prefect worker.

When a Prefect [flow](https://docs.prefect.io/latest/develop/write-flows/) is scheduled to run it goes into the work pool specified in the flow's [deployment](https://docs.prefect.io/latest/deploy/serve-flows). [Work pools](https://docs.prefect.io/latest/deploy/dynamic-infra/control-runs/?h=work#work-pool-overview) are typed according to the infrastructure the flow will run on. Flow runs scheduled in an `ecs` typed work pool are executed as ECS tasks. Only Prefect ECS [workers](https://docs.prefect.io/latest/deploy/dynamic-infra/control-runs/#worker-types) can poll an `ecs` typed work pool.
When a Prefect flow is scheduled to run it goes into the work pool specified in the flow's deployment. Work pools are typed according to the infrastructure the flow will run on. Flow runs scheduled in an `ecs` typed work pool are executed as ECS tasks. Only Prefect ECS workers can poll an `ecs` typed work pool.

When the ECS worker receives a scheduled flow run from the ECS work pool it is polling, it spins up the specified infrastructure on AWS ECS. The worker knows to build an ECS task definition for each flow run based on the configuration specified in the work pool.

Expand All @@ -91,7 +91,7 @@ If you specify a task definition [ARN (Amazon Resource Name)](https://docs.aws.a
You can use either EC2 or Fargate as the capacity provider. Fargate simplifies initiation, but lengthens infrastructure setup time for each flow run. Using EC2 for the ECS cluster can reduce setup time. In this example, we will show how to use Fargate.

<Tip>
If you prefer infrastructure as code check out this [Terraform module](https://github.com/PrefectHQ/prefect-recipes/tree/main/devops/infrastructure-as-code/aws/tf-prefect2-ecs-worker) to provision an ECS cluster with a worker.
If you prefer infrastructure as code check out this [Terraform module](https://registry.terraform.io/providers/PrefectHQ/prefect/latest/docs/resources/work_pool) to provision an ECS cluster with a worker.
</Tip>

## Prerequisites
Expand All @@ -104,7 +104,7 @@ If you prefer infrastructure as code check out this [Terraform module](https://g

## Step 1: Set up an ECS work pool

Before setting up the worker, create a [work pool](https://docs.prefect.io/latest/deploy/dynamic-infra/control-runs/#work-pool-configuration) of type ECS for the worker to pull work from. If doing so from the CLI, be sure to [authenticate with Prefect Cloud](https://docs.prefect.io/latest/cloud/cloud-quickstart/#log-into-prefect-cloud-from-a-terminal).
Before setting up the worker, create a [work pool](3.0rc/deploy/work-pools/control-runs) of type ECS for the worker to pull work from. If doing so from the CLI, be sure to authenticate with Prefect Cloud or run a local Prefect server instance.

Create a work pool from the CLI:

Expand All @@ -113,7 +113,7 @@ prefect work-pool create --type ecs my-ecs-pool
```

Or from the Prefect UI:
![WorkPool](img/Workpool_UI.png)
![WorkPool](images/Workpool_UI.png)

Because this guide uses Fargate as the capacity provider and the default VPC and ECS cluster, no further configuration is needed.

Expand Down
12 changes: 5 additions & 7 deletions docs/integrations/prefect-aws/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -14,31 +14,29 @@ For example, you can retrieve secrets using AWS Secrets Manager, read and write

### Install prefect-aws

<div class = "terminal">

```bash
pip install -U prefect-aws
```
</div>


### Register newly installed block types

Register the block types in the prefect-aws module to make them available for use.

<div class = "terminal">
```bash
prefect block register -m prefect_aws
```
</div>

## Examples

### Run flows on AWS ECS

Run flows on [AWS Elastic Container Service (ECS)](https://aws.amazon.com/ecs/) to dynamically scale your infrastructure.

See the [ECS guide](/3.0rc/ecs_guide/) for a walkthrough of using ECS in a hybrid work pool.
See the [ECS work pool docs](/integrations/prefect-aws/ecs_guide) for a walkthrough of using ECS in a hybrid work pool.

If you're using Prefect Cloud, [ECS push work pools](https://docs.prefect.io/latest/deploy/dynamic-infra/push-runs-serverless/#__tabbed_1_1) provide all the benefits of ECS with a quick setup and no worker needed.
If you're using Prefect Cloud, [ECS push work pools](/3.0rc/deploy/dynamic-infra/push-runs-serverless) provide all the benefits of ECS with a quick setup and no worker needed.

In the examples below, you create blocks with Python code.
Alternatively, each block can be created through the Prefect UI.
Expand Down Expand Up @@ -70,7 +68,7 @@ Prefect is using the Boto3 library under the hood.
To find credentials for authentication, any data not provided to the block are sourced at runtime in the order shown in the [Boto3 docs](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
Prefect creates the session object using the values in the block and then, any missing values follow the sequence in the Boto3 docs.

See an example of using the `AwsCredentials` block with [AWS Secrets Manager](#aws-secrets-manager) with third-party services without storing credentials in the block itself in [this guide](https://docs.prefect.io/latest/guides/secrets/).
See an example of using the `AwsCredentials` block with [AWS Secrets Manager](#aws-secrets-manager) with third-party services without storing credentials in the block itself in the [docs](/3.0rc/resources/secrets).

Here's how to load the saved credentials:

Expand Down
4 changes: 4 additions & 0 deletions docs/integrations/prefect-aws/sdk.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
---
title: "SDK docs"
url: "https://prefect-python-sdk-docs.netlify.app/prefect_aws/"
---
26 changes: 12 additions & 14 deletions docs/integrations/prefect-azure/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -9,36 +9,36 @@ For example, you can retrieve secrets, read and write Blob Storage objects, and

### Prerequisites

- [Prefect installed](https://docs.prefect.io/latest/get-started/install/) in a virtual environment.
- [Prefect installed](3.0rc/get-started/install) in a virtual environment.
- An [Azure account](https://azure.microsoft.com/) and the necessary permissions to access desired services.

### Install prefect-azure

<div class = "terminal">

```bash
pip install -U prefect-azure
```
</div>


If necessary, see [additional installation options for Blob Storage, Cosmos DB, and ML Datastore](#additional-installation-options).

To install with all additional functionality, use the following command:

<div class = "terminal">

```bash
pip install -U "prefect-azure[all_extras]"
```
</div>


### Register newly installed block types

Register the block types in the module to make them available for use.

<div class = "terminal">

```bash
prefect block register -m prefect_azure
```
</div>


## Examples

Expand Down Expand Up @@ -80,7 +80,7 @@ custom_blob_storage_download_flow = example_blob_storage_download_flow.with_opti

Run flows on [Azure Container Instances (ACI)](https://learn.microsoft.com/en-us/azure/container-instances/) to dynamically scale your infrastructure.

See the [Azure Container Instances Worker Guide](/3.0rc/aci_worker/) for a walkthrough of using ACI in a hybrid work pool.
See the [Azure Container Instances Worker Guide](/3.0rc/integrations/prefect-azure/aci_worker/) for a walkthrough of using ACI in a hybrid work pool.

If you're using Prefect Cloud, [ACI push work pools](/3.0rc/deploy/dynamic-infra/push-runs-serverless/#__tabbed_1_2) provide all the benefits of ACI with a quick setup and no worker needed.

Expand All @@ -94,24 +94,22 @@ Refer to the prefect-azure API documentation linked in the sidebar to explore al

To use Blob Storage:

<div class="terminal">
```bash
pip install -U "prefect-azure[blob_storage]"
```
</div>


To use Cosmos DB:

<div class="terminal">
```bash
pip install -U "prefect-azure[cosmos_db]"
```
</div>


To use ML Datastore:

<div class="terminal">

```bash
pip install -U "prefect-azure[ml_datastore]"
```
</div>

2 changes: 1 addition & 1 deletion docs/integrations/prefect-azure/sdk.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: "SDK docs"
url: "https://prefect-python-sdk-docs.netlify.app/prefect-azure/"
url: "https://prefect-python-sdk-docs.netlify.app/prefect_azure/"
---
10 changes: 5 additions & 5 deletions docs/integrations/prefect-bitbucket/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -13,21 +13,21 @@ The prefect-bitbucket library makes it easy to interact with Bitbucket repositor

### Install prefect-bitbucket

<div class = "terminal">

```bash
pip install -U prefect-bitbucket
```
</div>


### Register newly installed block types

Register the block types in the prefect-bitbucket module to make them available for use.

<div class = "terminal">

```bash
prefect block register -m prefect_bitbucket
```
</div>


## Examples

Expand All @@ -41,7 +41,7 @@ To create a deployment and run a deployment where the flow code is stored in a p
A deployment can use flow code stored in a Bitbucket repository without using this library in either of the following cases:

- The repository is public
- The deployment uses a [Secret block](https://docs.prefect.io/latest/develop/connect-third-party/) to store the token
- The deployment uses a [Secret block](/3.0rc/resources/secrets) to store the token

Create a Bitbucket Credentials block:

Expand Down
1 change: 0 additions & 1 deletion docs/integrations/prefect-dask/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -210,7 +210,6 @@ Run the code and check how long it takes each task to complete.

## Resources

For additional examples, check out the [Usage Guide](/3.0rc/usage_guide/).
Refer to the prefect-dask API documentation linked in the sidebar to explore all the capabilities of the prefect-dask library.

For assistance using Dask, consult the [Dask documentation](https://docs.dask.org/en/stable/)
4 changes: 1 addition & 3 deletions docs/integrations/prefect-databricks/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -12,16 +12,14 @@ The service's REST API documentation can be found [here](https://docs.databricks

### Prerequisites

- [Prefect installed](https://docs.prefect.io/latest/get-started/install/) in a virtual environment.
- [Prefect installed](/3.0rc/get-started/install) in a virtual environment.
- A [Databricks account](https://databricks.com/) and the necessary permissions to access desired services.

### Install `prefect-databricks`

<div class="terminal">
```bash
pip install prefect-databricks
```
</div>

### List jobs on the Databricks instance

Expand Down
1 change: 0 additions & 1 deletion docs/integrations/prefect-dbt/cli/commands.mdx

This file was deleted.

12 changes: 6 additions & 6 deletions docs/integrations/prefect-dbt/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Prefect provides a global view of the state of your workflows and allows you to

### Prerequisites

- [Prefect installed](https://docs.prefect.io/latest/get-started/install/).
- [Prefect installed](3.0rc/get-started/install).
- A [dbt Cloud account](https://cloud.getdbt.com/signup) if using dbt Cloud.

### Install prefect-dbt
Expand Down Expand Up @@ -39,7 +39,7 @@ Register the block types in the prefect-dbt module to make them available for us

```bash
prefect block register -m prefect_dbt

```

Explore the examples below to use Prefect with dbt.

Expand Down Expand Up @@ -188,14 +188,14 @@ DbtCloudJob.load("JOB-BLOCK-NAME-PLACEHOLDER")

**Available `TargetConfigs` blocks**

Visit the [API Reference](cli/configs/base) to see other built-in `TargetConfigs` blocks.
Visit the SDK reference in the side navigation to see other built-in `TargetConfigs` blocks.

If the desired service profile is not available, check out the [Examples Catalog](examples_catalog/#clicredentials-module) to see how you can build one from the generic `TargetConfigs` class.
If the desired service profile is not available, you can build one from the generic `TargetConfigs` class.
</Info>

To create dbt Core target config and profile blocks for BigQuery:

1. Save and load a [`GcpCredentials` block](https://prefecthq.github.io/prefect-gcp/#saving-credentials-to-a-block).
1. Save and load a `GcpCredentials` block.
2. Determine the schema / dataset you want to use in BigQuery.
3. Create a short script, replacing the placeholders.

Expand Down Expand Up @@ -287,5 +287,5 @@ According to dbt's [Databricks setup page](https://docs.getdbt.com/reference/war
pip install dbt-databricks
```

Check out the [desired profile setup page](https://docs.getdbt.com/reference/profiles.yml) on the sidebar for others.
Check out the [desired profile setup page](https://docs.getdbt.com/reference/profiles.yml) for other configuration.
</Warning>
2 changes: 1 addition & 1 deletion docs/integrations/prefect-dbt/sdk.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---
title: "SDK docs"
url: "https://prefect-python-sdk-docs.netlify.app/prefect-dbt/"
url: "https://prefect-python-sdk-docs.netlify.app/prefect_dbt/"
---
Loading

0 comments on commit 9e48e20

Please sign in to comment.