Merge pull request #1052 from PrefectHQ/cloud

Add cloud features
PrefectHQ · Mar 14, 2022 · bceb788 · bceb788
2 parents 38ff98e + 68e8d1b
commit bceb788
Show file tree

Hide file tree

Showing 35 changed files with 546 additions and 40 deletions.
diff --git a/docs/img/ui/cloud-api-keys.png b/docs/img/ui/cloud-api-keys.png
diff --git a/docs/img/ui/cloud-edit-profile.png b/docs/img/ui/cloud-edit-profile.png
diff --git a/docs/img/ui/cloud-edit-workspace.png b/docs/img/ui/cloud-edit-workspace.png
diff --git a/docs/img/ui/cloud-flow-run.png b/docs/img/ui/cloud-flow-run.png
diff --git a/docs/img/ui/cloud-new-login.png b/docs/img/ui/cloud-new-login.png
diff --git a/docs/img/ui/cloud-workspace-dashboard.png b/docs/img/ui/cloud-workspace-dashboard.png
diff --git a/docs/img/ui/cloud-workspace-details.png b/docs/img/ui/cloud-workspace-details.png
diff --git a/docs/img/ui/cloud-workspace-list.png b/docs/img/ui/cloud-workspace-list.png
diff --git a/docs/img/ui/flows-icon.png b/docs/img/ui/flows-icon.png
diff --git a/docs/img/ui/orion-dash-details.png b/docs/img/ui/orion-dash-details.png
diff --git a/docs/img/ui/orion-dash-filters.png b/docs/img/ui/orion-dash-filters.png
diff --git a/docs/img/ui/orion-dash-history.png b/docs/img/ui/orion-dash-history.png
diff --git a/docs/img/ui/orion-dashboard.png b/docs/img/ui/orion-dashboard.png
diff --git a/docs/img/ui/orion-deployments.png b/docs/img/ui/orion-deployments.png
diff --git a/docs/img/ui/orion-filters-failed.png b/docs/img/ui/orion-filters-failed.png
diff --git a/docs/img/ui/orion-filters-panel.png b/docs/img/ui/orion-filters-panel.png
diff --git a/docs/img/ui/orion-flow-radar.png b/docs/img/ui/orion-flow-radar.png
diff --git a/docs/img/ui/orion-flow-run-details.png b/docs/img/ui/orion-flow-run-details.png
diff --git a/docs/img/ui/orion-flow-run-logs.png b/docs/img/ui/orion-flow-run-logs.png
diff --git a/docs/img/ui/orion-flow-runs.png b/docs/img/ui/orion-flow-runs.png
diff --git a/docs/img/ui/orion-task-runs.png b/docs/img/ui/orion-task-runs.png
diff --git a/docs/img/ui/work-queue-create.png b/docs/img/ui/work-queue-create.png
diff --git a/docs/img/ui/work-queue-details.png b/docs/img/ui/work-queue-details.png
diff --git a/docs/img/ui/work-queue-list.png b/docs/img/ui/work-queue-list.png
diff --git a/docs/img/ui/work-queues-icon.png b/docs/img/ui/work-queues-icon.png
diff --git a/docs/img/ui/workspace-icon.png b/docs/img/ui/workspace-icon.png
diff --git a/docs/stylesheets/extra.css b/docs/stylesheets/extra.css
@@ -9,6 +9,8 @@
 
 /* Image drop shadow, separates browser from background */
 img {
+    margin-top: 1rem;
+    margin-bottom: 1rem;
     filter: drop-shadow(0.2rem 0.2rem 0.3rem rgba(0,0,0,0.2))
 }
 

diff --git a/docs/ui/cloud.md b/docs/ui/cloud.md
@@ -0,0 +1,172 @@
+---
+description: Using Prefect Cloud, including account creation, team and workspace management, and running flows.
+tags:
+    - UI
+    - dashboard
+    - Cloud
+    - accounts
+    - teams
+    - workspaces
+---
+
+# Prefect Cloud
+
+Prefect Cloud is an orchestration-as-a-service platform. Prefect Cloud provides all the capabilities of the [Orion UI](/ui/overview/) in a hosted environment, including:
+
+- Flow run summaries
+- Deployed flow details
+- Scheduled flow runs
+- Warnings for late or failed runs
+- Task run details 
+- Radar flow and task dependency visualizer 
+- Logs
+
+Features only available on Prefect Cloud include:
+
+- User accounts
+- Workspaces
+
+The following sections will get you set up and using Prefect Cloud, following these steps:
+
+1. [Sign in or register](#sign-in-or-register) a Prefect Cloud account.
+2. [Create workspaces](#create-a-workspace) for your account.
+3. [Create an API key](#create-an-api-key) to authorize a local execution environment.
+4. [Configure Orion settings](#configure-orion-for-cloud) to use Prefect Cloud.
+5. [Configure storage](#configure-storage).
+6. [Run a flow](#run-a-flow-with-cloud) and display the flow run in Prefect Cloud.
+
+## Sign in or register
+
+To sign in with an existing account or register an account, go to [http://beta.prefect.io/](http://beta.prefect.io/).
+
+You can create an account with:
+
+- Google account
+- GitHub account
+- Email and password
+
+## Create a workspace
+
+If you register a new account, you'll be prompted to create a workspace.  
+
+![Creating a new Prefect Cloud account.](/img/ui/cloud-new-login.png)
+
+Click **Create Workspace**. You'll be prompted to provide a name and description for your workspace.
+
+![Creating a new workspace in the Cloud UI.](/img/ui/cloud-workspace-details.png)
+
+Click **Create** to create the workspace. 
+
+![Viewing a list of available workspaces in the Cloud UI.](/img/ui/cloud-workspace-list.png)
+
+Click **Edit Workspace**. This lets you edit details about the workspace or delete the workspace. 
+
+It also provide the Prefect CLI command that configures Prefect to orchestrate flow runs with this workspace. Copy this command and run it in the environment in which you'll be running flows so they'll show up in your workspace.
+
+![Editing a workspace.](/img/ui/cloud-edit-workspace.png)
+
+Click the Prefect logo: this always returns to your workspace list. Then click on a workspace name to view the dashboard for that workspace.
+
+![Viewing a workspace dashboard in the Cloud UI.](/img/ui/cloud-workspace-dashboard.png)
+
+## Create an API key
+
+API keys enable you to authenticate an a local environment to work with Prefect Cloud. See [Configuring Orion for Cloud](#configuring-orion-for-cloud) for details on how API keys are configured in your execution environment.
+
+To create an API key, click the account icon at the bottom-left corner of the UI, then click **Profile**. This displays your account profile.
+
+![Viewing an account profile in the Cloud UI.](/img/ui/cloud-edit-profile.png)
+
+Click the **API Keys** tab. This displays a list of previously generated keys and lets you create new API keys or delete keys.
+
+![Editing and creating API keys in the Cloud UI.](/img/ui/cloud-api-keys.png)
+
+Click **Create** to create a new API key. You're prompted to provide a name for the key. Click **Confirm** to generate the key.
+
+Note that API keys cannot be revealed again in the UI after you generate them, so copy the key to a secure location.
+
+## Configure Orion for Cloud
+
+Your next step is to configure a local execution environment to use Cloud as the API server for local flow runs.
+
+First, [Install Orion](/getting-started/installation/) in the environment in which you want to execute flow runs.
+
+Next, use the Prefect CLI `prefect cloud login` command to log into Prefect Cloud from your environment, using the [API key](#create-an-api-key) generated previously.
+
+```bash
+$ prefect cloud login --key xxx_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
+```
+
+It will prompt you to choose a workspace if you haven't given one (you can specify a workspace with the `-w` or `--workspace` option).
+
+```bash
+$ prefect cloud login --key xxx_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
+┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
+┃              Select a Workspace: ┃
+┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
+│ > tprefectio/tp-workspace        │
+└──────────────────────────────────┘
+Successfully logged in and set workspace to 'tprefectio/tp-workspace' in profile:
+'default'.
+```
+
+It then sets `PREFECT_API_KEY` and `PREFECT_API_URL` for the current profile.
+
+Now you're ready to run flows locally and have the results displayed in the Cloud UI.
+
+The `prefect cloud logout` CLI command unsets those settings in the current profile, logging the environment out of interaction with Prefect Cloud.
+
+### Manually configuring Cloud settings
+
+Note that you can also manually configure the settings to interact with Prefect Cloud using an account ID, workspace ID, and API key.
+
+```BASH
+$ prefect config set PREFECT_API_URL="https://beta.prefect.io/api/accounts/[ACCOUNT-ID]/workspaces/[WORKSPACE-ID]"
+$ prefect config set PREFECT_API_KEY="[API-KEY]"
+```
+
+When you're in a Prefect Cloud workspace, you can copy the API URL directly from the page URL, or copy the entire command string from the **Workspace Details** page.
+
+In this example, we configured `PREFECT_API_URL` and `PREFECT_API_KEY` in the default profile. You can use `prefect profile` CLI commands to create settings profiles for different configurations. For example, you could have a profile configured to use the Cloud API URL and API key, and another profile for local development using a local Orion API server. See [Settings](/concepts/settings/) for details.
+
+## Configure storage 
+
+When using Prefect Cloud, we recommend configuring global storage for persisting flow and task data. See [Storage](/concepts/storage/) for details.
+
+By default, Orion uses local file system storage to persist flow code and flow and task results. For local development and testing this may be adequate. Be aware, however, that local storage is not guaranteed to persist data reliably between flow or task runs, particularly when using containers or distributed computing environments like Dask and Ray.
+
+## Run a flow with Cloud
+
+Okay, you're all set to run a local flow with Prefect Cloud. Notice that everything works just like running local flows with the Prefect Orion API server, but because you configured `PREFECT_API_URL` and `PREFECT_API_KEY`, your flow runs show up in Cloud!
+
+In your local environment, where you configured the previous steps, create a file named `basic_flow.py` with the following contents:
+
+```python
+from prefect import flow, get_run_logger
+
+@flow(name="Testing")
+def basic_flow():
+    logger = get_run_logger()
+    logger.warning("The fun is about to begin")
+
+if __name__ == "__main__":
+    basic_flow()
+```
+
+Now run `basic_flow.py`.
+
+<div class='termy'>
+```
+$ python basic_flow.py
+11:31:46.135 | INFO    | prefect.engine - Created flow run 'delicate-woodpecker' for flow 'Testing'
+11:31:46.135 | INFO    | Flow run 'delicate-woodpecker' - Using task runner 'ConcurrentTaskRunner'
+11:31:46.748 | WARNING | Flow run 'delicate-woodpecker' - The fun is about to begin
+11:31:47.643 | INFO    | Flow run 'delicate-woodpecker' - Finished in state Completed(None)
+```
+</div>
+
+Go to the dashboard for your workspace in Prefect Cloud. You'll see the flow run results right there in Cloud!
+
+![Viewing local flow run results in the Cloud UI.](/img/ui/cloud-flow-run.png)
+
+To run deployments using the API or directly from the Cloud UI, you'll need to configure [work queues](/ui/work-queues/) and agents. See the [Work Queues & Agents](/concepts/work-queues/) documentation for details, and the [Deployments tutorial](/tutorials/deployments/#work-queues-and-agents) for a hands-on example.
diff --git a/docs/ui/overview.md b/docs/ui/overview.md
@@ -7,9 +7,14 @@ tags:
     - Cloud
 ---
 
-# Orion UI Overview
+# Orion UI & Cloud Overview
 
-The Prefect Orion UI provides an overview of all of your flows. It was designed around a simple question: what's the health of my system?
+The Prefect UI provides an overview of all of your flows. It was designed around a simple question: what's the health of my system?
+
+There are two ways to access the UI:
+
+- The Prefect [Orion UI](#using-the-orion-ui) gives you insight into the flows running with any local Orion server instance.
+- [Prefect Cloud](/ui/cloud/) is a hosted service that provides all the capabilities of the Orion UI, plus personal accounts and workspaces.
 
 The UI displays many useful insights about your flow runs, including:
 
@@ -25,34 +30,7 @@ You can filter the information displayed in the UI by time, flow state, and tags
 
 ## Using the Orion UI
 
-The UI is available in any environment where the Prefect Orion server is running with `prefect orion start`.
-
-When Prefect Server is running, you can access the UI at [http://127.0.0.1:4200](http://127.0.0.1:4200).
-
-![Prefect Orion UI dashboard.](/img/ui/orion-dashboard.png)
-
-The following sections provide details about Orion UI pages and visualizations:
-
-- [Dashboard](/ui/dashboard/) provides a high-level overview of your flows, tasks, and deployments.
-- [Flows and Tasks](/ui/flows-and-tasks/) pages let you dig into details of flow runs and task runs.
-- [Filters](/ui/filters/) enable you to customize the display based on flow state, tags, execution time, and more.
-
-<!-- ## Prefect Cloud
-
-Prefect Cloud provides a hosted server and UI instance for running and monitoring deployed flows. Prefect Cloud includes:
-
-- All of the UI features of the local Orion server UI.
-- The ability to invite and manage teams. 
-- The option to create multiple workspaces to organize flows by team, project, or business function.
-
-See the [Prefect Cloud](/ui/cloud.md) documentation for details about setting up accounts, workspaces, and teams. -->
-
-<!-- This section to be removed for moved to dev docs closer to release -->
-## Using the dev UI
-
-If you're working directly with the Prefect Orion repository &mdash; whether for contributing to Prefect or simply developing against the very latest codebase &mdash; you may need to utilize the `prefect dev` CLI commands to build the UI or launch a hot-reloading server instance.
-
-If the UI has not been built, you'll see output like the following noting that the "dashboard is not built" when starting the Orion server:
+The Orion UI is available in any environment where the Prefect Orion server is running with `prefect orion start`.
 
 ```bash
 $ prefect orion start
@@ -65,18 +43,41 @@ Starting...
 
 Configure Prefect to communicate with the server with:
 
-    prefect config set PREFECT_ORION_HOST=http://127.0.0.1:4200/api
+    prefect config set PREFECT_API_URL=http://127.0.0.1:4200/api
 
-The dashboard is not built. It looks like you're on a development version.
-See `prefect dev` for development commands.
+Check out the dashboard at http://127.0.0.1:4200
 ```
 
-To install dependencies and build the UI, run the following command:
+When Prefect Orion server is running, you can access the UI at [http://127.0.0.1:4200](http://127.0.0.1:4200).
 
-```bash
-prefect dev build-ui
-```
+![Prefect Orion UI dashboard.](/img/ui/orion-dashboard.png)
 
-When the build process completes, start the server and UI with `prefect orion start`.
+The following sections provide details about Orion UI pages and visualizations:
+
+- [Dashboard](/ui/dashboard/) provides a high-level overview of your flows, tasks, and deployments.
+- [Flows and Tasks](/ui/flows-and-tasks/) pages let you dig into details of flow runs and task runs.
+- [Filters](/ui/filters/) enable you to customize the display based on flow state, tags, execution time, and more.
+- [Work Queues](/ui/work-queues/) enable you to create and manage work queues that enable agents to pick up flow runs.
+
+## Navigating the UI
+
+Icons on the left side of the Orion UI help you navigate to commonly used pages.
+
+The Prefect icon always takes you back to the Orion UI dashboard. In Prefect Cloud, it returns you to the list of workspaces.
+
+| Icon | Description |
+| --- | --- |
+| ![Workspace](/img/ui/workspace-icon.png) | **Workspace** returns to the dashboard of the current workspace. ([Prefect Cloud](#prefect-cloud) only) |
+| ![Flows](/img/ui/flows-icon.png) | **Flows** displays a searchable list of flows tracked by the API. |
+| ![Work Queues](/img/ui/work-queues-icon.png) | **Work Queues** displays configured [work queues](/ui/work-queues/) and enables creating new work queues. |
+
+## Prefect Cloud
+
+[Prefect Cloud](https://beta.prefect.io) provides a hosted server and UI instance for running and monitoring deployed flows. Prefect Cloud includes:
+
+- All of the UI features of the local Orion server UI.
+- A personal account and workspace.
+- API keys to sync deployments and flow runs with the Prefect Cloud API.
+- A hosted Orion metadata database that stores flow and task run history.
 
-To see other options for running the server or UI in development mode, run `prefect dev --help`.
+See the [Prefect Cloud](/ui/cloud/) documentation for details about setting up accounts, workspaces, and API keys.
diff --git a/docs/ui/work-queues.md b/docs/ui/work-queues.md
@@ -0,0 +1,49 @@
+---
+description: Create and manage work queues from the Orion or Cloud UI.
+tags:
+    - Orion
+    - UI
+    - deployments
+    - flow runs
+    - Cloud
+    - work queues
+    - agents
+    - tags
+---
+
+# Work Queues
+
+[Work Queues](/concepts/work-queues/) and agents work together to bridge your orchestration environment &mdash; Orion or Prefect Cloud &mdash; and your execution environments. Work queues gather flow runs for scheduled deployments, and agents pick up work from their configured work queues.
+
+Work queue configuration lets you specify which queues handle which flow runs. You can filter runs based on tags, flow runners, and even specific deployments.
+
+You can create, edit, manage, and delete work queues through the Orion or Prefect Cloud UI or [Prefect CLI commands](/concepts/work-queues/#work-queue-configuration).
+
+## Managing work queues
+
+To manage work queues in the UI, click the **Work Queues** icon. This displays a list of currently configured work queues.
+
+![The UI displays a list of configured work queues](/img/ui/work-queue-list.png)
+
+For each work queue you'll see its:
+
+- Name
+- Status (for example, "active")
+- Tags (the work queue serves flow runs with these tags)
+- Concurrency limit (the maximum number of flow runs that can be running concurrently on this queue)
+
+Click **Create Work Queue** to create a new work queue. You'll be able to specify the details for work served by this queue.
+
+![Creating a new work queue in the Orion UI](/img/ui/work-queue-create.png)
+
+Click on the name of any work queue to see details about it. This panel shows you the:
+
+- Status
+- CLI command to [run an agent](/concepts/work-queues/#agent-overview) that polls this work queue
+- Configuration details
+
+You can also pause a work queue from this panel.
+
+![Viewing details of a work queue including agent configuration string](/img/ui/work-queue-details.png)
+
+Click **Edit** to make changes to the work queue configuration, including deleting the work queue.
diff --git a/mkdocs.yml b/mkdocs.yml
@@ -90,11 +90,13 @@ nav:
         - Storage: concepts/storage.md
         - Settings: concepts/settings.md
         - Logging: concepts/logs.md
-    - 'Orion UI':
+    - Orion UI & Cloud:
         - Overview: ui/overview.md
         - ui/dashboard.md
         - ui/flows-and-tasks.md
         - ui/filters.md
+        - ui/work-queues.md
+        - ui/cloud.md
     - Collections:
         - Overview: collections/overview.md
         - Collections Catalog: collections/catalog.md

diff --git a/requirements.txt b/requirements.txt
@@ -25,6 +25,7 @@ python-slugify >= 5.0
 pytz >= 2021.1
 pyyaml >= 5.1b1
 ray[default] >= 1.9; python_version < '3.10' and platform_machine == "x86_64"
+readchar >= 3.0.5
 rich >= 10.0
 sqlalchemy[asyncio] >= 1.4.20
 toml >= 0.10.0

diff --git a/src/prefect/cli/__init__.py b/src/prefect/cli/__init__.py
@@ -10,4 +10,5 @@
 import prefect.cli.dev
 import prefect.cli.profile
 import prefect.cli.storage
+import prefect.cli.cloud
 import prefect.cli.work_queue