Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve language and formatting in Profiles and Configuration guide #11996

Merged
merged 3 commits into from
Feb 16, 2024
Merged

Improve language and formatting in Profiles and Configuration guide #11996

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
cb7a6ce
Improve language and formatting in Profiles and Configuration guide
discdiver Feb 15, 2024
f06f1f8
fix typos
discdiver Feb 15, 2024
ac9752b
Update settings.md
discdiver Feb 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
141 changes: 108 additions & 33 deletions docs/guides/settings.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,51 +12,75 @@ search:

# Profiles & Configuration

Prefect's settings are [documented][prefect.settings.Settings] and type-validated. By modifying the default settings, you can customize various aspects of the system.
Prefect's local settings are [documented][prefect.settings.Settings] and type-validated.

Settings can be viewed from the CLI or the UI.
By modifying the default settings, you can customize various aspects of the system.
You can override a setting with an environment variable or by updating the setting in a Prefect [profile](#configuration-profiles).

You can override the setting for a profile with environment variables.
Prefect profiles are persisted groups of settings on your local machine. A single profile is always active.

Prefect [profiles](#configuration-profiles) are groups of settings that you can persist on your machine. When you change profiles, all of the settings configured in the profile are applied. You can apply profiles to individual commands or set a profile for your environment.
Initially, a default profile named `default` is active and contains no settings overrides.

## Commonly configured settings
All currently active settings can be viewed from the command line by running the following command:

This section describes some commonly configured settings for Prefect installations. See [Configuring settings](#configuring-settings) for details on setting and unsetting configuration values.
<div class="terminal">
```bash
prefect config view --show-defaults
```
</div>

### PREFECT_API_URL
When you switch to a different profile, all of the settings configured in the newly activated profile are applied.

The `PREFECT_API_URL` value specifies the API endpoint of your Prefect Cloud workspace or a Prefect server instance.
## Commonly configured settings

For example, using a local Prefect server instance.
This section describes some commonly configured settings.
See [Configuring settings](#configuring-settings) for details on setting and unsetting configuration values.

### PREFECT_API_KEY

The `PREFECT_API_KEY` value specifies the [API key](/ui/cloud-api-keys/#create-an-api-key) used to authenticate with Prefect Cloud.

```bash
PREFECT_API_URL="http://127.0.0.1:4200/api"
PREFECT_API_KEY="[API-KEY]"
```

Using Prefect Cloud:
Generally, you will set the `PREFECT_API_URL` and `PREFECT_API_KEY` for your active profile by running `prefect cloud login`.
If you're curious, read more about [managing API keys](/cloud/users/api-keys/).

### PREFECT_API_URL

The `PREFECT_API_URL` value specifies the API endpoint of your Prefect Cloud workspace or a self-hosted Prefect server instance.

For example, if using Prefect Cloud:

<div class="terminal">
```bash
PREFECT_API_URL="https://api.prefect.cloud/api/accounts/[ACCOUNT-ID]/workspaces/[WORKSPACE-ID]"
```
</div>

You can view your Account ID and Workspace ID in your browser URL when at a Prefect Cloud workspace page.
For example: <https://app.prefect.cloud/account/abc-my-account-id-is-here/workspaces/123-my-workspace-id-is-here>.

View your Account ID and Workspace ID in your browser URL when logged into Prefect Cloud. For example: <https://app.prefect.cloud/account/abc-my-account-id-is-here/workspaces/123-my-workspace-id-is-here>.
If using a local Prefect server instance, set your API URL like this:

<div class="terminal">
```bash
PREFECT_API_URL="http://127.0.0.1:4200/api"
```
</div>

!!! tip "`PREFECT_API_URL` setting for workers"
When using [workers and work pools](/concepts/work-pools/) (agent and block-based deployments are legacy) that can create flow runs for deployments in remote environments, [`PREFECT_API_URL`](/concepts/settings/) must be set for the environment in which your worker is running.

If using a [worker](/concepts/work-pools/) (agent and block-based deployments are legacy) that can create flow runs for deployments in remote environments, [`PREFECT_API_URL`](/concepts/settings/) must be set for the environment in which your worker is running.

If you want the worker to communicate with Prefect Cloud or a Prefect server instance from a remote execution environment such as a VM or Docker container, you must configure `PREFECT_API_URL` in that environment.

!!! tip "Running the Prefect UI behind a reverse proxy"
When using a reverse proxy (such as [Nginx](https://nginx.org) or [Traefik](https://traefik.io)) to proxy traffic to a locally-hosted Prefect UI instance, the Prefect server also needs to be configured to know how to connect to the API. The [`PREFECT_UI_API_URL`](../../api-ref/prefect/settings/#PREFECT_UI_API_URL) should be set to the external proxy URL (e.g. if your external URL is <https://prefect-server.example.com/> then set `PREFECT_UI_API_URL=https://prefect-server.example.com/api` for the Prefect server process). You can also accomplish this by setting [`PREFECT_API_URL`](/concepts/settings/#prefect.settings.PREFECT_API_URL) to the API URL, as this setting is used as a fallback if `PREFECT_UI_API_URL` is not set.

### PREFECT_API_KEY

The `PREFECT_API_KEY` value specifies the [API key](/ui/cloud-api-keys/#create-an-api-key) used to authenticate with your Prefect Cloud workspace.

```bash
PREFECT_API_KEY="[API-KEY]"
```
When using a reverse proxy (such as [Nginx](https://nginx.org) or [Traefik](https://traefik.io)) to proxy traffic to a locally-hosted Prefect UI instance, the Prefect server instance also needs to be configured to know how to connect to the API.
The [`PREFECT_UI_API_URL`](../../api-ref/prefect/settings/#PREFECT_UI_API_URL) should be set to the external proxy URL (e.g. if your external URL is <https://prefect-server.example.com/> then set `PREFECT_UI_API_URL=https://prefect-server.example.com/api` for the Prefect server process).
You can also accomplish this by setting [`PREFECT_API_URL`](/concepts/settings/#prefect.settings.PREFECT_API_URL) to the API URL, as this setting is used as a fallback if `PREFECT_UI_API_URL` is not set.

### PREFECT_HOME

Expand All @@ -76,7 +100,7 @@ PREFECT_LOCAL_STORAGE_PATH='${PREFECT_HOME}/storage'

### Database settings

Prefect provides several self-hosting database configuration settings you can read about [here](/host/).
If running a self-hosted Prefect server instance, there are several database configuration settings you can read about [here](/host/).

### Logging settings

Expand Down Expand Up @@ -149,23 +173,27 @@ All settings have keys that match the environment variable that can be used to o

For example, configuring the home directory:

```shell
<div class="terminal">
```bash
# environment variable
export PREFECT_HOME="/path/to/home"
```
</div>

```python
# python
import prefect.settings
prefect.settings.PREFECT_HOME.value() # PosixPath('/path/to/home')
```

Configuring the server's port:
Configuring the a server instance's port:

```shell
<div class="terminal">
```bash
# environment variable
export PREFECT_SERVER_API_PORT=4242
```
</div>

```python
# python
Expand All @@ -190,157 +218,195 @@ The `prefect profile` CLI commands enable you to create, review, and manage prof

If you configured settings for a profile, `prefect profile inspect` displays those settings:

<div class="terminal">
```bash
$ prefect profile inspect
PREFECT_PROFILE = "default"
PREFECT_API_KEY = "pnu_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
PREFECT_API_URL = "http://127.0.0.1:4200/api"
```
</div>

You can pass the name of a profile to view its settings:

<div class="terminal">
```bash
$ prefect profile create test
$ prefect profile inspect test
PREFECT_PROFILE="test"
```
</div>

### Creating and removing profiles

Create a new profile with no settings:

<div class="terminal">
```bash
$ prefect profile create test
Created profile 'test' at /Users/terry/.prefect/profiles.toml.
```
</div>

Create a new profile `foo` with settings cloned from an existing `default` profile:

<div class="terminal">
```bash
$ prefect profile create foo --from default
Created profile 'cloud' matching 'default' at /Users/terry/.prefect/profiles.toml.
```
</div>

Rename a profile:

<div class="terminal">
```bash
$ prefect profile rename temp test
Renamed profile 'temp' to 'test'.
```
</div>

Remove a profile:

<div class="terminal">
```bash
$ prefect profile delete test
Removed profile 'test'.
```
</div>

Removing the default profile resets it:

<div class="terminal">
```bash
$ prefect profile delete default
Reset profile 'default'.
```
</div>

### Change values in profiles

Set a value in the current profile:

<div class="terminal">
```bash
$ prefect config set VAR=X
Set variable 'VAR' to 'X'
Updated profile 'default'
```
</div>

Set multiple values in the current profile:

<div class="terminal">
```bash
$ prefect config set VAR2=Y VAR3=Z
Set variable 'VAR2' to 'Y'
Set variable 'VAR3' to 'Z'
Updated profile 'default'
```
</div>

You can set a value in another profile by passing the `--profile NAME` option to a CLI command:

<div class="terminal">
```bash
$ prefect --profile "foo" config set VAR=Y
Set variable 'VAR' to 'Y'
Updated profile 'foo'
```
</div>

Unset values in the current profile to restore the defaults:

<div class="terminal">
```bash
$ prefect config unset VAR2 VAR3
Unset variable 'VAR2'
Unset variable 'VAR3'
Updated profile 'default'
```
</div>

### Inspecting profiles

See a list of available profiles:

<div class="terminal">
```bash
$ prefect profile ls
* default
cloud
test
local
```
</div>

View all settings for a profile:

<div class="terminal">
```bash
$ prefect profile inspect cloud
PREFECT_API_URL='https://api.prefect.cloud/api/accounts/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx
x/workspaces/xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
PREFECT_API_KEY='xxx_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
```
</div>

### Using profiles

The profile `default` is used by default. There are several methods to switch to another profile.
The profile named `default` is used by default. There are several methods to switch to another profile.

The recommended method is to use the `prefect profile use` command with the name of the profile:

<div class="terminal">
```bash
$ prefect profile use foo
Profile 'test' now active.
```
</div>

Alternatively, you may set the environment variable `PREFECT_PROFILE` to the name of the profile:
Alternatively, you can set the environment variable `PREFECT_PROFILE` to the name of the profile:

<div class="terminal">
```bash
export PREFECT_PROFILE=foo
```
</div>

Or, specify the profile in the CLI command for one-time usage:

<div class="terminal">
```bash
prefect --profile "foo" ...
```
</div>

Note that this option must come before the subcommand. For example, to list flow runs using the profile `foo`:

<div class="terminal">
```bash
prefect --profile "foo" flow-run ls
```
</div>

You may use the `-p` flag as well:

<div class="terminal">
```bash
prefect -p "foo" flow-run ls
```
</div>

You may also create an 'alias' to automatically use your profile:

<div class="terminal">
```bash
$ alias prefect-foo="prefect --profile 'foo' "
# uses our profile!
$ prefect-foo config view
```
</div>

## Conflicts with environment variables

Expand All @@ -350,39 +416,48 @@ In all other cases, environment variables will take precedence over the value in

For example, a value set in a profile will be used by default:

<div class="terminal">
```bash
$ prefect config set PREFECT_LOGGING_LEVEL="ERROR"
$ prefect config view --show-sources
PREFECT_PROFILE="default"
PREFECT_LOGGING_LEVEL='ERROR' (from profile)
```
</div>

But, setting an environment variable will override the profile setting:

<div class="terminal">
```bash
$ export PREFECT_LOGGING_LEVEL="DEBUG"
$ prefect config view --show-sources
PREFECT_PROFILE="default"
PREFECT_LOGGING_LEVEL='DEBUG' (from env)
```
</div>

Unless the profile is explicitly requested when using the CLI:

<div class="terminal">
```bash
$ prefect --profile default config view --show-sources
PREFECT_PROFILE="default"
PREFECT_LOGGING_LEVEL='ERROR' (from profile)
```
</div>

## Profile files

Profiles are persisted to the `PREFECT_PROFILES_PATH`, which can be changed with an environment variable.
Profiles are persisted to the file location specified by `PREFECT_PROFILES_PATH`.
The default location is a `profiles.toml` file in the `PREFECT_HOME` directory:

By default, it is stored in your `PREFECT_HOME` directory:

```
$ prefect config view --show-defaults | grep PROFILES_PATH
PREFECT_PROFILES_PATH='~/.prefect/profiles.toml'
<div class="terminal">
```bash
$ prefect config view --show-defaults
...
PREFECT_PROFILES_PATH='${PREFECT_HOME}/profiles.toml'
...
```
</div>

The [TOML](https://toml.io/en/) format is used to store profile data.