Blocks

Prefector deploys Prefect blocks by reading field values from the environment (or a secrets manager) and saving the resulting block to Prefect via the API.

Commands

# List block specs found in the directory
prefector blocks list --blocks-dir path/to/specs

# Deploy all blocks
prefector blocks deploy \
  --blocks-dir path/to/specs \
  --api-url "$PREFECT_API_URL"

# Deploy specific blocks only
prefector blocks deploy \
  --blocks-dir path/to/specs \
  --api-url "$PREFECT_API_URL" \
  --target trino-credentials \
  --target s3-bucket

Writing block spec modules

Each .py file in --blocks-dir is a block spec module. A module must expose a BLOCKS list of BlockSpec objects. Files whose names start with _ and __init__.py are ignored.

Each BlockSpec pairs a Pydantic BaseSettings subclass — which reads field values from the environment — with a Prefect Block subclass.

# blocks/trino.py
from pydantic_settings import BaseSettings, SettingsConfigDict
from prefect_sqlalchemy import DatabaseCredentials, SyncDriver
from prefector.blocks.base import BlockSpec

class TrinoSettings(BaseSettings):
    model_config = SettingsConfigDict(env_prefix="TRINO_")
    user: str
    password: str
    host: str
    port: int = 8080

BLOCKS = [
    BlockSpec(
        name="trino-credentials",
        settings_cls=TrinoSettings,
        block_cls=DatabaseCredentials,
    )
]

When prefector blocks deploy runs, it instantiates TrinoSettings() — which reads TRINO_USER, TRINO_PASSWORD, TRINO_HOST from the environment — and passes the values to DatabaseCredentials, then saves the block to Prefect.

Nested block dependencies

A block can depend on another Prefect block by name. Set the field type to BlockSpec in the settings class and Prefector will load the named block from Prefect at deploy time.

from prefector.blocks.base import BlockSpec

class BucketSettings(BaseSettings):
    credentials: BlockSpec = BlockSpec(
        name="aws-credentials",
        settings_cls=...,
        block_cls=AwsCredentials,
    )
    bucket_name: str

Block sources

A block-sources.yaml file lets you control where block field values come from without modifying the spec modules. This is useful when:

• The same spec modules are shared across environments (dev/staging/prod), each with different secret locations or naming conventions
• A data pipeline is used by multiple teams that use different secret management solutions. Any team can reuse the same block spec.
• Secrets live in a secrets manager rather than environment variables

The file is optional. Blocks without a sources entry continue to use their settings_cls as-is.

Loading block sources

Place block-sources.yaml inside --blocks-dir and it is picked up automatically:

blocks/
  trino.py
  s3.py
  block-sources.yaml   ← picked up automatically

Or provide an explicit path:

prefector blocks deploy \
  --blocks-dir path/to/specs \
  --sources path/to/block-sources.yaml \
  --api-url "$PREFECT_API_URL"

File format

Three equivalent YAML shapes are accepted:

Flat mapping (simplest):

trino-credentials:
  source: env
  env_var_prefix: TRINO_

List (useful when ordering matters or you prefer the list style):

- trino-credentials:
    source: env
    env_var_prefix: TRINO_
- s3-bucket:
    source: keeper
    record_title: s3-credentials

blocks: wrapper:

blocks:
  - trino-credentials:
      source: env
      env_var_prefix: TRINO_

Environment variable source

Reads block field values from environment variables.

trino-credentials:
  source: env
  env_var_prefix: TRINO_       # env vars are read as <prefix><field>
  fields:                       # optional: override individual field names
    user: USERNAME              # reads TRINO_USERNAME into field `user`
    password: PASSWORD          # reads TRINO_PASSWORD into field `password`
    # unlisted fields use the field name as-is: `host` -> TRINO_HOST

The fields mapping is optional. Without it each block field maps to <env_var_prefix><field_name> (case-insensitive). Only add fields entries when the env var suffix differs from the field name.

If a required variable is missing the command exits with an error naming the variable that needs to be set.

Keeper Secrets Manager source

Reads block field values from a record in Keeper Secrets Manager.

trino-credentials:
  source: keeper
  record_title: trino-credentials   # required: base record title
  record_prefix: dlh                # optional: prepended before title
  record_suffix: ${ENVIRONMENT}     # optional: appended after title
  separator: ":"                    # optional: joins the parts (default: ":"); must be quoted in YAML
  ksm_token: ${KSM_TOKEN}           # optional: one-time token; falls back to KSM_CONFIG env var
  fields:                           # optional: map block field -> KSM field label
    user: login                     # reads KSM field "login" into block field `user`
    # unlisted fields use the field name as-is

The full record title is assembled from the non-empty parts joined by separator:

record_prefix	record_title	record_suffix	Result
dlh	trino-credentials	prod	dlh:trino-credentials:prod
(empty)	trino-credentials	prod	trino-credentials:prod
(empty)	trino-credentials	(empty)	trino-credentials

Requires prefector[keeper]:

pip install "prefector[keeper]"

Environment variable substitution

Any string value in block-sources.yaml may use ${VAR_NAME} syntax. Substitution happens when prefector blocks deploy runs, before the source is used.

trino-credentials:
  source: keeper
  record_title: trino-credentials
  record_suffix: ${ENVIRONMENT}    # resolves to "prod", "staging", etc.
  ksm_token: ${KSM_TOKEN}

All referenced variables must be set at deploy time or the command will exit with an error naming the missing variable.