How do I set per-code-location deployment configuration in Docker/K8s/ECS?

[gibsondan]

If I am running Dagster in production in Docker, K8s, or ECS, how do I set different environment variables / secrets / volumes / task definitions / other deployment-related configuration for different code locations?

[gibsondan]

In Kubernetes:
Individual code locations can be configured in the Helm chart in the "dagster-user-deployments" part of the Helm chart, and that configuration will be automatically passed through to any launched runs as well - see https://docs.dagster.io/deployment/guides/kubernetes/deploying-with-helm#step-61-configure-the-deployment for an example.

(The Helm chart uses the same DAGSTER_CONTAINER_CONTEXT environment variable under the hood)

In ECS:
Similar to the Docker example above, you can set the DAGSTER_CONTAINER_CONTEXT environment variable with an "ecs" key to configure ECS-specific configuration for launched runs from the gRPC server's code location. For example, to say that all runs should be launched with a particular task definition ARN, using a specific container name for the main container for the run:

ENV DAGSTER_CONTAINER_CONTEXT='{"ecs": {"task_definition_arn": "<your arn here>", "container_name": "my_cool_container"} }'

In Docker:
You can set the DAGSTER_CONTAINER_CONTEXT environment variable in your gRPC server container to include additional configuration that will be applied to each run launched from that code location.

For example:

ENV DAGSTER_CONTAINER_CONTEXT='{"docker": {"env_vars": ["FOO_ENV_VAR=bar_value", "OTHER_ENV_VAR=other_value"], "container_kwargs": {"auto_remove": true}}}'

[moleary-gsa]

Sorry to reopen this discussion but I'm having trouble making my environment variables accessible to runs kicked off with the docker launcher using the DAGSTER_CONTAINER_CONTEXT method for getting env vars available to launched jobs

I get the following error thrown from the daemon:

dagster_daemon         | 2023-09-04 11:37:56 +0000 - dagster.daemon.QueuedRunCoordinatorDaemon - ERROR - Caught an unrecoverable error while dequeuing the run. Marking the run as failed and dropping it from the queue: Exception: Tried to load environment variable DBT_PROJECT_DIR, but it was not set
dagster_daemon         | 
dagster_daemon         | Stack Trace:
dagster_daemon         |   File "/usr/local/lib/python3.10/site-packages/dagster/_daemon/run_coordinator/queued_run_coordinator_daemon.py", line 333, in _dequeue_run
dagster_daemon         |     instance.run_launcher.launch_run(LaunchRunContext(dagster_run=run, workspace=workspace))
dagster_daemon         |   File "/usr/local/lib/python3.10/site-packages/dagster_docker/docker_run_launcher.py", line 161, in launch_run
dagster_daemon         |     self._launch_container_with_command(run, docker_image, command)
dagster_daemon         |   File "/usr/local/lib/python3.10/site-packages/dagster_docker/docker_run_launcher.py", line 101, in _launch_container_with_command
dagster_daemon         |     docker_env = dict([parse_env_var(env_var) for env_var in container_context.env_vars])
dagster_daemon         |   File "/usr/local/lib/python3.10/site-packages/dagster_docker/docker_run_launcher.py", line 101, in <listcomp>
dagster_daemon         |     docker_env = dict([parse_env_var(env_var) for env_var in container_context.env_vars])
dagster_daemon         |   File "/usr/local/lib/python3.10/site-packages/dagster/_core/utils.py", line 123, in parse_env_var
dagster_daemon         |     raise Exception(f"Tried to load environment variable {env_var_str}, but it was not set")

I'm confused by this for 2 reasons -

1, I have the DAGSTER_CONTAINER_CONTEXT set on the user code image as an environment variable and it contains all the environment variables needed for running the code:

  forecasting_user_code:
    image: __IMAGE__
    container_name: forecasting_user_code
    restart: always
    environment:
      - DAGSTER_CONTAINER_CONTEXT=$FC_DAGSTER_CONTAINER_CONTEXT
      - DAGSTER_CURRENT_IMAGE=__IMAGE__
    env_file: .env_fc
    networks:
      - dagster_network
    entrypoint:
      - dagster
      - api
      - grpc
      - --lazy-load-user-code
      - -h
      - "0.0.0.0"
      - -p
      - "4001"
      - -f
      - defs.py
    healthcheck:
      test: dagster api grpc-health-check --host forecasting_user_code --port 4001 || exit 0
      interval: 10s
      start_period: 10s # Estimated time to boot.

The content of the FC_DAGSTER_CONTAINER_CONTEXT is as described above:

FC_DAGSTER_CONTAINER_CONTEXT={"docker": {"env_vars": ["ENV=DEV", "PG_HOST=__HIDDEN__", "PG_DATABASE=__HIDDEN__", "PG_PORT=__HIDDEN__", "PG_USER=__HIDDEN__", "PG_PASSWORD=__HIDDEN__", "PG_SCHEMA=__HIDDEN__", "PG_INTERMEDIATE_SCHEMA=__HIDDEN__", "FC_STORAGE_BUCKET=__HIDDEN__", "FC_LOCAL_BASE_PATH=__HIDDEN__", "SLACK_BOT_TOKEN=__HIDDEN__", "SLACK_ALERT_CHANNEL=__HIDDEN__",  "DAGSTER_HOST=__HIDDEN__", "DBT_TARGET_PATH=__HIDDEN__", "DBT_PROJECT_DIR=__HIDDEN__", "DBT_PROFILES_DIR=__HIDDEN__"], "container_kwargs": {"auto_remove": true}}}

2.My understanding was that the environment variables needed to be set in the user code container on launch but this seems to show that the environment variables are set on the daemon??

I feel like I'm missing something here right?

[gibsondan]

@moleary-gsa apologies, I just saw this - is there possibly somewhere else in your project where you are configuring DBT_PROJECT_DIR in a list of env vars without an = sign? That stack trace that you hit only fires when there is no "=" in the environment variable string: https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/utils.py#L117-L123

But I see the example that you shared says "DBT_PROJECT_DIR=HIDDEN", which has an = sign.