Unexpected exception: Docker image name is not correctly formatted

[seanpden]

Dagster version

1.7.2

What's the issue?

I am attempting to deploy Dagster via a docker compose file. I've gotten the images all connected, the daemon and web server are both seeing the user code image, but when I attempt to materialize an asset, I get the following "ENGINE_EVENT" error:

Exception: Docker image name "dagster-user-code" is not correctly formatted

  File "/usr/local/lib/python3.10/site-packages/dagster/_daemon/run_coordinator/queued_run_coordinator_daemon.py", line 379, in _dequeue_run
    instance.run_launcher.launch_run(LaunchRunContext(dagster_run=run, workspace=workspace))
  File "/usr/local/lib/python3.10/site-packages/dagster_docker/docker_run_launcher.py", line 148, in launch_run
    docker_image = self._get_docker_image(job_code_origin)
  File "/usr/local/lib/python3.10/site-packages/dagster_docker/docker_run_launcher.py", line 96, in _get_docker_image
    validate_docker_image(docker_image)
  File "/usr/local/lib/python3.10/site-packages/dagster_docker/utils.py", line 58, in validate_docker_image
    raise Exception(f"Docker image name {docker_image} is not correctly formatted") from e

The above exception was caused by the following exception:
docker_image.reference.ReferenceInvalidFormat: invalid reference format

  File "/usr/local/lib/python3.10/site-packages/dagster_docker/utils.py", line 56, in validate_docker_image
    reference.Reference.parse(docker_image)
  File "/usr/local/lib/python3.10/site-packages/docker_image/reference.py", line 137, in parse
    raise ReferenceInvalidFormat.default()

The image name should be correct, below I have listed the output of docker ps

CONTAINER ID   IMAGE                COMMAND                  CREATED         STATUS         PORTS                    NAMES
722e0cfec43b   dagster-daemon       "dagster-daemon run"     8 minutes ago   Up 8 minutes                            dagster_daemon
86d1e78f9e95   dagster-web-server   "dagster-webserver -…"   8 minutes ago   Up 8 minutes   0.0.0.0:3000->3000/tcp   dagster_web_server
8ad997e703b9   postgres:13          "docker-entrypoint.s…"   8 minutes ago   Up 8 minutes   5432/tcp                 dagster_metadata_db
e8c85a861e7c   dagster-user-code    "dagster api grpc --…"   8 minutes ago   Up 8 minutes   0.0.0.0:4000->4000/tcp   dagster_user_code

I'm also adding my compose file, dagster.yaml file, and workspace.yaml file for reference.

An interesting note, I have propped up a working instance at work with an almost identical setup - the only obvious difference is the service names in my compose file. Most of this was ripped or tailored directly from the github repo listed in the docker deployment page.

compose.yaml

services:
  dagster_metadata_db:
    image: postgres:13
    container_name: dagster_metadata_db
    environment:
      - POSTGRES_USER=${POSTGRES_USER}
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
      - POSTGRES_DB=${POSTGRES_DB}
    networks:
        - dagster_network

  dagster_user_code:
    build:
      context: .
      dockerfile: ./dockerfile_user_code
    container_name: dagster_user_code
    image: dagster-user-code
    environment:
      - DAGSTER_POSTGRES_USER=${POSTGRES_USER}
      - DAGSTER_POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
      - DAGSTER_POSTGRES_DB=${POSTGRES_DB}
      - DAGSTER_CURRENT_IMAGE="dagster-user-code"
    ports:
      - 4000:4000
    networks:
        - dagster_network

  dagster_web_server:
    build:
      context: .
      dockerfile: dockerfile_web_and_daemon
    entrypoint:
      - dagster-webserver
      - -h
      - "0.0.0.0"
      - -p
      - "3000"
      - -w
      - "workspace.yaml"
    container_name: dagster_web_server
    image: dagster-web-server
    expose:
      - 3000
    ports:
      - 3000:3000
    depends_on:
      - dagster_user_code
      - dagster_metadata_db
    environment:
      - DAGSTER_POSTGRES_USER=${POSTGRES_USER}
      - DAGSTER_POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
      - DAGSTER_POSTGRES_DB=${POSTGRES_DB}
      - DAGSTER_CURRENT_IMAGE="dagster-web-server"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /tmp/io_manager_storage:/tmp/io_manager_storage
    networks:
        - dagster_network

  dagster_daemon:
    build:
      context: .
      dockerfile: dockerfile_web_and_daemon
    entrypoint:
      - dagster-daemon
      - run
    container_name: dagster_daemon
    image: dagster-daemon
    depends_on:
      - dagster_user_code
      - dagster_metadata_db
    networks:
        - dagster_network
    environment:
      - DAGSTER_POSTGRES_USER=${POSTGRES_USER}
      - DAGSTER_POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
      - DAGSTER_POSTGRES_DB=${POSTGRES_DB}
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /tmp/io_manager_storage:/tmp/io_manager_storage

networks:
  dagster_network:
    driver: bridge
    name: dagster_network

dagster.yaml

scheduler:
  module: dagster.core.scheduler
  class: DagsterDaemonScheduler

run_coordinator:
  module: dagster.core.run_coordinator
  class: QueuedRunCoordinator

run_launcher:
  module: dagster_docker
  class: DockerRunLauncher
  config:
    env_vars:
      - DAGSTER_POSTGRES_USER
      - DAGSTER_POSTGRES_PASSWORD
      - DAGSTER_POSTGRES_DB
    network: dagster_network
    container_kwargs:
      volumes: # Make docker client accessible to any launched containers as well
        - /var/run/docker.sock:/var/run/docker.sock
        - /tmp/io_manager_storage:/tmp/io_manager_storage

run_storage:
  module: dagster_postgres.run_storage
  class: PostgresRunStorage
  config:
    postgres_db:
      hostname: dagster_metadata_db
      username:
        env: DAGSTER_POSTGRES_USER
      password:
        env: DAGSTER_POSTGRES_PASSWORD
      db_name:
        env: DAGSTER_POSTGRES_DB
      port: 5432

schedule_storage:
  module: dagster_postgres.schedule_storage
  class: PostgresScheduleStorage
  config:
    postgres_db:
      hostname: dagster_metadata_db
      username:
        env: DAGSTER_POSTGRES_USER
      password:
        env: DAGSTER_POSTGRES_PASSWORD
      db_name:
        env: DAGSTER_POSTGRES_DB
      port: 5432

event_log_storage:
  module: dagster_postgres.event_log
  class: PostgresEventLogStorage
  config:
    postgres_db:
      hostname: dagster_metadata_db
      username:
        env: DAGSTER_POSTGRES_USER
      password:
        env: DAGSTER_POSTGRES_PASSWORD
      db_name:
        env: DAGSTER_POSTGRES_DB
      port: 5432

workspace.yaml

load_from:
  - grpc_server:
      host: dagster_user_code
      port: 4000
      location_name: "dagster-user-code"

What did you expect to happen?

I expected the asset to materialize. The image name it is referencing seems to be correct and exists in according to the docker ps command.

How to reproduce?

1. Copy my previously mentioned yaml files,
2. Create the following 'repo.py' file in the root of the project (with workspace, compose, etc):

repo.py

from dagster import asset


@asset
def my_asset():
    return

4. Create the following dockerfiles in the root of the project:

dockerfile_user_code

# syntax=docker/dockerfile:1

FROM python:3.10-slim

RUN pip install \ 
  dagster \ 
  dagster-docker \ 
  dagster-postgres \ 
  dagster-webserver

WORKDIR /opt/dagster/app

COPY ./repo.py /opt/dagster/app/

EXPOSE 4000
CMD ["dagster", "api", "grpc", "--python-file", "repo.py", "--host", "0.0.0.0", "--port", "4000"]

dockerfile_web_and_daemon

FROM python:3.10-slim

RUN pip install \
  dagster \
  dagster-webserver \
  dagster-postgres \ 
  dagster-docker

ENV DAGSTER_HOME=/opt/dagster/dagster_home/

RUN mkdir -p $DAGSTER_HOME
COPY dagster.yaml workspace.yaml $DAGSTER_HOME

WORKDIR $DAGSTER_HOME

5. Run docker compose up -d --build
6. Attempt to materialize an asset
7. See error (hopefully)

Deployment type

Docker Compose

Deployment details

docker --version:

Docker version 24.0.6, build ed223bc

OS: Ubuntu in WSL2

Additional information

Like I mentioned earlier, I had a build working that is very similar to this. Unsure of the difference other than a few image names.

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

[garethbrickman]

In your dagster.yaml file, could you try adding an image key under config to specify which Docker image to use when launching runs. This image should be the one that contains your Dagster user code.

Example:

run_launcher:
  module: dagster_docker
  class: DockerRunLauncher
  config:
    image: "your_image_name:your_tag"  # Replace with your actual image name and tag if it haso ne
    env_vars:
      - DAGSTER_POSTGRES_USER
      - DAGSTER_POSTGRES_PASSWORD
      - DAGSTER_POSTGRES_DB
    network: dagster_network
    container_kwargs:
      volumes:
        - /var/run/docker.sock:/var/run/docker.sock
        - /tmp/io_manager_storage:/tmp/io_manager_storage

[seanpden]

Added the image: dagster-user-code key and value to the dagster.yaml file and the output remains the same.

[seanpden]

Wanted to update: I fixed the issue I was having by removing any quotation marks around the "DAGSTER_CURRENT_IMAGE" environment variable in the compose.yaml file. The "dagster_user_code" service now looks like this:

  dagster_user_code:
    build:
      context: .
      dockerfile: ./dockerfile_user_code
    container_name: dagster_user_code
    image: dagster_user_code
    environment:
      - DAGSTER_POSTGRES_USER=${POSTGRES_USER}
      - DAGSTER_POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
      - DAGSTER_POSTGRES_DB=${POSTGRES_DB}
      - DAGSTER_CURRENT_IMAGE=dagster_user_code # THIS WAS THE PROBLEM - ORIGINAL VALUE: '"dagster-user-code"'
    ports:
      - 4000:4000
    networks:
        - dagster_network

While this fixed my immediate problem, I believe this issue should remain open. At work, I am using quotations - like this:

 dagster_user_code:
    build:
      context: .
      dockerfile: ./Dockerfile_user_code
    container_name: docker_user_code
    image: docker_user_code_image
    ports:
      - 4000:4000
    networks:
      - docker_example_network
    environment:
      DAGSTER_POSTGRES_USER: "postgres_user"
      DAGSTER_POSTGRES_PASSWORD: "postgres_password"
      DAGSTER_POSTGRES_DB: "postgres_db"
      DAGSTER_CURRENT_IMAGE: "docker_user_code_image" # <<< RIGHT HERE

That code snippet works in my work environment. I'm not sure which is supposed to be correct, but I can't imagine it is working as intended.