Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Comfyui keeps restarting on prompt #58

Closed
TechanisYT opened this issue Mar 10, 2024 · 1 comment
Closed

Comfyui keeps restarting on prompt #58

TechanisYT opened this issue Mar 10, 2024 · 1 comment

Comments

@TechanisYT
Copy link

I just freshly installed ComfyUI with only a CPU on a Ubuntu-Server VM within Proxmox. I had to fiddle around to get it to start with only the cpu but i can't generate any images. If I start the Promt, the server uses all its ram and cpu cores and does not resbond for about 15 minutes.
the docker-compose file i use:

version: "3.8"
# Compose file build variables set in .env
services:
  supervisor:
    restart: always
    platform: linux/amd64
    build:
      context: ./build
      args:
        IMAGE_BASE: ${IMAGE_BASE:-ghcr.io/ai-dock/jupyter-pytorch:2.2.0-py3.10-cpu-22.04}
      #tags:
        #- "ghcr.io/ai-dock/comfyui:${IMAGE_TAG:-jupyter-pytorch-2.2.0-py3.10-cpu-22.04}"
        #- "IMAGE_TAG:-jupyter-pytorch-2.2.0-py3.10-cpu-22.04"
    image: ghcr.io/ai-dock/comfyui:${IMAGE_TAG:-jupyter-pytorch-2.2.0-py3.10-cpu-22.04}
    ## For Nvidia GPU's - You probably want to uncomment this
    #deploy:
    #  resources:
    #    reservations:
    #      devices:
    #        - driver: nvidia
    #          count: all
    #          capabilities: [gpu]
    devices:
      - "/dev/dri:/dev/dri"
      ## For AMD GPU
      #- "/dev/kfd:/dev/kfd"
    volumes:
      ## Workspace
      - ./workspace:${WORKSPACE:-/workspace/}:rshared
      # You can share /workspace/storage with other non-ComfyUI containers. See README
      #- /path/to/common_storage:${WORKSPACE:-/workspace/}storage/:rshared
      # Will echo to root-owned authorized_keys file;
      # Avoids changing local file owner
      - ./config/authorized_keys:/root/.ssh/authorized_keys_mount
      - ./config/provisioning/default.sh:/opt/ai-dock/bin/provisioning.sh
    ports:
        # SSH available on host machine port 2222 to avoid conflict. Change to suit
        - ${SSH_PORT_HOST:-2222}:22
        # Caddy port for service portal
        - ${SERVICEPORTAL_PORT_HOST:-1111}:${SERVICEPORTAL_PORT_HOST:-1111}
        # ComfyUI web interface
        - ${COMFYUI_PORT_HOST:-8188}:${COMFYUI_PORT_HOST:-8188}
        # Jupyter server
        - ${JUPYTER_PORT_HOST:-8888}:${JUPYTER_PORT_HOST:-8888}
        # Rclone webserver for interactive configuration
        - ${RCLONE_PORT_HOST:-53682}:${RCLONE_PORT_HOST:-53682}
    environment:
        # Don't enclose values in quotes
        - DIRECT_ADDRESS=${DIRECT_ADDRESS:-192.168.1.101}
        - DIRECT_ADDRESS_GET_WAN=${DIRECT_ADDRESS_GET_WAN:-false}
        - WORKSPACE=${WORKSPACE:-/workspace}
        - WORKSPACE_SYNC=${WORKSPACE_SYNC:-false}
        #- CF_TUNNEL_TOKEN=${CF_TUNNEL_TOKEN:-}
        - CF_QUICK_TUNNELS=${CF_QUICK_TUNNELS:-false}
        - WEB_ENABLE_AUTH=${WEB_ENABLE_AUTH:-true}
        - WEB_USER=${WEB_USER:-user}
        - WEB_PASSWORD=${WEB_PASSWORD:-password}
        - SSH_PORT_HOST=${SSH_PORT_HOST:-2222}
        - SERVICEPORTAL_PORT_HOST=${SERVICEPORTAL_PORT_HOST:-1111}
        - SERVICEPORTAL_METRICS_PORT=${SERVICEPORTAL_METRICS_PORT:-21111}
        - COMFYUI_FLAGS=${COMFYUI_FLAGS:-}
        - COMFYUI_PORT_HOST=${COMFYUI_PORT_HOST:-8188}
        - COMFYUI_METRICS_PORT=${COMFYUI_METRICS_PORT:-28188}
        - JUPYTER_PORT_HOST=${JUPYTER_PORT_HOST:-8888}
        - JUPYTER_METRICS_PORT=${JUPYTER_METRICS_PORT:-28888}
        - SERVERLESS=${SERVERLESS:-false}

the Dockerfile in the build folder:

  GNU nano 6.2                                                                                                                                            Dockerfile                                                                                                                                                      # For build automation - Allows building from any ai-dock base image
# Use a *cuda*base* image as default because pytorch brings the libs
ARG IMAGE_BASE="ghcr.io/ai-dock/jupyter-pytorch:2.2.0-py3.10-cpu-22.04"
FROM ${IMAGE_BASE}

LABEL org.opencontainers.image.source https://github.com/ai-dock/comfyui
LABEL org.opencontainers.image.description "ComfyUI Stable Diffusion backend and GUI"
LABEL maintainer="Rob Ballantyne <rob@dynamedia.uk>"

ENV IMAGE_SLUG="comfyui"
ENV OPT_SYNC=ComfyUI:serverless

# Copy early so we can use scripts in the build - Changes to these files will invalidate the cache and cause a rebuild.
COPY --chown=0:1111 ./COPY_ROOT/ /

# Use build scripts to ensure we can build all targets from one Dockerfile in a single layer.
# Don't put anything heavy in here - We can use multi-stage building above if necessary.

ARG IMAGE_BASE
RUN set -eo pipefail && /opt/ai-dock/bin/build/layer0/init.sh | tee /var/log/build.log

# Must be set after layer0
ENV MAMBA_DEFAULT_ENV=comfyui
ENV MAMBA_DEFAULT_RUN="micromamba run -n $MAMBA_DEFAULT_ENV"

# Copy overrides and models into later layers for fast rebuilds
COPY --chown=0:1111 ./COPY_ROOT_EXTRA/ /
RUN set -eo pipefail && /opt/ai-dock/bin/build/layer1/init.sh | tee -a /var/log/build.log

# Keep init.sh as-is and place additional logic in /opt/ai-dock/bin/preflight.sh
CMD ["init.sh"]

other than that i just cloned the github repo yesterday.
the error i am getting:

supervisor_1  | got prompt
supervisor_1  | model_type EPS
supervisor_1  | Using split attention in VAE
supervisor_1  | Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
supervisor_1  | Using split attention in VAE
supervisor_1  | 2024-03-10 17:02:44,181 INFO exited: comfyui (exit status 137; not expected)
supervisor_1  | 2024-03-10 17:02:44,632 INFO spawned: 'comfyui' with pid 768
supervisor_1  |
supervisor_1  | ==> /var/log/supervisor/caddy.log <==
supervisor_1  | {"level":"error","ts":1710090163.5710387,"logger":"http.log.error","msg":"dial tcp: lookup localhost: i/o timeout","request":{"remote_ip":"192.168.2.101","remote_port":"53571","client_ip":"192.168.2.101","proto":"HTTP/1.1","method":"POST","host":"192.168.1.101:1111","uri":"/ajax/logs","headers":{"Dnt":["1"],"Accept-Encoding":["gzip, deflate"],"Hx-Target":["page"],"Referer":["http://192.168.1.101:1111/"],"Hx-Current-Url":["http://192.168.1.101:1111/"],"Content-Length":["0"],"Content-Type":["application/x-www-form-urlencoded"],"Cookie":[],"Sec-Gpc":["1"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123.0"],"Hx-Request":["true"],"Origin":["http://192.168.1.101:1111"],"Connection":["keep-alive"],"Accept":["*/*"],"Accept-Language":["en-US,en;q=0.7,de-AT;q=0.3"]}},"duration":81.496884408,"status":502,"err_id":"y5ew3hics","err_trace":"reverseproxy.statusError (reverseproxy.go:1267)"}
supervisor_1  | {"level":"error","ts":1710090163.9898791,"logger":"http.log.error","msg":"dial tcp 127.0.0.1:18188: connect: connection refused","request":{"remote_ip":"192.168.2.101","remote_port":"53602","client_ip":"192.168.2.101","proto":"HTTP/1.1","method":"GET","host":"192.168.1.101:8188","uri":"/queue","headers":{"Dnt":["1"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123.0"],"Referer":["http://192.168.1.101:8188/"],"Connection":["keep-alive"],"Cookie":[],"Sec-Gpc":["1"],"Accept":["*/*"],"Accept-Language":["en-US,en;q=0.7,de-AT;q=0.3"],"Accept-Encoding":["gzip, deflate"],"Comfy-User":["undefined"]}},"duration":0.024025992,"status":502,"err_id":"47kp5rvq8","err_trace":"reverseproxy.statusError (reverseproxy.go:1267)"}
supervisor_1  | {"level":"error","ts":1710090163.991298,"logger":"http.log.error","msg":"dial tcp 127.0.0.1:18188: connect: connection refused","request":{"remote_ip":"192.168.2.101","remote_port":"53603","client_ip":"192.168.2.101","proto":"HTTP/1.1","method":"GET","host":"192.168.1.101:8188","uri":"/history?max_items=200","headers":{"Accept-Encoding":["gzip, deflate"],"Comfy-User":["undefined"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123.0"],"Referer":["http://192.168.1.101:8188/"],"Connection":["keep-alive"],"Cookie":[],"Dnt":["1"],"Sec-Gpc":["1"],"Accept":["*/*"],"Accept-Language":["en-US,en;q=0.7,de-AT;q=0.3"]}},"duration":0.000925198,"status":502,"err_id":"pdq6nmh3q","err_trace":"reverseproxy.statusError (reverseproxy.go:1267)"}
supervisor_1  | {"level":"error","ts":1710090164.604226,"logger":"http.log.error","msg":"dial tcp 127.0.0.1:18188: connect: connection refused","request":{"remote_ip":"192.168.2.101","remote_port":"53604","client_ip":"192.168.2.101","proto":"HTTP/1.1","method":"GET","host":"192.168.1.101:8188","uri":"/ws?clientId=5015f3e3d12b455f90d1aa312ca028ac","headers":{"Accept-Language":["en-US,en;q=0.7,de-AT;q=0.3"],"Sec-Websocket-Key":["dwWr+fR6Q71khkaZvhXXSg=="],"Cookie":[],"Cache-Control":["no-cache"],"Upgrade":["websocket"],"Accept":["*/*"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123.0"],"Pragma":["no-cache"],"Sec-Websocket-Version":["13"],"Origin":["http://192.168.1.101:8188"],"Sec-Websocket-Extensions":["permessage-deflate"],"Connection":["keep-alive, Upgrade"],"Accept-Encoding":["gzip, deflate"]}},"duration":0.00243078,"status":502,"err_id":"4ezxns4f4","err_trace":"reverseproxy.statusError (reverseproxy.go:1267)"}
supervisor_1  |
supervisor_1  | ==> /var/log/supervisor/comfyui.log <==
supervisor_1  | Starting ComfyUI...
supervisor_1  | Starting ComfyUI...
supervisor_1  |
supervisor_1  | ==> /var/log/supervisor/supervisor.log <==
supervisor_1  | 2024-03-10 17:02:44,181 INFO exited: comfyui (exit status 137; not expected)
supervisor_1  | 2024-03-10 17:02:44,632 INFO spawned: 'comfyui' with pid 768
supervisor_1  |
supervisor_1  | ==> /var/log/supervisor/caddy.log <==
supervisor_1  | {"level":"error","ts":1710090165.2220185,"logger":"http.log.error","msg":"dial tcp 127.0.0.1:18188: connect: connection refused","request":{"remote_ip":"192.168.2.101","remote_port":"53607","client_ip":"192.168.2.101","proto":"HTTP/1.1","method":"GET","host":"192.168.1.101:8188","uri":"/ws?clientId=b083dc43d61044c6ab65b32f1cdde643","headers":{"Pragma":["no-cache"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123.0"],"Accept":["*/*"],"Origin":["http://192.168.1.101:8188"],"Cookie":[],"Upgrade":["websocket"],"Sec-Websocket-Version":["13"],"Sec-Websocket-Extensions":["permessage-deflate"],"Accept-Encoding":["gzip, deflate"],"Cache-Control":["no-cache"],"Accept-Language":["en-US,en;q=0.7,de-AT;q=0.3"],"Sec-Websocket-Key":["kNwEmADTHsl/vJ/0d/P9Fg=="],"Connection":["keep-alive, Upgrade"]}},"duration":0.001117775,"status":502,"err_id":"sm5g3mwcy","err_trace":"reverseproxy.statusError (reverseproxy.go:1267)"}
supervisor_1  |
supervisor_1  | ==> /var/log/supervisor/comfyui.log <==
supervisor_1  | ** ComfyUI startup time: 2024-03-10 17:02:45.025688
supervisor_1  | ** Platform: Linux
supervisor_1  | ** Python version: 3.10.13 | packaged by conda-forge | (main, Dec 23 2023, 15:36:39) [GCC 12.3.0]
supervisor_1  | ** Python executable: /opt/micromamba/envs/comfyui/bin/python
supervisor_1  | ** Log path: /workspace/ComfyUI/comfyui.log
supervisor_1  |
supervisor_1  | Prestartup times for custom nodes:
supervisor_1  |    0.1 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-Manager
supervisor_1  |
supervisor_1  |
supervisor_1  | ==> /var/log/supervisor/caddy.log <==
supervisor_1  | {"level":"error","ts":1710090166.064027,"logger":"http.log.error","msg":"dial tcp 127.0.0.1:18188: connect: connection refused","request":{"remote_ip":"192.168.2.101","remote_port":"53608","client_ip":"192.168.2.101","proto":"HTTP/1.1","method":"GET","host":"192.168.1.101:8188","uri":"/ws?clientId=5015f3e3d12b455f90d1aa312ca028ac","headers":{"Sec-Websocket-Version":["13"],"Sec-Websocket-Extensions":["permessage-deflate"],"Sec-Websocket-Key":["R2uKGn4klgocfuTu+fSF+A=="],"Connection":["keep-alive, Upgrade"],"Accept":["*/*"],"Accept-Language":["en-US,en;q=0.7,de-AT;q=0.3"],"Upgrade":["websocket"],"Cookie":[],"Pragma":["no-cache"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123.0"],"Accept-Encoding":["gzip, deflate"],"Cache-Control":["no-cache"],"Origin":["http://192.168.1.101:8188"]}},"duration":0.000765085,"status":502,"err_id":"fkc66rk5b","err_trace":"reverseproxy.statusError (reverseproxy.go:1267)"}
supervisor_1  | {"level":"error","ts":1710090167.3226793,"logger":"http.log.error","msg":"dial tcp 127.0.0.1:18188: connect: connection refused","request":{"remote_ip":"192.168.2.101","remote_port":"53609","client_ip":"192.168.2.101","proto":"HTTP/1.1","method":"GET","host":"192.168.1.101:8188","uri":"/ws?clientId=b083dc43d61044c6ab65b32f1cdde643","headers":{"Sec-Websocket-Key":["/7oTZHXZkskiKv9SMGrrew=="],"Accept":["*/*"],"Accept-Encoding":["gzip, deflate"],"Accept-Language":["en-US,en;q=0.7,de-AT;q=0.3"],"Sec-Websocket-Version":["13"],"Pragma":["no-cache"],"Sec-Websocket-Extensions":["permessage-deflate"],"Connection":["keep-alive, Upgrade"],"Cache-Control":["no-cache"],"Upgrade":["websocket"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123.0"],"Origin":["http://192.168.1.101:8188"],"Cookie":[]}},"duration":0.001707567,"status":502,"err_id":"i6zh30sn6","err_trace":"reverseproxy.statusError (reverseproxy.go:1267)"}
supervisor_1  |
supervisor_1  | ==> /var/log/supervisor/comfyui.log <==
supervisor_1  | Total VRAM 7937 MB, total RAM 7937 MB
supervisor_1  | Set vram state to: DISABLED
supervisor_1  | Device: cpu
supervisor_1  | VAE dtype: torch.float32
supervisor_1  | Using sub quadratic optimization for cross attention, if you have memory or speed issues try using: --use-split-cross-attention
supervisor_1  | 2024-03-10 17:02:49,846 INFO success: comfyui entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
supervisor_1  |
supervisor_1  | ==> /var/log/supervisor/caddy.log <==
supervisor_1  | {"level":"error","ts":1710090169.2041223,"logger":"http.log.error","msg":"dial tcp 127.0.0.1:18188: connect: connection refused","request":{"remote_ip":"192.168.2.101","remote_port":"53610","client_ip":"192.168.2.101","proto":"HTTP/1.1","method":"GET","host":"192.168.1.101:8188","uri":"/ws?clientId=5015f3e3d12b455f90d1aa312ca028ac","headers":{"Accept-Language":["en-US,en;q=0.7,de-AT;q=0.3"],"Connection":["keep-alive, Upgrade"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123.0"],"Accept-Encoding":["gzip, deflate"],"Sec-Websocket-Extensions":["permessage-deflate"],"Accept":["*/*"],"Pragma":["no-cache"],"Cache-Control":["no-cache"],"Upgrade":["websocket"],"Sec-Websocket-Version":["13"],"Origin":["http://192.168.1.101:8188"],"Sec-Websocket-Key":["k72s8QRipsT6yoKJQwv9MA=="],"Cookie":[]}},"duration":0.001642951,"status":502,"err_id":"tp6ne2h6p","err_trace":"reverseproxy.statusError (reverseproxy.go:1267)"}
supervisor_1  |
supervisor_1  | ==> /var/log/supervisor/comfyui.log <==
supervisor_1  | ### Loading: ComfyUI-Manager (V2.9)
supervisor_1  |
supervisor_1  | ==> /var/log/supervisor/supervisor.log <==
supervisor_1  | 2024-03-10 17:02:49,846 INFO success: comfyui entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
supervisor_1  |
supervisor_1  | ==> /var/log/supervisor/comfyui.log <==
supervisor_1  | ### ComfyUI Revision: 2057 [65397ce6] | Released on '2024-03-10'
supervisor_1  |
supervisor_1  | Import times for custom nodes:
supervisor_1  |    0.1 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-Manager
supervisor_1  |
supervisor_1  | Starting server
supervisor_1  |
supervisor_1  | To see the GUI go to: http://127.0.0.1:18188
supervisor_1  | [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/model-list.json
supervisor_1  | [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/alter-list.json
supervisor_1  | [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/custom-node-list.json
supervisor_1  | [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/extension-node-map.json

I probably messed something up with the cpu-only config, but i dont know where. could you please help me and maybe provide better documentation for cpu-only mode?
@robballantyne
Copy link
Member

Out of memory, please follow instructions in logfile

@robballantyne robballantyne closed this as not planned Won't fix, can't repro, duplicate, stale Aug 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@robballantyne @TechanisYT