Comfyui single gpu model switch

[jamilnielsen]

was trying to expand my cmdStop to be more robust+dynamic, but i'm having some trouble.
i don't know how to make commands in cmdstop output debug info either to logs that i can read from UI or container logs.

it just gets stuck, i test some commands from my own pc on same network and it looks fundamentally fine to me, what am i doing wrong?

  comfyui:
    proxy: http://192.168.0.103:8188
    checkEndpoint: /system_stats
    unlisted: true
    unloadTimeout: 10
    cmd: tail -f /dev/null

    # When switching away from ComfyUI, free ComfyUI model VRAM without
    # stopping the ComfyUI process.
    cmdStop: >
      sh -lc '
      # Wait for ComfyUI to finish its current job
      while [ 1 ]; do
        # Use curl with --fail to handle errors gracefully
        response=$(curl -fsS --max-time 10 http://192.168.0.103:8188/prompt 2>/dev/null)
        if [ -n "$response" ]; then
          # Extract queue_remaining using basic string manipulation
          queue_remaining=$(echo "$response" | sed -n "s/.*\"queue_remaining\": *\\([0-9]*\\).*/\\1/p")
          if [ -n "$queue_remaining" ] && [ "$queue_remaining" = "0" ]; then
            break
          fi
        fi
        echo "Waiting for ComfyUI to finish job..."
        sleep 2
      done

      # Free ComfyUI model VRAM
      curl -fsS -X POST http://192.168.0.103:8188/api/free \
      -H "Content-Type: application/json" \
      -d "{\"unload_models\":true,\"free_memory\":true}" || true;

      # Wait until GPU is sufficiently free (90% free VRAM)
      while [ 1 ]; do
        # Get system stats and parse using basic string manipulation
        stats=$(curl -fsS --max-time 10 http://192.168.0.103:8188/api/system_stats 2>/dev/null)
        if [ -n "$stats" ]; then
          vram_total=$(echo "$stats" | sed -n "s/.*\"vram_total\": *\\([0-9]*\\).*/\\1/p")
          vram_free=$(echo "$stats" | sed -n "s/.*\"vram_free\": *\\([0-9]*\\).*/\\1/p")
          if [ -n "$vram_total" ] && [ -n "$vram_free" ] && [ "$vram_free" -gt "$((vram_total * 9 / 10))" ]; then
            echo "GPU has sufficient free memory (90%+)."
            break
          fi
        fi
        echo "Waiting for GPU to free up..."
        sleep 2
      done

      # Kill the process if it\'s still running
      kill -TERM ${PID} 2>/dev/null || true
      '

here's the old working example i found and tested. (but lacks proper shutdown control)

  comfyui:
    proxy: http://192.168.0.103:8188
    checkEndpoint: /system_stats
    unlisted: true
    unloadTimeout: 10  # Adds a 10-second graceful timeout for unloading
    # ComfyUI is already running separately. This command is only a
    # llama-swap lifecycle sentinel for arbitration.
    cmd: tail -f /dev/null

    # When switching away from ComfyUI, free ComfyUI model VRAM without
    # stopping the ComfyUI process.
    cmdStop: >
      sh -lc 'curl -fsS -X POST http://192.168.0.103:8188/api/free
      -H "Content-Type: application/json"
      -d "{\"unload_models\":true,\"free_memory\":true}" || true;
      kill -TERM ${PID} 2>/dev/null || true'

[mostlygeek]

A custom wrapper like https://github.com/mostlygeek/llama-swap/tree/main/cmd/vllm-wrapper but for comfy would be more effective. When the wrapper starts it can send an HTTP call to comfy to load models and when the wrapper stops (SIGINT) it can send the unload models call.

[jamilnielsen]

what my cmdstop tries to do.
A: check with /prompt if comfyui is working on a job, and waits until done.
B: unload model+free memory
C: use systemstats to compare vram to vram max, when 90% is free, good enough.

currently not sure how i would do that as a wrapper.
regardless i would probably need llama-swap to somehow "wait" until it recieves the ok signal to go on.
that or a on system gpu busy check+free vram vs total vram.

[jamilnielsen]

currently the main issue is that it seems cmdStop hates "while", because it will run everything else just fine.

[jamilnielsen]

okey here's my updated code, confirmed that the entire thing works fine even from within the container through sh, but llama-swap still forces a "[ERROR] sendStopSignal() stop command failed: exit status 2"

  comfyui:
    proxy: http://192.168.0.103:8188
    checkEndpoint: /system_stats
    unlisted: true
    unloadTimeout: 10
    cmd: tail -f /dev/null
    cmdStop: >
      sh -lc 'while [ 1 ]; do
        response=$(curl -fsS --max-time 10 http://192.168.0.103:8188/prompt 2>/dev/null)
        if [ -n "$response" ]; then
          queue_remaining=$(echo "$response" | sed -n "s/.*\"queue_remaining\": *\\([0-9]*\\).*/\\1/p")
          echo "Loop1"
          if [ -n "$queue_remaining" ] && [ "$queue_remaining" = "0" ]; then
            break
          fi
        fi
      done
      curl -fsS -X POST http://192.168.0.103:8188/api/free \
      -H "Content-Type: application/json" \
      -d "{\"unload_models\":true,\"free_memory\":true}" || true;
      echo "unloaded models"
      while [ 1 ]; do
        stats=$(curl -fsS --max-time 10 http://192.168.0.103:8188/api/system_stats 2>/dev/null)
        if [ -n "$stats" ]; then
          vram_total=$(echo "$stats" | sed -n "s/.*\"vram_total\": *\\([0-9]*\\).*/\\1/p")
          vram_free=$(echo "$stats" | sed -n "s/.*\"vram_free\": *\\([0-9]*\\).*/\\1/p")
          echo "Loop2"
          if [ -n "$vram_total" ] && [ -n "$vram_free" ] && [ "$vram_free" -gt "$((vram_total * 9 / 10))" ]; then
            break
          fi
        fi
      done
      kill -TERM ${PID} 2>/dev/null || true'

so far the biggest problem is that i cant do any kind of proper debugging, because i cant get echo signals to reach any logging.

[mostlygeek]

The reason there are no logs for cmdStop is because the stdout/stderr are not hooked up into it.

I can make a PR to add that.

[jamilnielsen]

Alright, for anyone who wants a setting for Comfyui that achieves the following.
A: llama-swap pokes comfyui's api/prompt, that tells us how many active jobs/queue, we then wait until they are done.
B: api/free unload_models and free_memory, telling comfyui to clean up now that its hopefully not busy.
C: api/system_stats, we fetch vram_total and vram_free, and do some math, accounting for comfyui being a mess, if 93% is free, comfyui is probably done clearing up, if not, repeat from B.

Idle time between vram_threshold is met and llama-swap loads something else is at best 3 seconds or so, good enough!
The result is with one GPU, you can now reliably have Comfyui hooked up to llama-swap, without having to constantly manually clear comfyui.

Things for you to change.
IP/PORT (4 of them) - point it at your comfyui.
vram_threshold (leaving it at 93 might not be ideal for everyone)

  comfyui:
    proxy: http://localhost:8188
    checkEndpoint: /system_stats
    unlisted: true
    unloadTimeout: 180
    cmd: tail -f /dev/null
    cmdStop: |
      sh -c '
        for i in $(seq 1 30); do
          response=$(curl -fsS --max-time 10 http://localhost:8188/prompt 2>/dev/null)
          if [ -n "$response" ]; then
            queue_remaining=$(echo "$response" | grep -o "\"queue_remaining\": [0-9]*" | cut -d " " -f 2)
            if [ -n "$queue_remaining" ] && [ "$queue_remaining" = "0" ]; then
              break
            fi
          fi
          sleep 2
        done
        vram_threshold=93
        for i in $(seq 1 60); do
          sleep 1
          curl -s -X POST http://localhost:8188/api/free -H "Content-Type: application/json" -d "{\"unload_models\": true, \"free_memory\": true}"
          sleep 2
          stats=$(curl -fsS --max-time 10 http://localhost:8188/api/system_stats 2>/dev/null)
          if [ -n "$stats" ]; then
            vram_total=$(echo "$stats" | grep -o "\"vram_total\": [0-9]*" | head -1 | cut -d " " -f 2)
            vram_free=$(echo "$stats" | grep -o "\"vram_free\": [0-9]*" | head -1 | cut -d " " -f 2)
            if [ -n "$vram_total" ] && [ -n "$vram_free" ] && [ "$vram_total" -gt 0 ] 2>/dev/null; then
              pct=$((vram_free * 100 / vram_total))
              if [ "$pct" -ge "$vram_threshold" ] 2>/dev/null; then
                sleep 2
                break
              fi
            fi
          fi
        done
        kill -TERM ${PID} 2>/dev/null || true
      '

this works on my setup, burned probably about 140k-200k tokens on my local hardware, electric bill go brr.

@mostlygeek, feel free to add this as an example or something if its up to your standard.
oh and i managed to do debugging using curl commands that poked an external container that would snitch to its docker logs.

[LvNaTiCC]

Thanks, it is some helpful scripting you did there!

While this works, it can be made better my using a reverse proxy in front of ComfyUI. The basic problem is the fact that any request to /upstream/comfyui will result in llama-swap considering availability of the resources, and evictions. And since ComfyUI does constant healthchecks/job checks/websocket requets without you even doing anything, so a single comfyui frontend opened anywhere will make llama-swap unusuable by hogging GPU.

A solution I arrived at is using a nginx proxy that proxies /api/prompt location to the /comfyui/upstream/api/prompt, and everything else goes to comfy directly. This results in ComfyUI being usable without triggering llama-swap until user actually prompts.