Requested dask.distributed scheduler but no Client active

[Cognitus-Stuti]

Hey, I am getting the above mentioned error when trying to do multiple lazy computations on a dask graph of size around 15 MB
Dask version: 2024.5.2
Dask Scheduler Compute: 1core, 1GB mem
Dask Workers: 3, 1core, 1GB mem each

Any workarounds for the same?

Environment:

• Dask version:2024.5.2
• Python version: 3.12
• Operating System: linux (docker)
• Install method (conda, pip, source): pip

[phofl]

Hi, could you provide a reproducer?

[Cognitus-Stuti]

I setup a distibuted dask scheduler with 3 workers and and scheduler with 1 core each 1 GB memory each
Using this as a base file for testing :https://github.com/tiwari91/Housing-Prices/blob/master/train.csv

using this as the docker setup
dask-scheduler:
image: ghcr.io/dask/dask:2024.5.2-py3.12
command:
- bash
- -c
- |
pip install requests aiohttp
dask-scheduler
networks:
- lambdax
extra_hosts:
- "host.docker.internal:host-gateway"

dask-worker:
image: ghcr.io/dask/dask:2024.5.2-py3.12
deploy:
replicas: 3
command:
- bash
- -c
- |
pip install requests aiohttp
dask worker dask-scheduler:8786

code:
df =dd.read_csv('train.csv, n_partitions=4)
df = dd.concat([df,df,df,df,df,df])
df = df.sort_values(by=['Id'],ascending=True)
async with Client("tcp://dask-scheduler:8786",asynchronous=True,direct_to_workers=True) as client:
df = await client.compute(df)

[Cognitus-Stuti]

#10452 this seems to be a similar issue

[phofl]

Can you create something with a LocalCluster instead and something that we can copy-paste?

[Cognitus-Stuti]

the main reason for adding this was to create a mini production like cluster, you can directly add this docker compose and do a
docker compose up -d

services:
  dask-scheduler:
    image: ghcr.io/dask/dask:2024.5.2-py3.12
    ports:
      - "8786:8786"
      - "8787:8787"
    command:
      - bash
      - -c
      - |
        pip install requests aiohttp
        dask-scheduler
    environment:
      - PYTHONWARNINGS=ignore
    networks:
      - dask
    extra_hosts:
      - "host.docker.internal:host-gateway"
    volumes:
      - /data:/data
      - ./etc/dask:/etc/dask
    deploy:
      resources:
        limits:
          cpus: "2"
          memory: "3G"
        reservations:
          cpus: "2"
          memory: "3G"
    restart: always

  dask-worker:
    image: ghcr.io/dask/dask:2024.5.2-py3.12
    environment:
      - DASK_CONFIG=/etc/dask
      - PYTHONWARNINGS=ignore

    volumes:
      - ./etc/dask:/etc/dask
      - ./tmp/dask-worker-space:/tmp/dask-worker-space
      - ./data:/data
    deploy:
      replicas: 3
      resources:
        limits:
          cpus: "2.0"
          memory: "3G"
        reservations:
          cpus: "2.0"
          memory: "3G"
    command:
      - bash
      - -c
      - |
        pip install requests aiohttp
        dask worker dask-scheduler:8786 
    networks:
      - dask
    extra_hosts:
      - "host.docker.internal:host-gateway"
    restart: always
networks:
  dask:
    driver: bridge
  default:
    external:
      name: host

[phofl]

Please use a LocalCluster for a reproducer, we don't want to install docker compose to reproduce something like this