Workers not using environment variables for default config options

[rjzamora]

WARNING: The behavior reported here may not be a bug. For example, the consensus may be: "You must set your environment variables before importing dask." However, I want to document the recent change in behavior, because it's breaking dask-cuda CI.

What happened:

The workers within a LocalCluster do not use environment variables to set default config options unless the environment variables were set before dask was imported.

What you expected to happen:

Environment variables should still be used to set dask.config defaults after importing dask.

Minimal Complete Verifiable Example:

import os
import dask
from dask.distributed import LocalCluster, Client

def check_config():
    assert os.environ.get("DASK_DISTRIBUTED__COMM__COMPRESSION") == "False"
    print(f"distributed.comm.compression={dask.config.get('distributed.comm.compression')}")

os.environ["DASK_DISTRIBUTED__COMM__COMPRESSION"] = "False"
with LocalCluster(n_workers=1) as cluster:
    with Client(cluster) as client:
        client.run(check_config)

# Prints: distributed.comm.compression=auto