Unsetting `MALLOC_TRIM_THRESHOLD_` is sensitive to import order

[crusaderky]

import distributed respects any previous calls to dask.config.set that change a value, but not those that remove a key:

>>> import dask
>>> dask.config.set({"distributed.nanny.pre-spawn-environ": {'OMP_NUM_THREADS': 2}})
>>> dask.config.get("distributed.nanny.pre-spawn-environ")
{'OMP_NUM_THREADS': 2}
>>> import distributed
>>> dask.config.get("distributed.nanny.pre-spawn-environ")
{'OMP_NUM_THREADS': 2, 'MALLOC_TRIM_THRESHOLD_': 65536, 'MKL_NUM_THREADS': 1, 'OPENBLAS_NUM_THREADS': 1}

The problem here is that an unset MALLOC_TRIM_THRESHOLD_ has a special meaning for the glibc so it can't just simply be overridden with a different explicit value.

[ntabris]

For Coiled, it would be very helpful if the solution also worked for env vars and yaml, not just the in memory config dict.

[crusaderky]

For Coiled, it would be very helpful if the solution also worked for env vars and yaml, not just the in memory config dict.

Yes, with the linked PR you can set:

$ export DASK_DISTRIBUTED__NANNY__PRE_SPAWN_ENVIRON__MALLOC_TRIM_THRESHOLD_=None

(case sensitive)

or you can use a yaml file:

distributed:
  nanny:
    pre-spawn-environ:
      MALLOC_TRIM_THRESHOLD_: null

For clarity, the above should not be the default for coiled.