You do not need CliMetLab to use dask. Dask is an independent python package often use with. to xarray and pandas. As CliMetLab seeks a strong integration to dask, some high-level functions may be useful to ease the interaction with dask.
This whole part about dask is EXPERIMENTAL, it will change in the future, may be removed or (very likely) moved to another package.
from climetlab.utils.dask import start
start('local')
# or $ climetlab dask start local
# or $ climetlab dask local --start
from climetlab.utils.dask import start
start('ssh')
from climetlab.utils.dask import start
start('slurm')
Note
This is assumes that your HPC admin set up the hpc-name-config-1.yaml file on the appropriate location.
from climetlab.utils.dask import start
start('hpc-name-config-1')
Not implemented yet.
Not implemented yet.
The dask cluster and client will usually stop automatically when the python process ends. Nevertheless, it is possible to stop dask if it has been started from climetlab.
from climetlab.utils.dask import stop
stop()
Note: In this section a "dask deployement" refers to a client and a cluster. It does not refers to a Cloud deployement using Kubernetes, etc.
Create the yaml file $HOME/.climetlab/dask/hpc-name-config-1.yaml. Then use it with:
from climetlab.utils.dask import start
start('hpc-name-config-1')
This is EXPERIMENTAL.
Note
For HPC system admin: Adding yaml files in /opt/climetlab/dask/*.yaml will give global access to all users.
from climetlab.utils.dask import start
client = start('local').client
Define what "scale" mean in this context.
from climetlab.utils.dask import start
deploy = start('slurm')
deploy.scale(..)