Skip to content

Commit

Permalink
Improve doc about GiB vs GB
Browse files Browse the repository at this point in the history
  • Loading branch information
lesteve committed Jun 18, 2020
1 parent 9ffa053 commit 0049ba7
Showing 1 changed file with 10 additions and 11 deletions.
21 changes: 10 additions & 11 deletions docs/source/configuration-setup.rst
Original file line number Diff line number Diff line change
Expand Up @@ -50,20 +50,19 @@ string, like "100 GB".
Gigabyte vs Gibibyte
~~~~~~~~~~~~~~~~~~~~

It is important to note that Dask makes the difference between
power of 2 and power of 10 when specifying memory. This means that:
Dask makes the difference between GB (gigabyte) and GiB (gibibyte):

- 1GB = :math:`10^9` bytes
- 1GiB = :math:`2^{30}` bytes
- 1GiB = :math:`2^{30} = 1024^{3}` bytes :math:`\approx 1.074` GB

``memory`` configuration is interpreted by Dask memory parser, and for most
JobQueueCluster implementation translated as a resource requirement for job
submission.
But most job schedulers (this is the case with PBS and Slurm at least) uses
KB or GB, but mean KiB or GiB. Dask jobqueue takes that into account, so you
may not find the amount of memory you were expecting when querying your job
queuing system. To give an example, with PBSCluster, if you specify '20GB' for
the ``memory`` kwarg, you will end up with a request for 19GB on PBS side.
This is because 20GB ~= 18.6GiB, which is rounded up.
JobQueueCluster implementation translated as a resource requirement for job
submission. But most job schedulers (this is the case with PBS and Slurm at
least) uses KB or GB, but mean KiB or GiB. Dask jobqueue takes that into
account, so you may not find the amount of memory you were expecting when
querying your job queuing system. To give an example, with PBSCluster, if you
specify ``memory='20GB'`` you will end up with a request for 19GB on PBS side.
This is because 20GB :math:`\approx` 18.6GiB, which is rounded up.

This can be avoided by always using 'GiB' in dask-jobqueue configuration.

Expand Down

0 comments on commit 0049ba7

Please sign in to comment.