update for executors, 0.2 (#43)

.readthedocs.yml

@@ -13,3 +13,4 @@ python:
       path: .
       extra_requirements:
         - docs
+        - complete

docs/api.rst

@@ -24,3 +24,25 @@ The Rechunked Object
 
 .. warning::
    You must manually delete the intermediate store when ``execute`` is finished.
+
+
+.. _api.executors:
+
+Executors
+---------
+
+Rechunking plans can be executed on a variety of backends. The following table lists the current options.
+
+.. autosummary::
+
+   rechunker.executors.beam.BeamExecutor
+   rechunker.executors.dask.DaskExecutor
+   rechunker.executors.prefect.PrefectExecutor
+   rechunker.executors.python.PythonExecutor
+   rechunker.executors.pywren.PywrenExecutor
+
+.. autoclass:: rechunker.executors.beam.BeamExecutor
+.. autoclass:: rechunker.executors.dask.DaskExecutor
+.. autoclass:: rechunker.executors.prefect.PrefectExecutor
+.. autoclass:: rechunker.executors.python.PythonExecutor
+.. autoclass:: rechunker.executors.pywren.PywrenExecutor

docs/executors.rst

@@ -0,0 +1,30 @@
+.. _executors:
+
+Executors
+=========
+
+``rechunker`` plans can be executed by a variety of executors. The default is ``dask``, which executes the plan on a Dask cluster.
+
+For example, we can use :ref:`rechunker.executors.python.PythonExecutor` to execute a plan as a simple Python for loop, which might be useful for debugging.
+
+.. code-block:: python
+
+  >>> import zarr
+  >>> from rechunker import rechunk
+  >>> source = zarr.ones((4, 4), chunks=(2, 2), store="source.zarr")
+  >>> intermediate = "intermediate.zarr"
+  >>> target = "target.zarr"
+  >>> rechunked = rechunk(
+  ...    source, target_chunks=(4, 1), target_store=target,
+  ...    max_mem=256000000, temp_store=intermediate,
+  ...    executor="python"
+  ... )
+  >>> rechunked.execute()
+
+.. note::
+
+   Most executors will require installing additional optional
+   dependencies.
+
+
+See :ref:`api.executors` for a list of all the different executors.

docs/release_notes.rst

@@ -1,7 +1,14 @@
 Release Notes
 =============
 
-v0.1 - Unreleased
+v0.2.0
+------
+
+- Added ``rechunker.executors`` for executing plans with other
+  backends like Apache Beam, prefect, and pywren. See :ref:`executors` for more.
+- Fixed overflow bug when computing the number of chunks needed for a memory target.
+
+v0.0.1
 -----------------
 
 - Documentation update and tutorial.