Daskify Stats (#70)

* Makes main stats functions dask-compatible. * Allows for datetimes to be submitted as the x-axis in regression functions.
bradyrx · Jul 8, 2020 · 9239fa3 · 9239fa3
1 parent 71c8de3
commit 9239fa3
Show file tree

Hide file tree

Showing 30 changed files with 1,955 additions and 576 deletions.
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -22,7 +22,7 @@ repos:
       rev: v2.4.0
       hooks:
       - id: flake8
-        args: ["--max-line-length=88", "--exclude=__init__.py", "--ignore=W605,W503,F722,C901"]
+        args: ["--max-line-length=88", "--exclude=__init__.py", "--ignore=W605,W503,F722,C901,F401"]
 
   -   repo: https://github.com/pre-commit/mirrors-isort
       rev: v4.3.21

diff --git a/CHANGELOG.rst b/CHANGELOG.rst
@@ -7,49 +7,89 @@ esmtools v1.1.1 (2019-12-##)
 
 Features
 --------
-- xarray implementation of ``statsmodels.stats.multitest.multipletests``. (:pr:`71`) `Aaron Spring`_
+- ``xarray`` implementation of ``statsmodels.stats.multitest.multipletests``.
+  (:pr:`71`) `Aaron Spring`_
+- Implements ``nan_policy=...`` keyword for ``stats.linear_slope``,
+  ``stats.linregress``, ``stats.polyfit``, ``stats.rm_poly``, ``stats.rm_trend``.
+  (:pr:`70`) `Riley X. Brady`_.
+
+  * ``'none', 'propagate'``: Propagate nans through function. I.e., return a nan for
+    a given grid cell if a nan exists in it.
+  * ``'raise'``: Raise an error if there are any nans in the datasets.
+  * ``'drop', 'omit'``: Like ``skipna``, compute statistical function after removing
+    nans.
+
+- Adds support for datetime axes in ``stats.linear_slope``, ``stats.linregress``,
+  ``stats.polyfit``, ``stats.rm_poly``, ``stats.rm_trend``. Converts datetimes to
+  numeric time, computes function, and then converts back to datetime.
+  (:pr:`70`)`Riley X. Brady`_.
+- ``stats.linear_slope``, ``stats.linregress``, ``stats.polyfit``, ``stats.rm_poly``,
+  and ``stats.rm_trend`` are now dask-compatible and vectorized better.
+  (:pr:`70`) `Riley X. Brady`_.
+
 
 Bug Fixes
 ---------
+- Does not eagerly evaluate ``dask`` arrays anymore. (:pr:`70`) `Riley X. Brady`_.
 
 Internals/Minor Fixes
 ---------------------
-- Adds ``isort`` and ``nbstripout`` to CI for development. Blacken and isort code. (:pr:`73`) `Riley X. Brady`_
+- Adds ``isort`` and ``nbstripout`` to CI for development. Blacken and isort code.
+  (:pr:`73`) `Riley X. Brady`_
 
 Documentation
 -------------
-- Add more robust API docs page, information on how to contribute, CHANGELOG, etc. to ``sphinx``. (:pr:`67`) `Riley X. Brady`_.
+- Add more robust API docs page, information on how to contribute, CHANGELOG, etc. to
+  ``sphinx``. (:pr:`67`) `Riley X. Brady`_.
 
 Deprecations
 ------------
-- Remove ``mpas`` and ``vis`` modules. The former is better for a project-dependent package. The latter essentially poorly replicates some of ``proplot`` functionality. (:pr:`69`) `Riley X. Brady`_.
+- Removes ``mpas`` and ``vis`` modules. The former is better for a project-dependent
+  package. The latter essentially poorly replicates some of ``proplot`` functionality.
+  (:pr:`69`) `Riley X. Brady`_.
+- Removes ``stats.smooth_series``, since there is an easy ``xarray`` function for it.
+  (:pr:`70`) `Riley X. Brady`_.
+- Changes ``stats.linear_regression`` to ``stats.linregress``.
+  (:pr:`70`) `Riley X. Brady`_.
+- Changes ``stats.compute_slope`` to ``stats.linear_slope``.
+  (:pr:`70`) `Riley X. Brady`_.
 
 esmtools v1.1 (2019-09-04)
 ==========================
 
 Features
 --------
-- ``co2_sol`` and ``schmidt`` now can be computed on grids and do not do time-averaging (:pr:`45`) `Riley X. Brady`_.
-- ``temp_decomp_takahashi`` now returns a dataset with thermal/non-thermal components (:pr:`45`) `Riley X. Brady`_.
-- ``temporal`` module that includes a ``to_annual()`` function for weighted temporal resampling (:pr:`50`) `Riley X. Brady`_.
-- ``filtering`` module renamed to ``spatial`` and ``find_indices`` made public. (:pr:`52`) `Riley X. Brady`_.
+- ``co2_sol`` and ``schmidt`` now can be computed on grids and do not do time-averaging
+  (:pr:`45`) `Riley X. Brady`_.
+- ``temp_decomp_takahashi`` now returns a dataset with thermal/non-thermal components
+  (:pr:`45`) `Riley X. Brady`_.
+- ``temporal`` module that includes a ``to_annual()`` function for weighted temporal
+  resampling (:pr:`50`) `Riley X. Brady`_.
+- ``filtering`` module renamed to ``spatial`` and ``find_indices`` made public.
+  (:pr:`52`) `Riley X. Brady`_.
 - ``standardize`` function moved to stats. (:pr:`52`) `Riley X. Brady`_.
 - ``loadutils`` removed (:pr:`52`) `Riley X. Brady`_.
-- ``calculate_compatible_emissions`` following Jones et al. 2013  (:pr:`54`) `Aaron Spring`_
-- Update ``corr`` to broadcast ``x`` and ``y`` such that a single time series can be correlated across a grid. (:pr:`58`) `Riley X. Brady`_.
-- ``convert_lon_to_180to180`` and ``convert_lon_to_0to360`` now wrapped with ``convert_lon`` and now supports 2D lat/lon grids. ``convert_lon()`` is also available as an accessor.  (:pr:`60`) `Riley X. Brady`_.
+- ``calculate_compatible_emissions`` following Jones et al. 2013
+  (:pr:`54`) `Aaron Spring`_
+- Update ``corr`` to broadcast ``x`` and ``y`` such that a single time series can be
+  correlated across a grid. (:pr:`58`) `Riley X. Brady`_.
+- ``convert_lon_to_180to180`` and ``convert_lon_to_0to360`` now wrapped with
+  ``convert_lon`` and now supports 2D lat/lon grids. ``convert_lon()`` is also
+  available as an accessor.  (:pr:`60`) `Riley X. Brady`_.
 
 Internals/Minor Fixes
 ---------------------
-- Changed name back to ``esmtools`` now that the readthedocs domain was cleared up. Thanks Andrew Walter! (:pr:`61`) `Riley X. Brady`_.
+- Changed name back to ``esmtools`` now that the readthedocs domain was cleared up.
+  Thanks Andrew Walter! (:pr:`61`) `Riley X. Brady`_.
 - ``esmtools`` documentation created with docstring updates for all functions.
 
 esm_analysis v1.0.2 (2019-07-27)
 ================================
 
 Internals/Minor Fixes
 ---------------------
-- Changed name from ``esmtools`` to ``esm_analysis`` since the former was registered on readthedocs.
+- Changed name from ``esmtools`` to ``esm_analysis`` since the former was registered
+  on readthedocs.
 
 esmtools v1.0.1 (2019-07-25)
 ============================

diff --git a/ci/run-linter.sh b/ci/run-linter.sh
@@ -7,10 +7,7 @@ echo "Code Styling with (black, flake8, isort)"
 source activate esmtools-dev
 
 echo "[flake8]"
-flake8 esmtools --max-line-length=88 --exclude=__init__.py --ignore=W605,W503
+flake8 esmtools --max-line-length=88 --exclude=__init__.py --ignore=W605,W503,F722,C901,F401
 
 echo "[black]"
 black --check --line-length=88 -S esmtools
-
-echo "[isort]"
-isort --recursive -w 88 --check-only esmtools 
diff --git a/docs/source/api.rst b/docs/source/api.rst
@@ -34,7 +34,9 @@ Composite Analysis
 
 .. currentmodule:: esmtools.composite
 
-Functions pertaining to composite analysis. Composite analysis takes the mean view of some field (e.g., sea surface temperature) when some climate index (e.g., El Nino Southern Oscillation) is in its negative or positive mode.
+Functions pertaining to composite analysis. Composite analysis takes the mean view of
+some field (e.g., sea surface temperature) when some climate index
+(e.g., El Nino Southern Oscillation) is in its negative or positive mode.
 
 .. autosummary::
     :toctree: api/
@@ -97,18 +99,18 @@ Statistical functions. A portion directly wrap functions from ``climpred``.
 .. autosummary::
     :toctree: api/
 
-    standardize
-    nanmean
-    cos_weight
+    ACF
     area_weight
-    smooth_series
-    fit_poly
-    linear_regression
+    autocorr
     corr
+    cos_weight
+    linear_slope
+    linregress
+    polyfit
+    nanmean
     rm_poly
     rm_trend
-    autocorr
-    ACF
+    standardize
 
 Testing
 -------
@@ -123,7 +125,7 @@ Statistical tests.
     :toctree: api/
 
     ttest_ind_from_stats
-    multipletest
+    multipletests
 
 Temporal
 --------

diff --git a/docs/source/api/esmtools.stats.fit_poly.rst b/docs/source/api/esmtools.stats.fit_poly.rst
diff --git a/docs/source/api/esmtools.stats.linear_regression.rst b/docs/source/api/esmtools.stats.linear_regression.rst
diff --git a/docs/source/api/esmtools.stats.linear_slope.rst b/docs/source/api/esmtools.stats.linear_slope.rst
@@ -0,0 +1,6 @@
+esmtools.stats.linear\_slope
+============================
+
+.. currentmodule:: esmtools.stats
+
+.. autofunction:: linear_slope
diff --git a/docs/source/api/esmtools.stats.linregress.rst b/docs/source/api/esmtools.stats.linregress.rst
@@ -0,0 +1,6 @@
+esmtools.stats.linregress
+=========================
+
+.. currentmodule:: esmtools.stats
+
+.. autofunction:: linregress
diff --git a/docs/source/api/esmtools.stats.polyfit.rst b/docs/source/api/esmtools.stats.polyfit.rst
@@ -0,0 +1,6 @@
+esmtools.stats.polyfit
+======================
+
+.. currentmodule:: esmtools.stats
+
+.. autofunction:: polyfit
diff --git a/docs/source/api/esmtools.stats.smooth_series.rst b/docs/source/api/esmtools.stats.smooth_series.rst
diff --git a/docs/source/api/esmtools.testing.multipletests.rst b/docs/source/api/esmtools.testing.multipletests.rst
@@ -0,0 +1,6 @@
+esmtools.testing.multipletests
+==============================
+
+.. currentmodule:: esmtools.testing
+
+.. autofunction:: multipletests
diff --git a/...i/esmtools.stats.ttest_ind_from_stats.rst → ...esmtools.testing.ttest_ind_from_stats.rst b/...i/esmtools.stats.ttest_ind_from_stats.rst → ...esmtools.testing.ttest_ind_from_stats.rst