Numba-based StdDevUDF

docs/source/changelog/features/stddev.rst

@@ -1,4 +1,4 @@
 [Feature] :code:`StdDevUDF` speed-up
 ====================================
 
-* More than 2x speed-up of standard deviation calculation with tiled processing (:pr:`625`)
+* About 10x speed-up of standard deviation calculation for large frames (:pr:`625,640`)

docs/source/changelog/misc/stddev.rst

@@ -0,0 +1,11 @@
+[Misc] Rename result buffers of StdDevUDF
+=========================================
+
+* Rename result buffers of :class:`~libertem.udf.stddev.StdDevUDF`,
+  :meth:`~libertem.udf.stddev.run_stddev` and
+  :meth:`~libertem.udf.stddev.consolidate_result` from :code:`'sum_frame'` to
+  :code:`'sum'`, :code:`'num_frame'` to :code:`'num_frames'` (:pr:`640`)
+* Resolve ambiguity between variance and sum of variances in result buffer names of
+  :class:`~libertem.udf.stddev.StdDevUDF`,
+  :meth:`~libertem.udf.stddev.run_stddev` and
+  :meth:`~libertem.udf.stddev.consolidate_result`. (:pr:`640`)

docs/source/citing.rst

@@ -4,7 +4,7 @@ Citing
 To help us spread the word, please credit and cite LiberTEM in publications where it has been significant. 
 Resources for citing LiberTEM are linked through the DOI badge below.
 
-.. image:: https://zenodo.org/badge/DOI/10.5281/zenodo.1478763.svg
-   :target: https://doi.org/10.5281/zenodo.1478763
+.. |zenodo| image:: https://zenodo.org/badge/DOI/10.5281/zenodo.1477847.svg
+.. _zenodo: https://doi.org/10.5281/zenodo.1477847
 
 Please refer to our :ref:`authorship` for details regarding authorship and credit for contributors.

docs/source/reference/udf.rst

@@ -50,36 +50,58 @@ Some generally useful UDFs are included with LiberTEM:
 
 .. _`sum udf`:
 
+Sum of frames
+#############
+
 .. autoclass:: libertem.udf.sum.SumUDF
     :members:
     :special-members: __init__
 
 .. _`logsum udf`:
 
+Sum of log-scaled frames
+########################
+
 .. autoclass:: libertem.udf.logsum.LogsumUDF
     :members:
     :special-members: __init__
 
 .. _`stddev udf`:
 
+Standard deviation
+##################
+
 .. autoclass:: libertem.udf.stddev.StdDevUDF
     :members:
     :special-members: __init__
 
+.. autofunction:: libertem.udf.stddev.run_stddev
+
+.. autofunction:: libertem.udf.stddev.consolidate_result
+
 .. _`sumsig udf`:
 
+Sum per frame
+#############
+
 .. autoclass:: libertem.udf.sumsigudf.SumSigUDF
     :members:
     :special-members: __init__
 
 .. _`masks udf`:
 
+Apply masks
+###########
+
 .. autoclass:: libertem.udf.masks.ApplyMasksUDF
     :members:
     :special-members: __init__
 
 .. _`pick udf`:
 
+Load data
+#########
+
 .. autoclass:: libertem.udf.raw.PickUDF
     :members:
     :special-members: __init__

docs/source/references-libertem.bib

@@ -1117,4 +1117,14 @@ @article{Lichte2008
 	pages = {016102},
 	file = {Electron holography—basics and applications - IOPscience:/Users/migunov/Zotero/storage/7C3UUE82/meta\;jsessionid=3F5DA019F9A86C312BFD8889CFC17030.c2.iopscience.cld.iop.html:text/html;Lichte_H,Lehmann_M-E-holog-basics&app(2008).pdf:/Users/migunov/Zotero/storage/WRB52ACC/Lichte_H,Lehmann_M-E-holog-basics&app(2008).pdf:application/pdf}
 }
+
+@InProceedings{Schubert2018,
+  author    = {Erich Schubert and Michael Gertz},
+  title     = {Numerically stable parallel computation of (co-)variance},
+  booktitle = {Proceedings of the 30th International Conference on Scientific and Statistical Database Management - {SSDBM} 18},
+  year      = {2018},
+  publisher = {{ACM} Press},
+  doi       = {10.1145/3221269.3223036},
+}
+
 @Comment{jabref-meta: databaseType:bibtex;}

docs/source/udf.rst

@@ -446,6 +446,9 @@ the :code:`.data` attribute, or by calling :meth:`numpy.array`:
 
 .. _`udf roi`:
 
+Regions of interest
+-------------------
+
 In addition, you can pass the :code:`roi` (region of interest) parameter, to
 run your UDF on a selected subset of data. :code:`roi` should be a NumPy array
 containing a bool mask, having the shape of the navigation axes of the dataset.

setup.py

@@ -144,9 +144,8 @@ def find_version(*file_paths):
         "pillow",
         "h5py",
         "psutil",
-        # Pinned due to https://github.com/pydata/sparse/issues/257
-        # Ensure compatibility with numpy 1.17
-        "numba>=0.46",
+        # Bounds checking in Numba
+        "numba>=0.47",
         "ncempy>=1.4",
         'pywin32!=226;platform_system=="Windows"',
         # FIXME pull request #259

src/libertem/analysis/sd.py

@@ -14,6 +14,7 @@ def get_roi(self):
         return get_roi(params=self.parameters, shape=self.dataset.shape.nav)
 
     def get_udf_results(self, udf_results, roi):
+        udf_results = std.consolidate_result(udf_results)
         return AnalysisResultSet([
             AnalysisResult(raw_data=udf_results['var'].data,
                            visualized=visualize_simple(