Merge remote-tracking branch 'upstream/RELEASE_next_patch' into check…

…_init_spanROI # Conflicts: # CHANGES.rst
hyperspy · Mar 4, 2021 · 0ca2b39 · 0ca2b39
2 parents 32b04d4 + 8394ef0
commit 0ca2b39
Show file tree

Hide file tree

Showing 43 changed files with 922 additions and 332 deletions.
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
@@ -24,7 +24,7 @@ jobs:
           - os: ubuntu
             PYTHON_VERSION: 3.6
             OLDEST_SUPPORTED_VERSION: true
-            DEPENDENCIES: matplotlib==2.2.3 numpy==1.16.0 scipy==1.1 imagecodecs==2019.12.3
+            DEPENDENCIES: matplotlib==2.2.3 numpy==1.16.0 scipy==1.1 imagecodecs==2019.12.3 dask==2.0.0
             PIP_SELECTOR: '[all, tests]'
             LABEL: -oldest
           # test minimum requirement

diff --git a/CHANGES.rst b/CHANGES.rst
@@ -8,11 +8,21 @@ What's new
 RELEASE_next_patch (Unreleased)
 +++++++++++++++++++++++++++++++
 
-* Widgets plotting improvement and add `pick_tolerance` to plot preferences (`#2615 <https://github.com/hyperspy/hyperspy/pull/2615>`_)
+* Widgets plotting improvement and add ``pick_tolerance`` to plot preferences (`#2615 <https://github.com/hyperspy/hyperspy/pull/2615>`_)
 * Update external links in the loading data section of the user guide (`#2627 <https://github.com/hyperspy/hyperspy/pull/2627>`_)
 * Pass keyword argument to the image IO plugins (`#2627 <https://github.com/hyperspy/hyperspy/pull/2627>`_)
 * Drop support for numpy<1.16, in line with NEP 29 and fix protochip reader for numpy 1.20 (`#2616 <https://github.com/hyperspy/hyperspy/pull/2616>`_)
 * Run test suite against upstream dependencies (numpy, scipy, scikit-learn and scikit-image) (`#2616 <https://github.com/hyperspy/hyperspy/pull/2616>`_)
+* Improve error message when file not found (`#2597 <https://github.com/hyperspy/hyperspy/pull/2597>`_)
+* Add update instructions to user guide (`#2621 <https://github.com/hyperspy/hyperspy/pull/2621>`_)
+* Fix disconnect event when closing navigator only plot (fixes `#996 <https://github.com/hyperspy/hyperspy/issues/996>`_), (`#2631 <https://github.com/hyperspy/hyperspy/pull/2631>`_)
+* Improve plotting navigator of lazy signals, add ``navigator`` setter to lazy signals (`#2631 <https://github.com/hyperspy/hyperspy/pull/2631>`_)
+* Use 'dask_auto' when rechunk=True in ``change_dtype`` for lazy signal (`#2645 <https://github.com/hyperspy/hyperspy/pull/2645>`_)
+* Use dask chunking when saving lazy signal instead of rechunking and leave the user to decide what is the suitable chunking (`#2629 <https://github.com/hyperspy/hyperspy/pull/2629>`_)
+* Fix incorrect chunksize when saving EMD NCEM file and specifying chunks (`#2629 <https://github.com/hyperspy/hyperspy/pull/2629>`_)
+* Fix ``find_peaks`` GUIs call with laplacian/difference of gaussian methods (`#2622 <https://github.com/hyperspy/hyperspy/issues/2622>`_ and `#2647 <https://github.com/hyperspy/hyperspy/pull/2647>`_)
+* Added lazy reading support for FFT and DPC datasets in FEI emd datasets (`#2651 <https://github.com/hyperspy/hyperspy/pull/2651>`_).
+* Fix various future and deprecation warnings from numpy and scikit-learn (`#2646 <https://github.com/hyperspy/hyperspy/pull/2646>`_)
 * Improve error message when initialising SpanROI with left >= right (`#2604 <https://github.com/hyperspy/hyperspy/pull/2604>`_)
 
 

diff --git a/doc/user_guide/big_data.rst b/doc/user_guide/big_data.rst
@@ -153,6 +153,118 @@ offers the following online algorithms:
   :py:meth:`~.learn.mva.MVA.decomposition` for more details on decomposition
   with non-lazy signals.
 
+
+Navigator plot
+--------------
+
+The default signal navigator is the sum of the signal across all signal
+dimensions and all but 1 or 2 navigation dimensions. If the dataset is large,
+this can take a significant amount of time to perform with every plot.
+By default, a navigator is computed with minimally required approach to obtain
+a good signal-to-noise ratio image: the sum is taken on a single chunk of the
+signal space, in order to avoid to compute the navigator for the whole dataset.
+In the following example, the signal space is divided in 25 chunks (5 along on
+each axis), and therefore computing the navigation will only be perfomed over
+a small subset of the whole dataset by taking the sum on only 1 chunk out
+of 25:
+
+.. code-block:: python
+
+    >>> import dask.array as da
+    >>> import hyperspy.api as hs
+    >>> data = da.random.random((100, 100, 1000, 1000), chunks=('auto', 'auto', 200, 200))
+    >>> s = hs.signals.Signal2D(data).as_lazy()
+    >>> s.plot()
+
+In the example above, the calculation of the navigation is fast but the actual
+visualisation of the dataset is slow, each for each navigation index change,
+25 chunks of the dataset needs to be fetched from the harddrive. In the
+following example, the signal space contains a single chunk (instead of 25, in
+the previous example) and the calculating the navigator will then be slower (~20x)
+because the whole dataset will need to processed, however in this case, the
+visualisation will be faster, because only a single chunk will fetched from the
+harddrive when changing navigation indices:
+
+.. code-block:: python
+
+    >>> data = da.random.random((100, 100, 1000, 1000), chunks=('auto', 'auto', 1000, 1000))
+    >>> s = hs.signals.Signal2D(data).as_lazy()
+    >>> s.plot()
+
+This approach depends heavily on the chunking of the data and may not be
+always suitable. The :py:meth:`~hyperspy._signals.lazy.LazySignal.compute_navigator`
+can be used to calculate the navigator efficient and store the navigator, so
+that it can be used when plotting and saved for the later loading of the dataset.
+The :py:meth:`~hyperspy._signals.lazy.LazySignal.compute_navigator` has optional
+argument to specify the index where the sum needs to be calculated and how to
+rechunk the dataset when calculating the navigator. This allows to
+efficiently calculate the navigator without changing the actual chunking of the
+dataset, since the rechunking only takes during the computation of the navigator:
+
+.. code-block:: python
+
+    >>> data = da.random.random((100, 100, 1000, 1000), chunks=('auto', 'auto', 100, 100))
+    >>> s = hs.signals.Signal2D(data).as_lazy()
+    >>> s.compute_navigator(chunks_number=5)
+    >>> s.plot()
+
+.. code-block:: python
+
+    >>> data = da.random.random((100, 100, 2000, 400), chunks=('auto', 'auto', 100, 100))
+    >>> s = hs.signals.Signal2D(data).as_lazy()
+    >>> s
+    <LazySignal2D, title: , dimensions: (100, 100|400, 2000)>
+    >>> s.compute_navigator(chunks_number=(2, 10))
+    >>> s.plot()
+    >>> s.navigator.original_metadata
+    └── sum_from = [slice(200, 400, None), slice(1000, 1200, None)]
+
+The index can also be specified following the
+:ref:`HyperSpy indexing signal1D <signal.indexing>` syntax for float and
+interger.
+
+.. code-block:: python
+
+    >>> data = da.random.random((100, 100, 2000, 400), chunks=('auto', 'auto', 100, 100))
+    >>> s = hs.signals.Signal2D(data).as_lazy()
+    >>> s
+    <LazySignal2D, title: , dimensions: (100, 100|400, 2000)>
+    >>> s.compute_navigator(index=0, chunks_number=(2, 10))
+    >>> s.navigator.original_metadata
+    └── sum_from = [slice(0, 200, None), slice(0, 200, None)]
+
+An alternative is to calculate the navigator separately and store it in the
+signal using the :py:attr:`~hyperspy._signals.lazy.LazySignal.navigator` setter.
+
+
+.. code-block:: python
+
+    >>> data = da.random.random((100, 100, 1000, 1000), chunks=('auto', 'auto', 100, 100))
+    >>> s = hs.signals.Signal2D(data).as_lazy()
+    >>> s
+    <LazySignal2D, title: , dimensions: (100, 100|1000, 1000)>
+    >>> # for fastest results, just pick one signal space pixel
+    >>> nav = s.isig[500, 500]
+    >>> # Alternatively, sum as per default behaviour of non-lazy signal
+    >>> nav = s.sum(s.axes_manager.signal_axes)
+    >>> nav
+    <LazySignal2D, title: , dimensions: (|100, 100)>
+    >>> # Compute the result
+    >>> nav.compute()
+    [########################################] | 100% Completed | 13.1s
+    >>> s.navigator = nav
+    >>> s.plot()
+
+Alternatively, it is possible to not have a navigator, and use sliders
+instead:
+
+.. code-block:: python
+
+    >>> s
+    <LazySignal2D, title: , dimensions: (200, 200|512, 512)>
+    >>> s.plot(navigator='slider')
+
+
 Practical tips
 --------------
 
@@ -162,7 +274,7 @@ performed lazily. Important points are:
 Chunking
 ^^^^^^^^
 
-Data saved in the HDF5 format is typically divided into smaller chunks which can be loaded separately into memory, 
+Data saved in the HDF5 format is typically divided into smaller chunks which can be loaded separately into memory,
 allowing lazy loading. Chunk size can dramatically affect the speed of various HyperSpy algorithms, so chunk size is
 worth careful consideration when saving a signal. HyperSpy's default chunking sizes are probably not optimal
 for a given data analysis technique. For more comprehensible documentation on chunking,
@@ -179,11 +291,11 @@ The following example shows how to chunk one of the two navigation dimensions in
     >>> data = da.random.random((10,200,300))
     >>> data.chunksize
     (10, 200, 300)
-    
+
     >>> s = hs.signals.Signal1D(data)
     >>> s # Note the reversed order of navigation dimensions
     <Signal1D, title: , dimensions: (200, 10|300)>
-    
+
     >>> s.save('chunked_signal.hspy', chunks=(10, 100, 300)) # Chunking first hyperspy dimension (second array dimension)
     >>> s2 = hs.load('chunked_signal.hspy', lazy=True)
     >>> s2.data.chunksize
@@ -215,40 +327,6 @@ signal) it has a :py:meth:`~._signals.lazy.LazySignal.compute` method:
     <Signal2D, title: , dimensions: (|512, 512)>
 
 
-Navigator plot
-^^^^^^^^^^^^^^
-
-The default signal navigator is the sum of the signal across all signal
-dimensions and all but 1 or 2 navigation dimensions. If the dataset is large,
-this can take a significant amount of time to perform with every plot. A more
-convenient alternative is to calculate the summed navigation signal manually
-once, and only pass it for all other plots. Pay attention to the transpose
-(``.T``):
-
-.. code-block:: python
-
-    >>> s
-    <LazySignal2D, title: , dimensions: (200, 200|512, 512)>
-    >>> # for fastest results, just pick one signal space pixel
-    >>> nav = s.transpose(optimize=True).inav[256, 256]
-    >>> # Alternatively, sum as per default behaviour
-    >>> nav = s.sum(s.axes_manager.signal_axes).T
-    >>> nav
-    <LazySignal2D, title: , dimensions: (|200, 200)>
-    >>> # Compute the result
-    >>> nav.compute()
-    [########################################] | 100% Completed | 13.1s
-    >>> s.plot(navigator=nav)
-
-Alternatively, it is possible to not have a navigator, and use sliders
-instead:
-
-.. code-block:: python
-
-    >>> s
-    <LazySignal2D, title: , dimensions: (200, 200|512, 512)>
-    >>> s.plot(navigator='slider')
-
 Lazy operations that affect the axes
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 

diff --git a/doc/user_guide/install.rst b/doc/user_guide/install.rst
@@ -33,7 +33,8 @@ read the :ref:`conda installation <anaconda-install>` and
 HyperSpy Bundle
 ---------------
 
-The HyperSpy bundle is very similar to the Anaconda distribution, and it includes:
+The `HyperSpy <https://github.com/hyperspy/hyperspy-bundle>`_ bundle is very similar
+to the Anaconda distribution, and it includes:
 
   * HyperSpy
   * HyperSpyUI
@@ -48,15 +49,16 @@ Portable distribution (Windows only)
 A portable version of the `HyperSpy bundle <https://github.com/hyperspy/hyperspy-bundle>`_
 based on the WinPython distribution is also available on Windows.
 
-
 .. _anaconda-install:
 
-Installation in an Anaconda/Miniconda distribution
---------------------------------------------------
+Installation using conda
+------------------------
 
-HyperSpy is packaged in the `conda-forge <https://conda-forge.org/>`_ channel
-and can be installed easily using the `conda <https://docs.conda.io/en/latest/>`_
-package manager.
+`Conda <https://docs.conda.io/en/latest/>`_ is a package manager for Anaconda-like
+distributions, such as the `Miniforge <https://github.com/conda-forge/miniforge>`_
+or the `HyperSpy-bundle <https://github.com/hyperspy/hyperspy-bundle>`_.
+Since HyperSpy is packaged in the `conda-forge <https://conda-forge.org/>`_ channel,
+it can easily be installed using conda.
 
 To install hyperspy run the following from the Anaconda Prompt on Windows or
 from a Terminal on Linux and Mac.
@@ -132,6 +134,7 @@ To learn more about the Anaconda eco-system:
 
 - Choose between `Anaconda or Miniconda <https://docs.conda.io/projects/conda/en/latest/user-guide/install/download.html#anaconda-or-miniconda>`_?
 - Understanding `conda and pip <https://www.anaconda.com/blog/understanding-conda-and-pip>`_.
+- What is `conda-forge <https://conda-forge.org/#about>`_.
 
 .. _install-with-pip:
 
@@ -145,16 +148,16 @@ install pip for the following commands to run.
 
 To install all hyperspy functionalities, run:
 
-.. code-block:: bash
+  .. code-block:: bash
 
-    $ pip install hyperspy[all]
+      $ pip install hyperspy[all]
 
 To install only the strictly required dependencies and limited functionalities,
 use:
 
-.. code-block:: bash
+  .. code-block:: bash
 
-    $ pip install hyperspy
+      $ pip install hyperspy
 
 See the following list of selectors to select the installation of optional
 dependencies required by specific functionalities:
@@ -184,6 +187,51 @@ tools installed in the system. If the above does not work for you remember that
 the easiest way to install HyperSpy is
 :ref:`using the HyperSpy bundle <hyperspy-bundle>`.
 
+.. _update-with-conda:
+
+Update HyperSpy
+---------------
+
+Using conda
+^^^^^^^^^^^
+
+To update hyperspy to the latest release using conda:
+
+   .. code-block:: bash
+
+       $ conda update hyperspy -c conda-forge
+
+Using pip
+^^^^^^^^^
+
+To update hyperspy to the latest release using pip:
+
+   .. code-block:: bash
+
+       $ pip install hyperspy --upgrade
+
+Install specific version
+------------------------
+
+Using conda
+^^^^^^^^^^^
+
+To install a specific version of hyperspy (for example ``1.6.1``) using conda:
+
+   .. code-block:: bash
+
+       $ conda install hyperspy=1.6.1 -c conda-forge
+
+Using pip
+^^^^^^^^^
+
+To install a specific version of hyperspy (for example ``1.6.1``) using pip:
+
+   .. code-block:: bash
+
+       $ pip install hyperspy==1.6.1
+
+
 .. _install-dev:
 
 Rolling release Linux distributions
@@ -196,11 +244,11 @@ or :ref:`Pip <install-with-pip>` is recommended.
 
 However, packages of the latest HyperSpy release and the related
 GUI packages are maintained for the rolling release distributions 
-**Arch-Linux** (in the `Arch User Repository 
+*Arch-Linux* (in the `Arch User Repository 
 <https://aur.archlinux.org/packages/python-hyperspy/>`_) (AUR) and 
-**openSUSE** (`Community Package <https://software.opensuse.org/package/python-hyperspy>`_)
-as ``python-hyperspy`` and ``python-hyperspy-gui-traitsui`` /
-``python-hyperspy-gui-ipywidgets``.
+*openSUSE* (`Community Package <https://software.opensuse.org/package/python-hyperspy>`_)
+as ``python-hyperspy`` and ``python-hyperspy-gui-traitsui``,
+``python-hyperspy-gui-ipywidgets`` for the GUIs packages.
 
 A more up-to-date package that contains all updates to be included
 in the next minor version release (likely including new features compared to