Merge 1f296a6 into 587ad2a

.github/CONTRIBUTING.md

@@ -0,0 +1,16 @@
+## Discussions
+
+For discussions on the use of hyperspy, please use the [mailing list](http://groups.google.com/group/hyperspy-users): useful to discuss use cases of HyperSpy and ask about specific applications.
+
+Discussion on the [gitter chat](https://gitter.im/hyperspy/hyperspy): useful to ask quick questions.
+
+## Issues
+
+The [issue tracker](https://github.com/hyperspy/hyperspy/issues) can be used to report bugs or propose new features. When reporting a bug, the following is useful:
+- give a minimal example demonstrating the bug,
+- copy and paste the error traceback.
+
+## Contribute
+
+If you want to contribute to the HyperSpy source code, you can send us a [pull requests](https://github.com/hyperspy/hyperspy/pulls). For more information, please read the [developer guide](http://hyperspy.org/hyperspy-doc/current/dev_guide.html).
+

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,29 @@
+### Requirements
+* Read the [developer guide](http://hyperspy.org/hyperspy-doc/current/dev_guide.html).
+* Filling out the template; it helps the review process and it is useful to summarise the PR.
+* This template can be updated during the progression of the PR to summarise its status. 
+
+*You can delete this section after you read it.*
+
+### Description of the change
+A few sentences and/or a bulleted list to describe and motivate the change:
+- Change A.
+- Change B.
+- etc.
+
+### Progress of the PR
+- [ ] Change implemented (can be split into several points),
+- [ ] update docstring (if appropriate),
+- [ ] update user guide (if appropriate),
+- [ ] add tests,
+- [ ] ready for review.
+
+### Minimal example of the bug fix or the new feature
+```python
+>>> import hyperspy.api as hs
+>>> import numpy as np
+>>> s = hs.signals.Signal1D(np.arange(10))
+>>> # Your new feature...
+```
+Note that this example can be useful to update the user guide.
+

.travis.yml

@@ -57,6 +57,16 @@ script:
   - pip install coverage coveralls pytest pytest-cov pytest-mpl;
     py.test --mpl --cov=hyperspy --pyargs hyperspy;
 
+#after_failure: # run only on failure in case there is a need to check matplotlib image comparison
+# This needs a service to upload the artifacts (and corresponding configuration)
+# See:
+# https://blog.travis-ci.com/2012-12-18-travis-artifacts/
+# https://docs.travis-ci.com/user/uploading-artifacts/
+# https://github.com/matplotlib/matplotlib/blob/master/.travis.yml
+#  - gem install travis-artifacts
+#  - zip -r image_comparison_failure.zip /tmp/ -i *.png
+#  - travis-artifacts upload image_comparison_failure.zip
+
 after_success:
 - coveralls
 - if [[ $MINIMAL_ENV == 'False' ]]; then

README.rst

@@ -8,8 +8,8 @@
 .. |AppVeyor| image:: https://ci.appveyor.com/api/projects/status/github/hyperspy/hyperspy?svg=true&branch=RELEASE_next_patch
 .. _AppVeyor: https://ci.appveyor.com/project/hyperspy/hyperspy/branch/RELEASE_next_patch
 
-.. |Coveralls| image:: https://coveralls.io/repos/github/hyperspy/hyperspy/badge.svg?branch=RELEASE_next_patch
-.. _Coveralls: https://coveralls.io/github/hyperspy/hyperspy?branch=RELEASE_next_patch
+.. |Coveralls| image:: https://coveralls.io/repos/github/hyperspy/hyperspy/badge.svg?branch=RELEASE_next_minor
+.. _Coveralls: https://coveralls.io/github/hyperspy/hyperspy?branch=RELEASE_next_minor
 
 .. |pypi_version| image:: http://img.shields.io/pypi/v/hyperspy.svg?style=flat
 .. _pypi_version: https://pypi.python.org/pypi/hyperspy
@@ -44,6 +44,13 @@ HyperSpy is released under the GPL v3 license.
     **Since version 0.8.4 HyperSpy only supports Python 3. If you need to install
     HyperSpy in Python 2.7 install HyperSpy 0.8.3.**
 
+
+Contributing
+------------
+
+Everyone is welcome to contribute. Please read our
+`contributing guidelines <https://github.com/hyperspy/hyperspy/blob/RELEASE_next_minor/.github/CONTRIBUTING.md>`_ and get started!
+
 Cite
 ----
 

appveyor.yml

@@ -87,6 +87,11 @@ artifacts:
   - path: dist\*.whl
     name: win_wheels
 
+on_failure:
+  - ps: $Env:image_comparison_filename = 'image_comparison_' + $Env:PYTHON_VERSION + '_' + $Env:PYTHON_ARCH + 'bits.zip'
+  - ps: 7z a -r $Env:image_comparison_filename C:\Users\appveyor\AppData\Local\Temp\1\*png
+  - ps: Push-AppveyorArtifact $Env:image_comparison_filename
+
 deploy:
   provider: GitHub
   auth_token:

doc/api/hyperspy.datasets.rst

@@ -13,6 +13,15 @@ hyperspy.datasets.example\_signals module
     :show-inheritance:
 
 
+hyperspy\.datasets\.artificial\_data module
+-------------------------------------------
+
+.. automodule:: hyperspy.datasets.artificial_data
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+
 Module contents
 ---------------
 

doc/api/hyperspy.io_plugins.rst

@@ -84,8 +84,16 @@ hyperspy.io\_plugins.mrc module
     :undoc-members:
     :show-inheritance:
 
-hyperspy.io\_plugins.msa module
--------------------------------
+hyperspy\.io\_plugins\.mrcz module
+---------------------------------
+
+.. automodule:: hyperspy.io_plugins.mrcz
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+hyperspy\.io\_plugins\.msa module
+---------------------------------
 
 .. automodule:: hyperspy.io_plugins.msa
     :members:

doc/conf.py

@@ -232,7 +232,7 @@
 
 # Add the hyperspy website to the intersphinx domains
 intersphinx_mapping = {'hyperspyweb': ('http://hyperspy.org/', None),
-                       'matplotlib':  ('https://matplotlib.org', None)}
+                       'matplotlib': ('https://matplotlib.org', None)}
 
 
 def setup(app):

doc/dev_guide.rst

@@ -206,6 +206,9 @@ Useful hints on testing:
   your PR. There should be a link to it at the bottom of your PR on the github
   PR page. This service can help you to find how well your code is being tested
   and exactly which part is not currently tested.
+* `pytest-sugar <https://pypi.python.org/pypi/pytest-sugar>`_ can be installed
+  to have a nicer look and feel of py.test in the console (encoding issue have
+  been reported in the Windows console).
 
 
 .. _plot-test-label:
@@ -241,6 +244,9 @@ after execution. It is necessary to delete the figure and this can be done
 by using mpl_cleanup fixture: you juste need to add ``mpl_cleanup``
 as the first argument of the tests.
 
+When the plotting tests are failling, it is possible to download the figure comparison images generated by pytest-mpl in the `artifacts tabs <https://ci.appveyor.com/project/hyperspy/hyperspy/build/1.0.2500/job/2c2qccaktd90po2q/artifacts>`_ of the corresponding build.
+
+
 The plotting tests need matplotlib > 2.0.0, since the matplotlib `style change
 <http://matplotlib.org/style_changes.html>`_ between matplotlib 1.x and 2.x is
 enough to make the test failed. Freetype>=2.8 is also required to pass the 
@@ -269,6 +275,10 @@ specification <https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT
 User-guide Documentation -- A description of the functionality of the code and
 how to use it with examples and links to the relevant code.
 
+When writing both the docstrings and user guide documentation, it is useful to
+have some kind of data which the users can use themselves. Artificial
+datasets for this purpose can be found in `hyperspy.datasets.artificial_data`.
+
 Build the documentation -- To check the output of what you wrote, you can build
 the documentation, which requires python-sphinx and numpydoc to be installed.
 Run the make command in the ``doc`` to build the documentation. For example

doc/sphinxext/ipython_console_highlighting.py

@@ -111,6 +111,7 @@ def setup(app):
     # suppresses the sphinx warning we'd get without it.
     pass
 
+
 #-----------------------------------------------------------------------------
 # Register the extension as a valid pygments lexer
 highlighting.lexers['ipython'] = IPythonConsoleLexer()

doc/user_guide/big_data.rst

@@ -67,6 +67,22 @@ around 35GB of memory. To store it in a floating-point format one would need
 almost 280GB of memory. However, with the lazy processing both of these steps
 are near-instantaneous and require very little computational resources.
 
+.. versionadded:: 1.4
+    :py:meth:`~._signals.lazy.LazySignal.close_file`
+
+Currently when loading an hdf5 file lazily the file remains open at
+least while the signal exists. In order to close it explicitly, use the
+:py:meth:`~._signals.lazy.LazySignal.close_file` method. Alternatively,
+you could close it on calling :py:meth:`~._signals.lazy.LazySignal.compute`
+by passing the keyword argument ``close_file=True`` e.g.:
+
+.. code-block:: python
+
+    >>> s = hs.load("file.hspy", lazy=True)
+    >>> ssum = s.sum(axis=0)
+    >>> ssum.compute(close_file=True) # closes the file.hspy file
+
+
 Lazy stacking
 ^^^^^^^^^^^^^
 

doc/user_guide/electron_holography.rst

@@ -133,3 +133,44 @@ assignment by
 :py:meth:`~._signals.hologram_image.HologramImage.estimate_sideband_position`
 and :py:meth:`~._signals.hologram_image.HologramImage.estimate_sideband_size`
 methods. This, however, is not recommended for not experienced users.
+
+
+Getting hologram statistics
+--------------------------
+There are many reasons to have an access to some parameters of holograms which describe the quality of the data.
+:meth:`~._signals.hologram_image.HologramImage.statistics` can be used to calculate carrier frequency,
+fringe spacing and estimate fringe contrast. The method outputs dictionary with the values listed above calculated also
+in different units. In particular fringe spacing is calculated in pixels (fringe sampling) as well as in
+calibrated units. Carrier frequency is calculated in inverse pixels or calibrated units as well as radians.
+Estimation of fringe contrast is either performed by division of standard deviation by mean value of hologram or
+in Fourier space as twice the fraction of amplitude of sideband centre and amplitude of center band (i.e. FFT origin).
+The first method is default and using it requires the fringe field to cover entire field of view; the method is
+highly sensitive to any artifacts in holograms like dud pixels,
+fresnel fringes and etc. The second method is less sensitive to the artifacts listed above and gives
+reasonable estimation of fringe contrast even if the hologram is not covering entire field of view, but it is highly
+sensitive to precise calculation of sideband position and therefore sometimes may underestimate the contrast.
+The selection between to algorithms can be done using parameter ``fringe_contrast_algorithm`` setting it to
+``'statistical'`` or to ``'fourier'``. The side band position typically provided by a ``sb_position``.
+The statistics can be accessed as follows:
+
+.. code-block:: python
+
+    >>> statistics = im.statistics(sb_position=sb_position)
+
+Note that by default the ``single_value`` parameter is ``True`` which forces the output of single values for each
+entry of statistics dictionary calculated from first navigation pixel. (I.e. for image stacks only first image
+will be used for calculating the statistics.) Otherwise:
+
+.. code-block:: python
+
+    >>> statistics = im.statistics(sb_position=sb_position, single_value=False)
+
+Entries of ``statistics`` are Hyperspy signals containing the hologram parameters for each image in a stack.
+
+The estimation of fringe spacing using ``'fourier'`` method applies apodization in real space prior calculating FFT.
+By default ``apodization`` parameter is set to ``hanning`` which applies Hanning window. Other options are using either
+``None`` or ``hamming`` for no apodization or Hamming window. Please note that for experimental conditions
+especially with extreme sampling of fringes and strong contrast variation due to Fresnel effects
+the calculated fringe contrast provides only an estimate and the values may differ strongly depending on apodization.
+
+For further information see documentation of :meth:`~._signals.hologram_image.HologramImage.statistics`.

doc/user_guide/getting_started.rst

@@ -209,6 +209,17 @@ signals:
 
     >>> hs.datasets.example_signals.EDS_TEM_Spectrum().plot()
 
+.. versionadded:: 1.4
+    :py:mod:`~.datasets.artificial_data`
+
+There are also artificial datasets, which are made to resemble real
+experimental data.
+
+.. code-block:: python
+
+    >>> s = hs.datasets.artificial_data.get_core_loss_eels_signal()
+    >>> s.plot()
+
 .. _eelsdb-label:
 
 .. versionadded:: 1.0
@@ -409,6 +420,56 @@ navigation dimensions:
 
     >>> s.axes_manager.indices = (5, 4)
 
+.. _quantity_and_converting_units:
+
+Using quantity and converting units
+-------------------------------------------
+
+The scale and the offset of each axis can be set and retrieved as quantity.
+
+.. code-block:: python
+
+    >>> s = hs.signals.Signal1D(np.arange(10))
+    >>> s.axes_manager[0].scale_as_quantity
+    1.0 dimensionless
+    >>> s.axes_manager[0].scale_as_quantity = '2.5 µm'
+    >>> s.axes_manager
+    <Axes manager, axes: (|10)>
+                Name |   size |  index |  offset |   scale |  units 
+    ================ | ====== | ====== | ======= | ======= | ====== 
+    ---------------- | ------ | ------ | ------- | ------- | ------ 
+         <undefined> |     10 |        |       0 |     2.5 |     µm
+    >>> s.axes_manager[0].offset_as_quantity = '2.5 nm'
+    <Axes manager, axes: (|10)>
+                Name |   size |  index |  offset |   scale |  units 
+    ================ | ====== | ====== | ======= | ======= | ====== 
+    ---------------- | ------ | ------ | ------- | ------- | ------ 
+         <undefined> |     10 |        |     2.5 | 2.5e+03 |     nm
+
+
+Internally, HyperSpy uses the `pint <http://pint.readthedocs.io>`_ library to manage the scale and offset quantities. The ``scale_as_quantity`` and ``offset_as_quantity`` attributes return pint object:
+
+.. code-block:: python
+
+    >>> q = s.axes_manager[0].offset_as_quantity
+    >>> type(q) # q is a pint quantity object
+    pint.quantity.build_quantity_class.<locals>.Quantity
+    >>> q
+    2.5 nanometer
+
+
+The ``convert_units`` method of the :py:class:`~.axes.AxesManager` converts units, which by default (no parameters provided) converts all axis units to an optimal units to avoid using too large or small number.
+
+Each axis can also be converted individually using the ``convert_to_units`` method of the :py:class:`~.axes.DataAxis`:
+
+.. code-block:: python
+
+    >>> axis = hs.hyperspy.axes.DataAxis(size=10, scale=0.1, offset=10, units='mm')
+    >>> axis.scale_as_quantity
+    0.1 millimeter
+    >>> axis.convert_to_units('µm')
+    >>> axis.scale_as_quantity
+    100.0 micrometer
 
 .. _saving:
 

doc/user_guide/install.rst

@@ -31,7 +31,7 @@ Bundle. This is a customised `WinPython <http://winpython.github.io/>`_
 distribution that includes HyperSpy, all its dependencies and many other
 scientific Python packages.
 
-For details and download links go to https://github.com/hyperspy/hyperspy-bundle 
+For details and download links go to https://github.com/hyperspy/hyperspy-bundle
 
 .. _quick-anaconda-install:
 
@@ -128,6 +128,9 @@ Alternatively you can select the extra functionalities required:
 * ``gui-traitsui`` to install required libraries to use the GUI elements based
   on `traitsui <http://docs.enthought.com/traitsui/>`_
 * ``test`` to install required libraries to run HyperSpy's unit tests.
+* ``lazy_FEI_EMD`` to install required libraries to load FEI spectrum imgages
+  lazily.
+* ``mrcz-blosc`` to install the blosc library to use compression with the mrcz plugin.
 * ``doc`` to install required libraries to build HyperSpy's documentation.
 
 For example:
@@ -254,6 +257,15 @@ is going to be installed from  source, Cython is also required. Also, to
 compile the documentation sphinxcontrib-napoleon and sphinx_rtd_theme are
 required.
 
+In case some of the required libraries are not automatically installed when
+installing from source in a conda environment, these can be obtained beforehand
+by installing and removing hyperspy from that environment;
+
+.. code-block:: bash
+    $ conda install hyperspy
+    $ conda remove hyperspy
+    $ sudo pip install -e ./
+
 .. _known-issues:
 
 Known issues