Merge branch 'master' into fix/spectral-gap

.gitignore

@@ -304,3 +304,4 @@ tags
 MANIFEST
 travis_wait_*.log
 testing.py
+heat/datasets/MNISTDataset

CHANGELOG.md

@@ -1,8 +1,9 @@
 # Pending additions
-- [#717] (https://github.com/helmholtz-analytics/heat/pull/717) Switch CPU CI over to Jenkins and pre-commit to GitHub action.
-- [#720] (https://github.com/helmholtz-analytics/heat/pull/720) Ignore test files in codecov report and allow drops in code coverage.
+- [#717](https://github.com/helmholtz-analytics/heat/pull/717) Switch CPU CI over to Jenkins and pre-commit to GitHub action.
+- [#720](https://github.com/helmholtz-analytics/heat/pull/720) Ignore test files in codecov report and allow drops in code coverage.
 
 ## New features
+- [#660](https://github.com/helmholtz-analytics/heat/pull/660) NN module for data parallel neural networks
 - [#680](https://github.com/helmholtz-analytics/heat/pull/680) New property: larray
 - [#683](https://github.com/helmholtz-analytics/heat/pull/683) New properties: nbytes, gnbytes, lnbytes
 - [#687](https://github.com/helmholtz-analytics/heat/pull/687) New DNDarray property: balanced
@@ -20,12 +21,21 @@
 - [#679](https://github.com/helmholtz-analytics/heat/pull/679) New feature: ``histc()`` and ``histogram()``
 ### Linear Algebra
 - [#658](https://github.com/helmholtz-analytics/heat/pull/658) Bugfix: `matmul` on GPU will cast away from `int`s to `float`s for the operation and cast back upon its completion. This may result in numerical inaccuracies for very large `int64` DNDarrays
+### NN
+- [#660](https://github.com/helmholtz-analytics/heat/pull/660) New submodule: `nn.DataParallel` for creating and training data parallel neural networks
+- [#660](https://github.com/helmholtz-analytics/heat/pull/660) New feature: Synchronous and Asynchronous gradient updates availble for `ht.nn.DataParallel`
+- [#660](https://github.com/helmholtz-analytics/heat/pull/660) New feature: `utils.data.datatools.DataLoader` for created a local `torch.utils.data.Dataloader` for use with `ht.nn.DataParallel`
+- [#660](https://github.com/helmholtz-analytics/heat/pull/660) New feature: `utils.data.datatools.Dataset` for created a local `torch.utils.data.Dataset` for use with `ht.nn.DataParallel`
+- [#660](https://github.com/helmholtz-analytics/heat/pull/660) Added MNIST example to `example/nn` to show the use of `ht.nn.DataParallel`. The `MNISTDataset` can be found in `ht.utils.data.mnist.py`
+- [#660](https://github.com/helmholtz-analytics/heat/pull/660) New feature: Data loader for H5 datasets which shuffles data in the background during training (`utils.data.partial_dataset.PartialH5Dataset`)
+### Logical
+- [#711](https://github.com/helmholtz-analytics/heat/pull/711) `isfinite()`, `isinf()`, `isnan()`
 
-### ...
 
 ## Bug fixes
 - [#709](https://github.com/helmholtz-analytics/heat/pull/709) Set the encoding for README.md in setup.py explicitly.
 - [#716](https://github.com/helmholtz-analytics/heat/pull/716) Bugfix: Finding clusters by spectral gap fails when multiple diffs identical
+- [#735](https://github.com/helmholtz-analytics/heat/pull/735) Set return type to bool in relational functions.
 
 # v0.5.2
 

Jenkinsfile

@@ -7,8 +7,8 @@ pipeline {
         stage ('Build') {
             steps {
                 withPythonEnv('/home/jenkins/allvenvs/') {
-	                sh 'pip3 install .[hdf5,netcdf] --no-cache-dir'
-                    sh 'pip3 install pytest coverage pre-commit --no-cache-dir'
+	                sh 'pip3 install .[hdf5,netcdf] --no-cache-dir --upgrade --upgrade-strategy eager'
+                    sh 'pip3 install pytest coverage pre-commit --no-cache-dir --upgrade'
                     sh 'pre-commit run --all-files'
                 }
             }
@@ -20,6 +20,10 @@ pipeline {
                     sh 'COVERAGE_FILE=report/cov/coverage2 mpirun -n 2 coverage run --source=heat --parallel-mode -m pytest --junitxml=report/test/report2.xml heat/'
                     sh 'COVERAGE_FILE=report/cov/coverage3 mpirun -n 3 coverage run --source=heat --parallel-mode -m pytest --junitxml=report/test/report3.xml heat/'
                     sh 'COVERAGE_FILE=report/cov/coverage4 mpirun -n 4 coverage run --source=heat --parallel-mode -m pytest --junitxml=report/test/report4.xml heat/'
+                    sh 'COVERAGE_FILE=report/cov/coverage5 mpirun -n 5 coverage run --source=heat --parallel-mode -m pytest --junitxml=report/test/report5.xml heat/'
+                    sh 'COVERAGE_FILE=report/cov/coverage6 mpirun -n 6 coverage run --source=heat --parallel-mode -m pytest --junitxml=report/test/report6.xml heat/'
+                    sh 'COVERAGE_FILE=report/cov/coverage7 mpirun -n 7 coverage run --source=heat --parallel-mode -m pytest --junitxml=report/test/report7.xml heat/'
+                    sh 'COVERAGE_FILE=report/cov/coverage8 mpirun -n 8 coverage run --source=heat --parallel-mode -m pytest --junitxml=report/test/report8.xml heat/'
                 }
             }
         }

README.md

@@ -12,6 +12,7 @@ Project Status
 [![Build Status](https://travis-ci.com/helmholtz-analytics/heat.svg?branch=master)](https://travis-ci.com/helmholtz-analytics/heat)
 [![Documentation Status](https://readthedocs.org/projects/heat/badge/?version=latest)](https://heat.readthedocs.io/en/latest/?badge=latest)
 [![codecov](https://codecov.io/gh/helmholtz-analytics/heat/branch/master/graph/badge.svg)](https://codecov.io/gh/helmholtz-analytics/heat)
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
 [![license: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
 [![Downloads](https://pepy.tech/badge/heat)](https://pepy.tech/project/heat)
 [![Mattermost Chat](https://img.shields.io/badge/chat-on%20mattermost-blue.svg)](https://mattermost-haf.fz-juelich.de/signup_user_complete/?id=iqrr6pmxb38fzqffa51qqhcu8w)

codecov.yml

@@ -2,6 +2,8 @@ ignore:
   - "heat/core/regression/lasso/demo.py"
   - "heat/core/regression/lasso/plotfkt.py"
   - "heat/examples/*"
+  - "heat/utils/data/mnist.py"
+  - "heat/utils/data/_utils.py"
   - "heat/**/test_*.py"
 
 coverage:
@@ -10,10 +12,11 @@ coverage:
       default:
         # basic
         target: auto
-        threshold: 5%
+        threshold: 3%
         base: auto
         flags:
           - unit
+          - gpu
         paths:
           - "heat"
        # advanced settings
@@ -22,3 +25,19 @@ coverage:
         if_ci_failed: error #success, failure, error, ignore
         informational: false
         only_pulls: false
+    patch:
+      default:
+        # basic
+        target: auto
+        threshold: 3%
+        base: auto
+        # advanced
+        branches:
+          - master
+        if_ci_failed: error #success, failure, error, ignore
+        only_pulls: false
+        flags:
+          - "unit"
+          - "gpu"
+        paths:
+          - "heat"