added gradient-enhanced neural nets (genn)

[shb84]

This pull request includes new files that implement gradient-enhanced neural nets using numpy. The original code was not modified, except for "test_surrogate_model_examples.py" where I added a test for GENN. The newly added files are as follows:

new file:   doc/_src_docs/surrogate_models/genn.rst -----> documentation
new file:   doc/_src_docs/surrogate_models/genn.rstx -----> documentation
new file:   smt/surrogate_models/genn.py -----> API interface
new file:   smt/surrogate_models/neural_net/* -----> neural net logic
modified:   smt/surrogate_models/tests/test_surrogate_model_examples.py -----> test

[AppVeyorBot]

❌ Build SMT 0.2.467 failed (commit 17ce5081a0 by @)

[AppVeyorBot]

❌ Build SMT 0.2.468 failed (commit 408e0ce07a by @)

[AppVeyorBot]

✅ Build SMT 0.2.469 completed (commit 99c0dfedf8 by @)

[coveralls]

Pull Request Test Coverage Report for Build 456

• 595 of 777 (76.58%) changed or added relevant lines in 9 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage decreased (-0.2%) to 77.37%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
smt/surrogate_models/genn.py	85	89	95.51%
smt/surrogate_models/neural_net/fwd_prop.py	46	51	90.2%
smt/surrogate_models/neural_net/activation.py	36	56	64.29%
smt/surrogate_models/neural_net/metrics.py	12	35	34.29%
smt/surrogate_models/neural_net/data.py	49	89	55.06%
smt/surrogate_models/neural_net/model.py	192	236	81.36%
smt/surrogate_models/neural_net/optimizer.py	106	152	69.74%

Totals
Change from base Build 455:	-0.2%
Covered Lines:	3118
Relevant Lines:	4030

---

💛 - Coveralls

[AppVeyorBot]

✅ Build SMT 0.2.472 completed (commit 294914f709 by @shb84)

[shb84]

In order to increase coverage, there exist additional tests within most of the neural_net files that could potentially be incorporated. These are implemented after the "if name == 'main'" statement. Please let me know how I can help increase coverage.

[relf]

Indeed you should exclude "main parts" from coverage and add tests under smt/tests and mimic what is done for other surrogates. You can run coverage in your dev environment to check for improvement.

[AppVeyorBot]

❌ Build SMT 0.2.473 failed (commit c773c0ffca by @)

[AppVeyorBot]

❌ Build SMT 0.2.474 failed (commit c8d357a956 by @)

[AppVeyorBot]

❌ Build SMT 0.2.475 failed (commit 328c90f0e7 by @shb84)

[AppVeyorBot]

✅ Build SMT 0.2.476 completed (commit b7ee7bd6d5 by @)

[shb84]

It looks like the reason the Travis CI build failed is not related to neural net additions, but has to do with "smt/tests/test_high_dim.py:Test.test_tanh_QP ... FAIL" instead. Please let me know if any additional corrections are needed on the neural net side.

[AppVeyorBot]

❌ Build SMT 0.2.487 failed (commit e1c3413bef by @)

[shb84]

Sorry for the late reply. This is the first time in 6 weeks that I've had a chance to work on the code. I sincerely appreciate the feedback. Thank you.

[AppVeyorBot]

❌ Build SMT 0.2.488 failed (commit 10d7a337ec by @shb84)

[AppVeyorBot]

✅ Build SMT 0.2.493 completed (commit 41d8012dda by @bouhlelma)