numpy data generators

[davidslater]

Signed-off-by: David Slater davidslater@users.noreply.github.com

Description

This PR adds a NumpyDataGenerator class that is backed by numpy arrays. This can be useful for avoiding OOM errors when using framework-specific model fitting.

Fixes #1926

Type of change

Please check all relevant options.

• Improvement (non-breaking)
• Bug fix (non-breaking)
• New feature (non-breaking)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Testing

Please describe the tests that you ran to verify your changes. Consider listing any relevant details of your test configuration.

• tests/test_data_generators.py::TestNumpyDataGenerator

Test Configuration:

• CI Test

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[codecov-commenter]

Codecov Report

Merging #1934 (0daa505) into dev_1.13.0 (caa606c) will increase coverage by 16.07%.
The diff coverage is 100.00%.

@@               Coverage Diff               @@
##           dev_1.13.0    #1934       +/-   ##
===============================================
+ Coverage       69.53%   85.60%   +16.07%     
===============================================
  Files             257      257               
  Lines           23890    23927       +37     
  Branches         4340     4346        +6     
===============================================
+ Hits            16611    20482     +3871     
+ Misses           6339     2335     -4004     
- Partials          940     1110      +170     

Impacted Files	Coverage Δ
art/data_generators.py	95.83% <100.00%> (+34.38%)	⬆️
art/estimators/classification/tensorflow.py	84.30% <0.00%> (+0.20%)	⬆️
art/estimators/classification/pytorch.py	86.38% <0.00%> (+0.46%)	⬆️
art/attacks/evasion/boundary.py	93.93% <0.00%> (+1.21%)	⬆️
art/summary_writer.py	84.48% <0.00%> (+1.72%)	⬆️
art/estimators/classification/blackbox.py	98.00% <0.00%> (+2.00%)	⬆️
art/estimators/tensorflow.py	79.79% <0.00%> (+2.02%)	⬆️
...hidden_trigger_backdoor/hidden_trigger_backdoor.py	92.30% <0.00%> (+3.07%)	⬆️
art/preprocessing/audio/l_filter/pytorch.py	100.00% <0.00%> (+4.08%)	⬆️
art/estimators/classification/lightgbm.py	95.00% <0.00%> (+6.66%)	⬆️
... and 70 more

[lgtm-com]

This pull request introduces 2 alerts when merging e6c64f2 into e7c763b - view on LGTM.com

new alerts:

• 2 for Module is imported more than once

---

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine ⚙️ that powers LGTM.com. For more information, please check out our post on the GitHub blog.

[lgtm-com]

This pull request introduces 2 alerts when merging 162dd27 into e7c763b - view on LGTM.com

new alerts:

• 2 for Module is imported more than once

---

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine ⚙️ that powers LGTM.com. For more information, please check out our post on the GitHub blog.

[lgtm-com]

This pull request introduces 2 alerts when merging 162dd27 into 6e82fe1 - view on LGTM.com

new alerts:

• 2 for Module is imported more than once

---

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine ⚙️ that powers LGTM.com. For more information, please check out our post on the GitHub blog.

[beat-buesser]

@davidslater Looks good to me! Thank you very much!