Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Model/format validation utilities + tests + npz fixes #7701

Merged
merged 17 commits into from May 9, 2019

Conversation

@AlexDBlack
Copy link
Contributor

commented May 9, 2019

Fixes: #7503

Adds numerous validation utils for checking if most common serialized formats are valid:

  • DL4JModelValidator
  • DL4JKerasModelValidator
  • Nd4jValidator
  • Nd4jCommonValidator

In most (but not all) cases, it should be cheaper/faster to use these validation tools to detect invalid formats than simply trying to load the format to see if it works or not.

Validation tools have common return format - ValidationResult object with the following information:

  • formatType: Human readable format/model type
  • formatClass; Actual class the format/model is (or should be)
  • path: Path of file (if applicable)
  • valid: Whether the file/model is valid
  • issues: List of issues/problems making format invalid (generally only present if not valid)
  • exception: Exception, if applicable

Also fixes and better tests for npz file reading.

Eventually we should integrate these better into actual loading, but we'll look at that separately

@AlexDBlack AlexDBlack merged commit 209412b into dev_20190506 May 9, 2019

@AlexDBlack AlexDBlack deleted the ab_ux branch May 9, 2019

AlexDBlack added a commit that referenced this pull request May 13, 2019
[WIP] Dev branch merge (#7718)
* few pydl4j fixes (#7634)

* few fixes

* rem ,

* py3 fix

* Delete build.py

* Replace DL4J UI theme; remove external dependencies (#7683)

* First steps for UI replacement

* round 2

* Round 3

* Overview page done

* Move final dependencies for overview page to webjars

* Model page done

* Clean up old resources

* Final cleanup

* Fix tests

* Final fixes

* Model/format validation utilities + tests + npz fixes (#7701)

* First steps for model/file format validation

* Next steps

* First steps on DL4J model validation

* Next

* More validation utils

* File + zip validation tests

* MultiLayerNetwork validation tests

* More model validation

* Compgraph validation tests

* Numpy format validation

* Fix Numpy NPZ reading + add tests

* NPZ validation

* Numpy text format reading improvements

* Numpy text format and SameDiff FlatBuffers validation

* Javadoc and text validation

* Javadoc

* DL4J Keras import model validator

* Numpy .npy scalar tests (#7708)

* [WIP] dev npy0d (#7712)

* npy 0D fix

* Shyrma pad (#7713)

* add pad test failing in java

* fix pad helper

* minor changes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.