Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Version upgrades #199

Merged
merged 97 commits into from Aug 30, 2019

Conversation

@AlexDBlack
Copy link

commented Aug 30, 2019

Fixes: eclipse#6655
Fixes: eclipse#3808
Fixes: eclipse#8151
Fixes: eclipse#6269
Fixes: eclipse#5259 (in progress, for next release)

Upgrades Jackson from 2.5.1 to 2.9.9/2.9.9.3
Upgrades commons compress from 1.16.1 to 1.18
Upgrades Play framework from 2.4.8 to 2.7.3
Upgrades Spark 2 from 2.1.0 to 2.4.3 (doesn't matter in practice as it's compile scope usually anyway...)
Upgrades Guava from 20.0 to 28.0-jre and shades (to avoid dependency clashes - spark etc)
Drops support for Scala 2.10
Adds support for Scala 2.12
Drops support for Spark 1.x

Fixes bug in shading (Jackson + new modules) where non-shaded imports could be used at compile time but would fail at runtime with NoClassDefFoundError, unless dependency was added explicitly.

Most of the work here is to reimplement JSON backward compatibility handling for DL4J and DataVec. Unfortunately the old approach no longer worked.
Some work required for Play framework API changes (in routing).

One small breaking changes here:
DL4J custom layers in legacy 1.0.0-alpha or earlier format: no longer supported (not worth the maintenance overhead). Users can load in any of 1.0.0-beta to beta4 and re-save, before loading in beta5 or later.


Eclipse IP

Version upgrades:

AlexDBlack added 30 commits Aug 14, 2019
DataVec fixes for Jackson version upgrade
Signed-off-by: AlexDBlack <blacka101@gmail.com>
DL4J jackson updates + databind version 2.9.9.3
Signed-off-by: AlexDBlack <blacka101@gmail.com>
Shade snakeyaml along with jackson
Signed-off-by: AlexDBlack <blacka101@gmail.com>
Version fix
Signed-off-by: AlexDBlack <blacka101@gmail.com>
Switch DataVec legacy JSON format handling to mixins
Signed-off-by: AlexDBlack <blacka101@gmail.com>
Next set of fixes
Signed-off-by: AlexDBlack <blacka101@gmail.com>
Cleanup for legacy JSON mapping
Signed-off-by: AlexDBlack <blacka101@gmail.com>
Upgrade commons compress to 1.18; small test fix
Signed-off-by: AlexDBlack <blacka101@gmail.com>
New Jackson backward compatibility for DL4J - Round 1
Signed-off-by: AlexDBlack <blacka101@gmail.com>
New Jackson backward compatibility for DL4J - Round 2
Signed-off-by: AlexDBlack <blacka101@gmail.com>
More fixes, all but legacy custom passing
Signed-off-by: AlexDBlack <blacka101@gmail.com>
Provide an upgrade path for custom layers for models in pre-1.0.0-bet…
…a JSON format

Signed-off-by: AlexDBlack <blacka101@gmail.com>
Legacy deserialization cleanup
Signed-off-by: AlexDBlack <blacka101@gmail.com>
Small amount of polish - legacy JSON
Signed-off-by: AlexDBlack <blacka101@gmail.com>
Upgrade guava version
Signed-off-by: AlexDBlack <blacka101@gmail.com>
IEvaluation legacy format deserialization fix
Signed-off-by: AlexDBlack <blacka101@gmail.com>
Upgrade play version to 2.7.3
Signed-off-by: AlexDBlack <blacka101@gmail.com>
Update nd4j-parameter-server-status to new Play API
Signed-off-by: AlexDBlack <blacka101@gmail.com>
Update DL4J UI for new play version
Signed-off-by: AlexDBlack <blacka101@gmail.com>
More play framework updates
Signed-off-by: AlexDBlack <blacka101@gmail.com>
Small fixes
Signed-off-by: AlexDBlack <blacka101@gmail.com>
Remove Spark 1/2 adapter code from DataVec
Signed-off-by: AlexDBlack <blacka101@gmail.com>
datavec-spark dependency cleanup
Signed-off-by: AlexDBlack <blacka101@gmail.com>
DL4J spark updates, pt 1
Signed-off-by: AlexDBlack <blacka101@gmail.com>
DL4J spark updates, pt 2
Signed-off-by: AlexDBlack <blacka101@gmail.com>
DL4J spark updates, pt 3
Signed-off-by: AlexDBlack <blacka101@gmail.com>
DL4J spark updates, pt 4
Signed-off-by: AlexDBlack <blacka101@gmail.com>
Test fix
Signed-off-by: AlexDBlack <blacka101@gmail.com>
Another fix
Signed-off-by: AlexDBlack <blacka101@gmail.com>
Breeze upgrade, dependency cleanup
Signed-off-by: AlexDBlack <blacka101@gmail.com>
AlexDBlack and others added 25 commits Aug 27, 2019
DataVec Play fixes
Signed-off-by: AlexDBlack <blacka101@gmail.com>
datavec play dependency fixes
Signed-off-by: AlexDBlack <blacka101@gmail.com>
Clean up old spark/jackson stuff
Signed-off-by: AlexDBlack <blacka101@gmail.com>
Cleanup jackson unused dependencies
Signed-off-by: AlexDBlack <blacka101@gmail.com>
Add shaded guava
Signed-off-by: AlexDBlack <blacka101@gmail.com>
Dropping redundant dependency
Signed-off-by: Alexander Stoyakin <alexander.stoyakin@gmail.com>
Removed scalaxy dependency
Signed-off-by: Alexander Stoyakin <alexander.stoyakin@gmail.com>
Ensure not possible to import pre-shaded classes, and remove direct g…
…uava dependencies in favor of shaded

Signed-off-by: AlexDBlack <blacka101@gmail.com>
ND4J Shaded guava import fixes
Signed-off-by: AlexDBlack <blacka101@gmail.com>
DataVec and DL4J guava shading
Signed-off-by: AlexDBlack <blacka101@gmail.com>
Arbiter, RL4J fixes
Signed-off-by: AlexDBlack <blacka101@gmail.com>
Build fixed
Signed-off-by: Alexander Stoyakin <alexander.stoyakin@gmail.com>
Fix dependency
Signed-off-by: Alexander Stoyakin <alexander.stoyakin@gmail.com>
Merge
Signed-off-by: AlexDBlack <blacka101@gmail.com>
Fix bad merge
Signed-off-by: AlexDBlack <blacka101@gmail.com>
Jackson shading fixes
Signed-off-by: AlexDBlack <blacka101@gmail.com>
Set play secret, datavec-spark-inference-server
Signed-off-by: AlexDBlack <blacka101@gmail.com>
Fix for datavec-spark-inference-server
Signed-off-by: AlexDBlack <blacka101@gmail.com>
Arbiter fixes
Signed-off-by: AlexDBlack <blacka101@gmail.com>
Arbiter fixes
Signed-off-by: AlexDBlack <blacka101@gmail.com>
Small test fix
Signed-off-by: AlexDBlack <blacka101@gmail.com>

@AlexDBlack AlexDBlack marked this pull request as ready for review Aug 30, 2019

@AlexDBlack AlexDBlack merged commit dcc2baa into master Aug 30, 2019

@AlexDBlack AlexDBlack deleted the ab_20190813_dep_upgrades branch Sep 5, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.