fix: Fix reflective class loading in IntelliJ #1456
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
svotaw
commented
Mar 26, 2022
lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/swig/SwigUtils.scala
Show resolved
Hide resolved
svotaw
changed the title
mhamilton723
requested changes
Mar 28, 2022
core/src/main/scala/com/microsoft/azure/synapse/ml/core/utils/JarLoadingUtils.scala
Show resolved
Hide resolved
core/src/main/scala/com/microsoft/azure/synapse/ml/core/utils/JarLoadingUtils.scala
Outdated
Show resolved
Hide resolved
core/src/main/scala/com/microsoft/azure/synapse/ml/core/utils/JarLoadingUtils.scala
Outdated
Show resolved
Hide resolved
core/src/main/scala/com/microsoft/azure/synapse/ml/core/utils/JarLoadingUtils.scala
Outdated
Show resolved
Hide resolved
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Codecov Report
@@ Coverage Diff @@
## master #1456 +/- ##
==========================================
- Coverage 84.39% 84.31% -0.09%
==========================================
Files 284 284
Lines 14427 14439 +12
Branches 718 722 +4
==========================================
- Hits 12176 12174 -2
- Misses 2251 2265 +14
Continue to review full report at Codecov.
|
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
mhamilton723
approved these changes
Mar 30, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Some tests rely on the ability to reflect all classes in the synpaseml package (e.g. to filter to a set of all classes derived from a particular parent class). The current implementation uses ClassPath.from() from a spark utility package. However, this utility only works for ClassLoaders derived from UrlClassLoader (not well documented limitation). In the context of sbt, this is true, so things work in pipelines and such. In IntelliJ, the ClassLoaders are the standard App/Ext/Boot trio, which are not, so any test using this pathway fails since ClassPath.from() returns an empty list.
This PR adds a double check so that if the reflected class list from ClassPath comes back empty, it falls back to a manual lookup. This lookup is slower, so the choice was to not replace it, but instead add a path that should only run when a test is run from inside IntelliJ. Caching was added so the lookup is only done once in either case.
Tests
Some tests that were failing before in IntelliJ now work (used first FuzzingTest as a canary). Due to the nature of the fix, there is no way to test this in the pipelines since they use sbt.
Dependency chances
N/A