New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
scicloj.ml not loading with lein on windows #5
Comments
@swapneils Thanks for reporting this issue on Windows. I think there are 2 ways forward to investigate this.
and / or
'Requiring' `scicloj.ml' does not do any calls to anything in nippy. |
@swapneils looks like jvm version is a silent culprit. If you are doing this from the REPL you would see some error messages; in my case scicloj.ml has been built on a newer jvm (I am stuck on 1.8 for work purposes):
nippy and nippy.crypto load fine though. |
encryption ns loads too
|
This java 8 imcompability has been fixed in |
The java 1.8 fix was not part of |
@swapneils Could you re-try your code with |
Thanks for the help, it definitely seems to have done something (I'm on OpenJDK-17, but I suspect maybe tech.parallel or lein-classpath-jar were compiled with an older version of Java, since this fix qualitatively changed the error list). Note that aside from the main issue of the classpath being too long, I have no way of knowing whether the subsequent errors are due to scicloj and its dependencies or simply problems with lein-classpath-jar. I'll add the other tests anyway for completeness, but the first two are the ones involving the primary issue, and the ones that don't have anything but scicloj.ml that they could be related to. Tests in order: Runnning a repl with nippy on its own and then requiring it works fine, and requiring nippy.encryption also works Trying to run the repl with scicloj.ml as a dependency but without lein-classpath-jar set up in the project.clj still fails, with the same error. Trying to run the repl with lein-classpath-jar succeeds, but now gives a different error when requiring the 3 main scicloj namespaces: enc refers to com.taonesso/encore. Requiring after adding that as a dependency gave me: However, I noticed that this error (unlike the prior ones) was placed after 2 error messages and some spec.test outputs for scicloj functions: I tried removing all spec namespaces from my require list in core.clj as well as all dependencies for spec-related projects (like spec-tools), but it did not prevent the instrumentation from taking place. Looking at the error message in the first "[nREPL..." line, I looked into dtype-next (which is supposed to be the "successor" to tech.datatype), whose README says notes that jdk-17 required special project.clj flags. Then I thought maybe importing scicloj.metamorph manually would work, but that gave me some issue regarding preprocessing.clj not being available. Final stack trace: As a side note, I'm not sure why spec is trying to instrument do-ctx despite it being a function in a projected namespace, and without either scicloj.metamorph.core (where do-ctx is implemented) or any of the .clj files between my core.clj and tech.parallel (where the code for export-symbols is implemented) requiring clojure.spec in any way. Thoughts? Current list of successive patches, for easier tracking of issue progress: Add lein-classpath-jar as detailed in its GitHub repository Add com.taonesso/encore to dependencies in project.clj, above the adding of scicloj.ml Add the following to :profiles in project.clj to allow vector ops in jdk-17: Add scicloj/metamorph to dependencies in project.clj, between scicloj.ml and com.taonesso/encore |
problem disappears when using clojure cli, deps.edn (for me). lack of do-ctx is result of old dependency being pulled in and colliding with new in the dependency resolution. |
@behrica looks like you have no explicit dependency for scicloj.metamorph.ml: https://github.com/scicloj/scicloj.ml/blob/main/deps.edn#L4 I think you are relying on something else to bring it in. It looks like scicloj/scicloj.ml.xgboost "5.03" is bringing in scicloj/metamorph.ml "0.3.0-beta1" instead of 0.4.1, which the older version is lacking do-ctx. |
Yup, I was right. @behrica @swapneils this works:
I am able to fully load everything if I specify the scicloj.metamorph.ml dependency and prevent scicloj.ml from resolving it. |
@behrica @joinr I can confirm that using lein-classpath-jar and @joinr's manual import of metamorph.ml, and removing malli from the project.clj dependencies, this works for the test code in the README. The malli incompatibility seems to be an invalid schema in scicloj.metamorph.ml. I've opened Issue 4 on that repository regarding this. |
This is very strange, that it makes a difference. |
This is wrong, and i will fix it. |
Issue is solved in scicloj/scicloj.ml "0.1.3" I tested l with lein on Linux |
Hello, I was trying to set up scicloj via lein on Windows, but the lein repl wouldn't work; judging from the stack trace, java wouldn't run because the 'classpath was too long'.
I looked this up, and ran into technomancy/leiningen#2452, which mentions this is a common JVM problem and proposes a lein plugin, lein-classpath-jar, to fix it. I set up the plugin in my project.clj and successfully opened the repl.
I then encountered a different issue when trying to actually load scicloj.ml.core, that the crypto module in "com.taonesso/nippy" wasn't available, so some other file in that library (I believe encryption.clj) couldn't load.
Manually importing com.taonesso/nippy led me to another error (which may or may not have been there before, and merely overshadowed by the encryption.clj problem), that "namespace 'scicloj.ml.core' not found".
Commenting out ml.core, ml.metamorph, and ml.dataset allows for loading with no issues.
I am honestly not certain whether the second set of issue is due to the way lein-classpath-jar deals with long-classpaths, or an issue with how scicloj is constructed. But the first issue is definitely problematic for scicloj.ml development on Windows. Any ideas for patching this, or avoiding it on the user side?
Steps to reproduce primary issue:
Create a lein project three levels below the Users folder in a Windows environment (not sure if this is necessary for reproduction, but it's how my case is positioned)
Add scicloj.ml (via the lein recipe on clojars) to the project.clj defproject macro's :dependencies list
Attempt to use 'cider-jack-in' or 'lein repl' in the same folder as the project's core.clj file. -> java cannot be run
Steps to reproduce secondary issue:
Add lein-classpath-jar to your lein :user settings as specified by the plugin's README
Attempt to use 'cider-jack-in' or 'lein repl' in the above-created project -> compilation issue regarding nippy
Add nippy to project.clj :dependencies
Attempt 'cider-jack-in' or 'lein repl' -> scicloj.ml.core not found
The text was updated successfully, but these errors were encountered: