Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support packages (libraries) in R #17

Closed
gaocegege opened this issue Mar 12, 2017 · 20 comments
Closed

Support packages (libraries) in R #17

gaocegege opened this issue Mar 12, 2017 · 20 comments

Comments

@gaocegege
Copy link
Member

gaocegege commented Mar 12, 2017

Renjin supports some of CRAN and BioConductor R libraries, Processing.R could benefit from it.

ref the post in forum

As for libraries, I did a feasibility study. Renjin is used in Processing.R as the backend to interpret R code in JVM. Documentation in renjin.org shows differences about packages between GNU R and renjin.

It supports some of CRAN and BioConductor R libraries and I want to integrate those libraries into Processing.R in GSoC. But I'm not sure whether those packages could be installed in runtime or ahead of time.

BTW I think Jython supports those libraries which do not require C compilation, although I do not have the experience on it.

@jeremydouglass
Copy link
Member

jeremydouglass commented May 19, 2017

@gaocegege --

What are some top CRAN / BioConductor R packages that you would be interested in testing (or in having someone test for you)? Ideally these would be popular packages, low level (not a lot of dependencies), not have their own graphical outputs (for simplicity, and not to overlap with Processing), and be something that Processing users might want to use .

For example, I might be interested in:

@jeremydouglass jeremydouglass changed the title Support libraries in R Support packages (libraries) in R May 19, 2017
@jeremydouglass
Copy link
Member

jeremydouglass commented May 19, 2017

A preliminary look at Renjin package listings shows that they automatically test -- however they don't provide a single listing of packages that pass, and the packages that I chose above didn't pass testing (possibly because they are dependency-heavy or C/++ optimized), so we need to be systematic about searching or pick well-documented examples.

  • graph: This package can be loaded by Renjin but there was an error compiling C/FORTRAN sources and 38 out 51 tests failed.
  • pcaMethods: This package can be loaded by Renjin but there was an error compiling C/FORTRAN sources and 18 out 28 tests failed.
  • dplyr: This package can be loaded by Renjin but there was an error compiling C/FORTRAN sources and 35 out 62 tests failed.
  • stringi: This package can be loaded by Renjin but there was an error compiling C/FORTRAN sources and 48 out 49 tests failed.
  • stringr: This package can be loaded by Renjin but all tests failed.
  • lubridate: This package can be loaded by Renjin but all tests failed.

By contrast, here is a package in a Renjin-specific version that passes all tests. We might use it for our preliminary testing of whether importing works:

@jeremydouglass
Copy link
Member

I've posted a question about listings of available test-passing renjin packages here: https://groups.google.com/forum/#!topic/renjin-dev/rrLM4wo3by8

@gaocegege
Copy link
Member Author

gaocegege commented May 19, 2017

Yeah, there are some packages which pass all the tests. And I agree that we could use them to test whether it works.

As for those packages which fails some test cases, I think we should dive into the packages and have a try. And if there are some important libraries, maybe we could help renjin to implement them in JVM. Then we could use it in our mode.

@gaocegege
Copy link
Member Author

ref https://github.com/bedatadriven/renjin/tree/master/cli/src/main/java/org/renjin/cli

Renjin CLI will download the package at runtime, we could take a look at it.

@gaocegege
Copy link
Member Author

Take plyr as an example, it has a lot of dependencies, put them into lib/mode manually is not what we want. I will take a look further.

<dependencies>
<dependency>
<groupId>org.renjin</groupId>
<artifactId>methods</artifactId>
<version>0.8.2343</version>
</dependency>
<dependency>
<groupId>org.renjin</groupId>
<artifactId>parallel</artifactId>
<version>0.8.2343</version>
</dependency>
<dependency>
<groupId>org.renjin</groupId>
<artifactId>datasets</artifactId>
<version>0.8.2343</version>
</dependency>
<dependency>
<groupId>org.renjin</groupId>
<artifactId>stats</artifactId>
<version>0.8.2343</version>
</dependency>
<dependency>
<groupId>org.renjin</groupId>
<artifactId>splines</artifactId>
<version>0.8.2343</version>
</dependency>
<dependency>
<groupId>org.renjin</groupId>
<artifactId>grDevices</artifactId>
<version>0.8.2343</version>
</dependency>
<dependency>
<groupId>org.renjin</groupId>
<artifactId>tcltk</artifactId>
<version>0.8.2343</version>
</dependency>
<dependency>
<groupId>org.renjin</groupId>
<artifactId>stats4</artifactId>
<version>0.8.2343</version>
</dependency>
<dependency>
<groupId>org.renjin.cran</groupId>
<artifactId>Rcpp</artifactId>
<version>0.12.5-renjin-7</version>
</dependency>
<dependency>
<groupId>org.renjin.cran</groupId>
<artifactId>Rcpp</artifactId>
<version>0.12.5-renjin-7</version>
<classifier>headers</classifier>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.renjin</groupId>
<artifactId>tools</artifactId>
<version>0.8.2343</version>
</dependency>
<dependency>
<groupId>org.renjin</groupId>
<artifactId>utils</artifactId>
<version>0.8.2343</version>
</dependency>
<dependency>
<groupId>org.renjin</groupId>
<artifactId>grid</artifactId>
<version>0.8.2343</version>
</dependency>
<dependency>
<groupId>org.renjin</groupId>
<artifactId>graphics</artifactId>
<version>0.8.2343</version>
</dependency>
<dependency>
<groupId>org.renjin</groupId>
<artifactId>compiler</artifactId>
<version>0.8.2343</version>
<scope>provided</scope>
</dependency>
</dependencies>

@jeremydouglass
Copy link
Member

jeremydouglass commented Jun 6, 2017

From the dependency list, tcltk in particular jumps out at me as probably huge and possibly not necessary. Interesting that plyr passed automatic testing on the renjin package site -- looking at these dependencies I would have guessed that it wouldn't.

@jeremydouglass
Copy link
Member

Strange -- I just noticed that renjin's CRAN listing for plyr only lists a single dependency -- RCPP.

http://packages.renjin.org/package/org.renjin.cran/plyr/1.8.4

...which doesn't list any dependencies. So the web listings aren't matching up with that dependencies report.

@jeremydouglass
Copy link
Member

Perhaps for future research we could instead try something like stringi, which is a renjin-specific version and lists (?) no dependencies.

http://packages.renjin.org/package/org.renjin.cran/stringi

@gaocegege
Copy link
Member Author

I will take a look

@jeremydouglass
Copy link
Member

Strange. For plyr I looked at the renjin build log and test results, and I don't see any of those dependencies (like tcltk) -- unless I'm misunderstanding.

@gaocegege
Copy link
Member Author

gaocegege commented Jun 12, 2017

Take plyr-1.8.4 as an example, the deps are defined in https://nexus.bedatadriven.com/content/groups/public/org/renjin/cran/plyr/1.8.4-b17/plyr-1.8.4-b17.pom. They are maven dependencies.

@jeremydouglass
Copy link
Member

Hmm, something is off about that this. Those aren't true dependencies, that is an expanded copy of the "Suggests" list from plyr 1.8.4 on CRAN:

https://cran.r-project.org/web/packages/plyr/index.html

Depends: R (≥ 3.1.0)
Imports: Rcpp (≥ 0.11.0)
LinkingTo: Rcpp
Suggests: abind, testthat, tcltk, foreach, doParallel, itertools, iterators, covr

tcltk is definitely not a dependency.

@gaocegege
Copy link
Member Author

🤔 I have tried to run without them and it returns an error, I will have a try again and get some details

@jeremydouglass
Copy link
Member

jeremydouglass commented Jun 13, 2017

Hmm. "Suggests" means that plyr "uses some functions from these packages in examples or in its vignettes."

So if a build is running tests against examples or vignettes that make calls to the "suggests" packages then those tests would fail if the "suggests" are not installed -- but plyr will still work, it just can't run those examples. What kind of error are you getting?

For discussion see:

@gaocegege
Copy link
Member Author

When I runs without Rcpp, it raises:

org.renjin.eval.EvalException: IOException while loading package org.renjin.cran:plyr: Could not load package Rcpp; tried org.renjin.bioconductor:Rcpp, org.renjin.cran:Rcpp
	at org.renjin.primitives.packaging.NamespaceRegistry.tryLoad(NamespaceRegistry.java:234)
	at org.renjin.primitives.packaging.NamespaceRegistry.tryGetNamespace(NamespaceRegistry.java:191)
	at org.renjin.primitives.packaging.NamespaceRegistry.getNamespace(NamespaceRegistry.java:153)
	at org.renjin.primitives.packaging.NamespaceRegistry.getNamespace(NamespaceRegistry.java:115)
	at org.renjin.primitives.packaging.Packages.library(Packages.java:41)
	at org.renjin.primitives.R$primitive$library.doApply(R$primitive$library.java:73)
	at org.renjin.primitives.R$primitive$library.apply(R$primitive$library.java:34)
	at org.renjin.primitives.special.InternalFunction.apply(InternalFunction.java:45)
	at org.renjin.eval.Context.evaluateCall(Context.java:368)
	at org.renjin.eval.Context.evaluate(Context.java:256)
	at org.renjin.primitives.special.BeginFunction.apply(BeginFunction.java:38)
	at org.renjin.eval.Context.evaluateCall(Context.java:368)
	at org.renjin.eval.Context.evaluate(Context.java:256)
	at org.renjin.eval.Context.evaluate(Context.java:197)
	at org.renjin.sexp.Closure.doApply(Closure.java:84)
	at org.renjin.eval.ClosureDispatcher.apply(ClosureDispatcher.java:76)
	at org.renjin.eval.ClosureDispatcher.applyClosure(ClosureDispatcher.java:58)
	at org.renjin.sexp.Closure.apply(Closure.java:79)
	at org.renjin.eval.Context.evaluateCall(Context.java:368)
	at org.renjin.eval.Context.evaluate(Context.java:256)
	at org.renjin.eval.Context.evaluateExpressionVector(Context.java:351)
	at org.renjin.eval.Context.evaluate(Context.java:254)
	at org.renjin.script.RenjinScriptEngine.eval(RenjinScriptEngine.java:168)
	at org.renjin.script.RenjinScriptEngine.eval(RenjinScriptEngine.java:127)
	at rprocessing.RLangPApplet.setup(Unknown Source)
	at processing.core.PApplet.handleDraw(PApplet.java:2414)
	at processing.awt.PSurfaceAWT$12.callDraw(PSurfaceAWT.java:1547)
	at processing.core.PSurfaceNone$AnimationThread.run(PSurfaceNone.java:316)
Caused by: org.renjin.eval.EvalException: Could not load package Rcpp; tried org.renjin.bioconductor:Rcpp, org.renjin.cran:Rcpp
	at org.renjin.primitives.packaging.NamespaceRegistry.getNamespace(NamespaceRegistry.java:160)
	at org.renjin.primitives.packaging.NamespaceRegistry.getNamespace(NamespaceRegistry.java:115)
	at org.renjin.primitives.packaging.Namespace.initImports(Namespace.java:181)
	at org.renjin.primitives.packaging.NamespaceRegistry.tryLoad(NamespaceRegistry.java:214)

When I run with plyr and Rcpp:

org.renjin.eval.EvalException: IOException while loading package org.renjin.cran:plyr: IOException while loading package org.renjin.cran:Rcpp: Expected object of class 'NativeSymbolInfo'
	at org.renjin.primitives.packaging.NamespaceRegistry.tryLoad(NamespaceRegistry.java:234)
	at org.renjin.primitives.packaging.NamespaceRegistry.tryGetNamespace(NamespaceRegistry.java:191)
	at org.renjin.primitives.packaging.NamespaceRegistry.getNamespace(NamespaceRegistry.java:153)
	at org.renjin.primitives.packaging.NamespaceRegistry.getNamespace(NamespaceRegistry.java:115)
	at org.renjin.primitives.packaging.Packages.library(Packages.java:41)
	at org.renjin.primitives.R$primitive$library.doApply(R$primitive$library.java:73)
	at org.renjin.primitives.R$primitive$library.apply(R$primitive$library.java:34)
	at org.renjin.primitives.special.InternalFunction.apply(InternalFunction.java:45)
	at org.renjin.eval.Context.evaluateCall(Context.java:368)
	at org.renjin.eval.Context.evaluate(Context.java:256)
	at org.renjin.primitives.special.BeginFunction.apply(BeginFunction.java:38)
	at org.renjin.eval.Context.evaluateCall(Context.java:368)
	at org.renjin.eval.Context.evaluate(Context.java:256)
	at org.renjin.eval.Context.evaluate(Context.java:197)
	at org.renjin.sexp.Closure.doApply(Closure.java:84)
	at org.renjin.eval.ClosureDispatcher.apply(ClosureDispatcher.java:76)
	at org.renjin.eval.ClosureDispatcher.applyClosure(ClosureDispatcher.java:58)
	at org.renjin.sexp.Closure.apply(Closure.java:79)
	at org.renjin.eval.Context.evaluateCall(Context.java:368)
	at org.renjin.eval.Context.evaluate(Context.java:256)
	at org.renjin.eval.Context.evaluateExpressionVector(Context.java:351)
	at org.renjin.eval.Context.evaluate(Context.java:254)
	at org.renjin.script.RenjinScriptEngine.eval(RenjinScriptEngine.java:168)
	at org.renjin.script.RenjinScriptEngine.eval(RenjinScriptEngine.java:127)
	at rprocessing.RLangPApplet.setup(Unknown Source)
	at processing.core.PApplet.handleDraw(PApplet.java:2414)
	at processing.awt.PSurfaceAWT$12.callDraw(PSurfaceAWT.java:1547)
	at processing.core.PSurfaceNone$AnimationThread.run(PSurfaceNone.java:316)
Caused by: org.renjin.eval.EvalException: IOException while loading package org.renjin.cran:Rcpp: Expected object of class 'NativeSymbolInfo'
	at org.renjin.primitives.packaging.NamespaceRegistry.tryLoad(NamespaceRegistry.java:234)
	at org.renjin.primitives.packaging.NamespaceRegistry.tryGetNamespace(NamespaceRegistry.java:191)
	at org.renjin.primitives.packaging.NamespaceRegistry.getNamespace(NamespaceRegistry.java:153)
	at org.renjin.primitives.packaging.NamespaceRegistry.getNamespace(NamespaceRegistry.java:115)
	at org.renjin.primitives.packaging.Namespace.initImports(Namespace.java:181)
	at org.renjin.primitives.packaging.NamespaceRegistry.tryLoad(NamespaceRegistry.java:214)
	... 27 more
Caused by: org.renjin.eval.EvalException: Expected object of class 'NativeSymbolInfo'
	at org.renjin.primitives.Native.external(Native.java:452)
	at org.renjin.primitives.R$primitive$$External.apply(R$primitive$$External.java:63)
	at org.renjin.eval.Context.evaluateCall(Context.java:368)
	at org.renjin.eval.Context.evaluate(Context.java:256)
	at org.renjin.eval.Context.evaluate(Context.java:197)
	at org.renjin.sexp.Closure.doApply(Closure.java:84)
	at org.renjin.eval.ClosureDispatcher.apply(ClosureDispatcher.java:76)
	at org.renjin.eval.ClosureDispatcher.applyClosure(ClosureDispatcher.java:58)
	at org.renjin.sexp.Closure.apply(Closure.java:79)
	at org.renjin.eval.Context.evaluateCall(Context.java:368)
	at org.renjin.eval.Context.evaluate(Context.java:256)
	at org.renjin.primitives.special.BeginFunction.apply(BeginFunction.java:38)
	at org.renjin.eval.Context.evaluateCall(Context.java:368)
	at org.renjin.eval.Context.evaluate(Context.java:256)
	at org.renjin.eval.Context.evaluate(Context.java:197)
	at org.renjin.sexp.Closure.doApply(Closure.java:84)
	at org.renjin.eval.ClosureDispatcher.apply(ClosureDispatcher.java:76)
	at org.renjin.eval.ClosureDispatcher.applyClosure(ClosureDispatcher.java:58)
	at org.renjin.sexp.Closure.apply(Closure.java:79)
	at org.renjin.eval.Context.evaluateCall(Context.java:368)
	at org.renjin.eval.Context.evaluate(Context.java:256)
	at org.renjin.primitives.packaging.NamespaceRegistry.tryLoad(NamespaceRegistry.java:220)
	... 32 more

@jeremydouglass
Copy link
Member

Maarten on the renjin-dev list has responded with a very useful list of suggested renjin R packages for testing.

https://groups.google.com/d/msg/renjin-dev/rrLM4wo3by8/O7cDR6V8AAAJ

These entries from his suggestion list look like good starting points:

regexr, foreach, and magrittr are all syntax / control structure packages, and they look light on dependencies. Perhaps one would work better for a quick test?

@gaocegege
Copy link
Member Author

Yeah, I will have a look. Thanks for your information :)

@jeremydouglass
Copy link
Member

Great!

Should we add an example sketch that demonstrates using an R library to the example set?

@gaocegege
Copy link
Member Author

Yeah, I will try to write an example.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment