Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ADAM-882] R API #1397

Merged
merged 1 commit into from Jul 21, 2017
Merged

[ADAM-882] R API #1397

merged 1 commit into from Jul 21, 2017

Conversation

@fnothaft
Copy link
Member

@fnothaft fnothaft commented Feb 21, 2017

WIP towards #882. Still bringing in tests, functions beyond reads. On top of #1391.

TODOs:

  • Full docs for loadAlignments
  • Add loadCoverage
  • Document catch 22 in build

Union method can't be added; necessary support in SparkR for dealing with lists of jobjs in R does not exist.

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Feb 21, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1808/
Test PASSed.

Type: Package
Version: 0.21.0
Title: R Frontend for Big Data Genomics/ADAM
Description: The SparkR package provides an R Frontend for Apache Spark.

This comment has been minimized.

@heuermh

heuermh Feb 21, 2017
Member

description should match title

@heuermh
Copy link
Member

@heuermh heuermh commented Feb 21, 2017

Want to remove adam-scripts/R as part of this pull request?

@fnothaft
Copy link
Member Author

@fnothaft fnothaft commented Feb 21, 2017

Want to remove adam-scripts/R as part of this pull request?

Let me look into that. IIRC, the code there is chart generation code for concordance testing, and not related to using ADAM from R.

@heuermh
Copy link
Member

@heuermh heuermh commented Feb 21, 2017

IIRC, the code there is chart generation code for concordance testing, and not related to using ADAM from R.

Correct, but the commit it came on (commit bf60cb9) refers to things no longer present (CompareAdam, FindReads)

@fnothaft fnothaft mentioned this pull request Feb 21, 2017
@fnothaft fnothaft added this to the 0.22.0 milestone Mar 3, 2017
@heuermh heuermh added this to Triage in Release 0.23.0 Mar 8, 2017
@fnothaft fnothaft force-pushed the fnothaft:issues/882-r branch from 12daddc to 57129c3 May 31, 2017
@coveralls
Copy link

@coveralls coveralls commented May 31, 2017

Coverage Status

Coverage decreased (-3.7%) to 79.064% when pulling 57129c3 on fnothaft:issues/882-r into b7762c2 on bigdatagenomics:master.

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented May 31, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2078/
Test PASSed.

@fnothaft fnothaft force-pushed the fnothaft:issues/882-r branch from 57129c3 to 5e7e535 Jun 23, 2017
@coveralls
Copy link

@coveralls coveralls commented Jun 23, 2017

Coverage Status

Coverage decreased (-4.0%) to 79.064% when pulling 5e7e535 on fnothaft:issues/882-r into 0306717 on bigdatagenomics:master.

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Jun 23, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2132/
Test PASSed.

@fnothaft
Copy link
Member Author

@fnothaft fnothaft commented Jun 23, 2017

Jenkins, retest this please.

@coveralls
Copy link

@coveralls coveralls commented Jun 23, 2017

Coverage Status

Coverage decreased (-0.1%) to 82.951% when pulling 5e7e535 on fnothaft:issues/882-r into 0306717 on bigdatagenomics:master.

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Jun 23, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2134/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1397/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 8781d0e855ab838a9c0327ddf3b9a878c18ef8f3 # timeout=10Checking out Revision 8781d0e855ab838a9c0327ddf3b9a878c18ef8f3 (origin/pr/1397/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 8781d0e855ab838a9c0327ddf3b9a878c18ef8f3 > /home/jenkins/git2/bin/git rev-list 8781d0e855ab838a9c0327ddf3b9a878c18ef8f3 # timeout=10Triggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.0.0,centosADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@fnothaft fnothaft force-pushed the fnothaft:issues/882-r branch from 5e7e535 to e779519 Jun 23, 2017
@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Jun 23, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2139/

Build result: ABORTED

[...truncated 17 lines...] > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains dbb59b56f13f204c73550ac2ff95de2d7a956d8a # timeout=10Checking out Revision dbb59b56f13f204c73550ac2ff95de2d7a956d8a (origin/pr/1397/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f dbb59b56f13f204c73550ac2ff95de2d7a956d8a > /home/jenkins/git2/bin/git rev-list 8781d0e855ab838a9c0327ddf3b9a878c18ef8f3 # timeout=10Triggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.0.0,centosAbortedAborted by fnothaftInterrupting #2139Interrupting #2139Interrupting #2139Interrupting #2139Interrupting #2139Interrupting #2139Interrupting #2139Interrupting #2139Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@fnothaft fnothaft force-pushed the fnothaft:issues/882-r branch from e779519 to 573882e Jun 23, 2017
@coveralls
Copy link

@coveralls coveralls commented Jun 23, 2017

Coverage Status

Coverage decreased (-0.03%) to 83.013% when pulling 573882e on fnothaft:issues/882-r into 0306717 on bigdatagenomics:master.

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Jun 23, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2142/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1397/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains f4b9fb720a579667a8942130aeb3366e28064be6 # timeout=10Checking out Revision f4b9fb720a579667a8942130aeb3366e28064be6 (origin/pr/1397/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f f4b9fb720a579667a8942130aeb3366e28064be6 > /home/jenkins/git2/bin/git rev-list dbb59b56f13f204c73550ac2ff95de2d7a956d8a # timeout=10Triggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.0.0,centosADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@fnothaft fnothaft force-pushed the fnothaft:issues/882-r branch from 573882e to 11be738 Jun 23, 2017
@coveralls
Copy link

@coveralls coveralls commented Jun 23, 2017

Coverage Status

Coverage decreased (-0.03%) to 83.013% when pulling 11be738 on fnothaft:issues/882-r into 0306717 on bigdatagenomics:master.

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Jun 24, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2144/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1397/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains f380044c36a67752156487d9f889df271421ab97 # timeout=10Checking out Revision f380044c36a67752156487d9f889df271421ab97 (origin/pr/1397/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f f380044c36a67752156487d9f889df271421ab97 > /home/jenkins/git2/bin/git rev-list f4b9fb720a579667a8942130aeb3366e28064be6 # timeout=10Triggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.0.0,centosADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@coveralls
Copy link

@coveralls coveralls commented Jun 24, 2017

Coverage Status

Coverage decreased (-0.03%) to 83.013% when pulling 1b8f3dd on fnothaft:issues/882-r into 0306717 on bigdatagenomics:master.

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Jun 24, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2145/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1397/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains f15b8b6c0826dd9a596fa5a74ef38bf6f84b131b # timeout=10Checking out Revision f15b8b6c0826dd9a596fa5a74ef38bf6f84b131b (origin/pr/1397/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f f15b8b6c0826dd9a596fa5a74ef38bf6f84b131b > /home/jenkins/git2/bin/git rev-list f380044c36a67752156487d9f889df271421ab97 # timeout=10Triggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.0.0,centosADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@fnothaft
Copy link
Member Author

@fnothaft fnothaft commented Jun 24, 2017

Jenkins, test this please.

@fnothaft
Copy link
Member Author

@fnothaft fnothaft commented Jul 19, 2017

@devin-petersohn can you cat /usr/local/lib/R/3.4/site-library/SparkR/DESCRIPTION and post the output?

@devin-petersohn
Copy link
Member

@devin-petersohn devin-petersohn commented Jul 19, 2017

$ cat /usr/local/lib/R/3.4/site-library/SparkR/DESCRIPTION
Package: SparkR
Type: Package
Version: 2.1.0
Title: R Frontend for Apache Spark
Description: The SparkR package provides an R Frontend for Apache Spark.
Authors@R: c(person("Shivaram", "Venkataraman", role = c("aut", "cre"),
                    email = "shivaram@cs.berkeley.edu"),
             person("Xiangrui", "Meng", role = "aut",
                    email = "meng@databricks.com"),
             person("Felix", "Cheung", role = "aut",
                    email = "felixcheung@apache.org"),
             person(family = "The Apache Software Foundation", role = c("aut", "cph")))
License: Apache License (== 2.0)
URL: http://www.apache.org/ http://spark.apache.org/
BugReports: http://spark.apache.org/contributing.html
Depends: R (>= 3.0), methods
Suggests: knitr, rmarkdown, testthat, e1071, survival
Collate: 'schema.R' 'generics.R' 'jobj.R' 'RDD.R' 'pairRDD.R'
        'column.R' 'group.R' 'DataFrame.R' 'SQLContext.R'
        'WindowSpec.R' 'backend.R' 'broadcast.R' 'client.R' 'context.R'
        'deserialize.R' 'functions.R' 'install.R' 'jvm.R' 'mllib.R'
        'serialize.R' 'sparkR.R' 'stats.R' 'types.R' 'utils.R'
        'window.R'
RoxygenNote: 5.0.1
VignetteBuilder: knitr
Author: Shivaram Venkataraman [aut, cre],
  Xiangrui Meng [aut],
  Felix Cheung [aut],
  The Apache Software Foundation [aut, cph]
Maintainer: Shivaram Venkataraman <shivaram@cs.berkeley.edu>
Built: R 3.1.1; ; 2016-12-16 02:06:48 UTC; unix
@fnothaft fnothaft force-pushed the fnothaft:issues/882-r branch from 004b1de to abcf028 Jul 19, 2017
@fnothaft
Copy link
Member Author

@fnothaft fnothaft commented Jul 19, 2017

Rebased and cleaned up a test issue that snuck in.

@coveralls
Copy link

@coveralls coveralls commented Jul 19, 2017

Coverage Status

Coverage decreased (-0.7%) to 84.092% when pulling abcf028 on fnothaft:issues/882-r into 7652536 on bigdatagenomics:master.

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Jul 19, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2250/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1397/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 6212a218fbcd2f05e5434def603f8a521677215b # timeout=10Checking out Revision 6212a218fbcd2f05e5434def603f8a521677215b (origin/pr/1397/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 6212a218fbcd2f05e5434def603f8a521677215b > /home/jenkins/git2/bin/git rev-list 2dc8a3d # timeout=10Triggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.10,2.1.0,centosTriggering ADAM-prb ? 2.6.0,2.10,2.1.0,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,2.1.0,centosTriggering ADAM-prb ? 2.3.0,2.11,2.1.0,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,2.1.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,2.1.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,2.1.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.11,2.1.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@fnothaft
Copy link
Member Author

@fnothaft fnothaft commented Jul 20, 2017

Jenkins, retest this please.

@fnothaft
Copy link
Member Author

@fnothaft fnothaft commented Jul 20, 2017

@devin-petersohn What do you get when you run cat /usr/local/lib/R/3.4/site-library/SparkR/NAMESPACE | grep sparkR.call? I get:

fnothaft$ cat /usr/local/lib/R/3.4/site-library/SparkR/NAMESPACE  | grep sparkR.call
export("sparkR.callJMethod")
export("sparkR.callJStatic")
@devin-petersohn
Copy link
Member

@devin-petersohn devin-petersohn commented Jul 20, 2017

I get the same.

$ cat /usr/local/lib/R/3.4/site-library/SparkR/NAMESPACE | grep sparkR.call
export("sparkR.callJMethod")
export("sparkR.callJStatic")
@coveralls
Copy link

@coveralls coveralls commented Jul 20, 2017

Coverage Status

Coverage remained the same at 84.743% when pulling abcf028 on fnothaft:issues/882-r into 7652536 on bigdatagenomics:master.

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Jul 20, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2251/
Test PASSed.

@fnothaft
Copy link
Member Author

@fnothaft fnothaft commented Jul 20, 2017

If you import(SparkR), are you able to call those functions? If you aren't able to, that'd explain why you're getting the following warnings:

* checking R code for possible problems ... NOTE
ADAMContext: no visible global function definition for
  ‘sparkR.callJMethod’
ADAMContext: no visible global function definition for
  ‘sparkR.newJObject’
@devin-petersohn
Copy link
Member

@devin-petersohn devin-petersohn commented Jul 20, 2017

Progress, but not there yet.

The problem was SPARK_HOME. To make a long, uninteresting story short, when Spark is installed with homebrew, you need SPARK_HOME=/usr/local/Cellar/apache-spark/2.1.0/libexec.

Now I can build a Spark in an R interpreter using ADAM's R API SPARKR_SUBMIT_ARGS and access the functions that were previously not visible:

> library(SparkR)

Attaching package:SparkRThe following objects are masked frompackage:stats:

    cov, filter, lag, na.omit, predict, sd, var, window

The following objects are masked frompackage:base:

    as.data.frame, colnames, colnames<-, drop, endsWith, intersect,
    rank, rbind, sample, startsWith, subset, summary, transform, union

> sparkR.session()
Spark package found in SPARK_HOME: /usr/local/Cellar/apache-spark/2.1.0/libexec
Launching java with spark-submit command /usr/local/Cellar/apache-spark/2.1.0/libexec/bin/spark-submit   --jars /Users/DevinPetersohn/software_builds/adam/adam-assembly/target/adam-assembly-spark2_2.11-0.23.0-SNAPSHOT.jar --driver-class-path /Users/DevinPetersohn/software_builds/adam/adam-assembly/target/adam-assembly-spark2_2.11-0.23.0-SNAPSHOT.jar sparkr-shell /var/folders/gt/ycvlk50j02j0144816474s8r0000gn/T//RtmpG8ADLf/backend_port6d11925cdf9 
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
2017-07-20 12:15:03 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Java ref type org.apache.spark.sql.SparkSession id 1 
> sparkR.callJMethod
function (x, methodName, ...) 
{
    callJMethod(x, methodName, ...)
}
<environment: namespace:SparkR>
> sparkR.newJObject
function (x, ...) 
{
    newJObject(x, ...)
}
<environment: namespace:SparkR>

But the build still fails:

...
* checking R code for possible problems ... NOTE
ADAMContext: no visible global function definition for
  ‘sparkR.callJMethod’
ADAMContext: no visible global function definition for
  ‘sparkR.newJObject’
ADAMContext: no visible global function definition for ‘new’
AlignmentRecordRDD: no visible global function definition for ‘new’
CoverageRDD: no visible global function definition for ‘new’
FeatureRDD: no visible global function definition for ‘new’
FragmentRDD: no visible global function definition for ‘new’
GenotypeRDD: no visible global function definition for ‘new’
NucleotideContigFragmentRDD: no visible global function definition for
  ‘new’
VariantRDD: no visible global function definition for ‘new’
aggregatedCoverage,CoverageRDD: no visible global function definition
  for ‘sparkR.callJMethod’
countKmers,AlignmentRecordRDD-numeric: no visible global function
  definition for ‘new’
countKmers,AlignmentRecordRDD-numeric: no visible global function
  definition for ‘sparkR.callJMethod’
coverage,CoverageRDD: no visible global function definition for
  ‘sparkR.callJMethod’
flankAdjacentFragments,NucleotideContigFragmentRDD-numeric: no visible
  global function definition for ‘sparkR.callJMethod’
flatten,CoverageRDD: no visible global function definition for
  ‘sparkR.callJMethod’
loadAlignments,ADAMContext-character: no visible global function
  definition for ‘sparkR.callJMethod’
loadContigFragments,ADAMContext-character: no visible global function
  definition for ‘sparkR.callJMethod’
loadFeatures,ADAMContext-character: no visible global function
  definition for ‘sparkR.callJMethod’
loadFragments,ADAMContext-character: no visible global function
  definition for ‘sparkR.callJMethod’
loadGenotypes,ADAMContext-character: no visible global function
  definition for ‘sparkR.callJMethod’
loadVariants,ADAMContext-character: no visible global function
  definition for ‘sparkR.callJMethod’
markDuplicates,AlignmentRecordRDD: no visible global function
  definition for ‘sparkR.callJMethod’
markDuplicates,FragmentRDD: no visible global function definition for
  ‘sparkR.callJMethod’
realignIndels,AlignmentRecordRDD-ANY: no visible global function
  definition for ‘sparkR.callJStatic’
realignIndels,AlignmentRecordRDD-ANY: no visible global function
  definition for ‘sparkR.callJMethod’
realignIndels,AlignmentRecordRDD-VariantRDD: no visible global function
  definition for ‘sparkR.callJStatic’
realignIndels,AlignmentRecordRDD-VariantRDD: no visible global function
  definition for ‘sparkR.callJMethod’
recalibrateBaseQualities,AlignmentRecordRDD-VariantRDD-character: no
  visible global function definition for ‘sparkR.callJStatic’
recalibrateBaseQualities,AlignmentRecordRDD-VariantRDD-character: no
  visible global function definition for ‘sparkR.callJMethod’
save,AlignmentRecordRDD-character: no visible global function
  definition for ‘sparkR.callJMethod’
save,CoverageRDD-ANY: no visible global function definition for
  ‘sparkR.callJMethod’
save,CoverageRDD-character: no visible global function definition for
  ‘sparkR.callJMethod’
save,FeatureRDD-character: no visible global function definition for
  ‘sparkR.callJMethod’
save,FragmentRDD-character: no visible global function definition for
  ‘sparkR.callJMethod’
save,GenotypeRDD-character: no visible global function definition for
  ‘sparkR.callJMethod’
save,NucleotideContigFragmentRDD-character: no visible global function
  definition for ‘sparkR.callJMethod’
save,VariantRDD-character: no visible global function definition for
  ‘sparkR.callJMethod’
saveAsSam,AlignmentRecordRDD-character: no visible global function
  definition for ‘sparkR.callJStatic’
saveAsSam,AlignmentRecordRDD-character: no visible global function
  definition for ‘sparkR.callJMethod’
saveAsVcf,GenotypeRDD-character: no visible global function definition
  for ‘sparkR.callJMethod’
saveAsVcf,GenotypeRDD-character: no visible global function definition
  for ‘sparkR.callJStatic’
saveAsVcf,VariantRDD-character: no visible global function definition
  for ‘sparkR.callJMethod’
saveAsVcf,VariantRDD-character: no visible global function definition
  for ‘sparkR.callJStatic’
sortReadsByReferencePosition,AlignmentRecordRDD: no visible global
  function definition for ‘sparkR.callJMethod’
sortReadsByReferencePositionAndIndex,AlignmentRecordRDD: no visible
  global function definition for ‘sparkR.callJMethod’
toCoverage,AlignmentRecordRDD: no visible global function definition
  for ‘sparkR.callJMethod’
toCoverage,FeatureRDD: no visible global function definition for
  ‘sparkR.callJMethod’
toDF,AlignmentRecordRDD: no visible global function definition for
  ‘sparkR.callJMethod’
toDF,AlignmentRecordRDD: no visible global function definition for
  ‘new’
toDF,CoverageRDD: no visible global function definition for ‘new’
toDF,CoverageRDD: no visible global function definition for
  ‘sparkR.callJMethod’
toDF,FeatureRDD: no visible global function definition for ‘new’
toDF,FeatureRDD: no visible global function definition for
  ‘sparkR.callJMethod’
toDF,FragmentRDD: no visible global function definition for ‘new’
toDF,FragmentRDD: no visible global function definition for
  ‘sparkR.callJMethod’
toDF,GenotypeRDD: no visible global function definition for ‘new’
toDF,GenotypeRDD: no visible global function definition for
  ‘sparkR.callJMethod’
toDF,NucleotideContigFragmentRDD: no visible global function definition
  for ‘new’
toDF,NucleotideContigFragmentRDD: no visible global function definition
  for ‘sparkR.callJMethod’
toDF,VariantRDD: no visible global function definition for ‘new’
toDF,VariantRDD: no visible global function definition for
  ‘sparkR.callJMethod’
toFeatureRDD,CoverageRDD: no visible global function definition for
  ‘sparkR.callJMethod’
toFragments,AlignmentRecordRDD: no visible global function definition
  for ‘sparkR.callJMethod’
toReads,FragmentRDD: no visible global function definition for
  ‘sparkR.callJMethod’
Undefined global functions or variables:
  new sparkR.callJMethod sparkR.callJStatic sparkR.newJObject
Consider adding
  importFrom("methods", "new")
to your NAMESPACE file (and ensure that your DESCRIPTION Imports field
contains 'methods').
* checking for missing documentation entries ... WARNING
Undocumented code objects:
  ‘ADAMContext’ ‘aggregatedCoverage’ ‘countKmers’ ‘coverage’
  ‘flankAdjacentFragments’ ‘flatten’ ‘loadAlignments’
  ‘loadContigFragments’ ‘loadFeatures’ ‘loadFragments’ ‘loadGenotypes’
  ‘loadVariants’ ‘markDuplicates’ ‘realignIndels’
  ‘recalibrateBaseQualities’ ‘save’ ‘sortReadsByReferencePosition’
  ‘sortReadsByReferencePositionAndIndex’ ‘toCoverage’ ‘toDF’
  ‘toFeatureRDD’ ‘toFragments’ ‘toReads’
Undocumented S4 classes:
  ‘ADAMContext’ ‘NucleotideContigFragmentRDD’ ‘VariantRDD’
  ‘GenotypeRDD’ ‘CoverageRDD’ ‘AlignmentRecordRDD’ ‘FragmentRDD’
  ‘FeatureRDD’
Undocumented S4 methods:
  generic 'aggregatedCoverage' and siglist 'CoverageRDD'
  generic 'countKmers' and siglist 'AlignmentRecordRDD,numeric'
  generic 'coverage' and siglist 'CoverageRDD'
  generic 'flankAdjacentFragments' and siglist
    'NucleotideContigFragmentRDD,numeric'
  generic 'flatten' and siglist 'CoverageRDD'
  generic 'loadAlignments' and siglist 'ADAMContext,character'
  generic 'loadContigFragments' and siglist 'ADAMContext,character'
  generic 'loadFeatures' and siglist 'ADAMContext,character'
  generic 'loadFragments' and siglist 'ADAMContext,character'
  generic 'loadGenotypes' and siglist 'ADAMContext,character'
  generic 'loadVariants' and siglist 'ADAMContext,character'
  generic 'markDuplicates' and siglist 'AlignmentRecordRDD'
  generic 'markDuplicates' and siglist 'FragmentRDD'
  generic 'realignIndels' and siglist 'AlignmentRecordRDD,ANY'
  generic 'realignIndels' and siglist 'AlignmentRecordRDD,VariantRDD'
  generic 'recalibrateBaseQualities' and siglist
    'AlignmentRecordRDD,VariantRDD,character'
  generic 'save' and siglist 'AlignmentRecordRDD,character'
  generic 'save' and siglist 'CoverageRDD,ANY'
  generic 'save' and siglist 'CoverageRDD,character'
  generic 'save' and siglist 'FeatureRDD,character'
  generic 'save' and siglist 'FragmentRDD,character'
  generic 'save' and siglist 'GenotypeRDD,character'
  generic 'save' and siglist 'NucleotideContigFragmentRDD,character'
  generic 'save' and siglist 'VariantRDD,character'
  generic 'sortReadsByReferencePosition' and siglist
    'AlignmentRecordRDD'
  generic 'sortReadsByReferencePositionAndIndex' and siglist
    'AlignmentRecordRDD'
  generic 'toCoverage' and siglist 'AlignmentRecordRDD'
  generic 'toCoverage' and siglist 'FeatureRDD'
  generic 'toDF' and siglist 'AlignmentRecordRDD'
  generic 'toDF' and siglist 'CoverageRDD'
  generic 'toDF' and siglist 'FeatureRDD'
  generic 'toDF' and siglist 'FragmentRDD'
  generic 'toDF' and siglist 'GenotypeRDD'
  generic 'toDF' and siglist 'NucleotideContigFragmentRDD'
  generic 'toDF' and siglist 'VariantRDD'
  generic 'toFeatureRDD' and siglist 'CoverageRDD'
  generic 'toFragments' and siglist 'AlignmentRecordRDD'
  generic 'toReads' and siglist 'FragmentRDD'
All user-level objects in a package (including S4 classes and methods)
should have documentation entries.
See chapter ‘Writing R documentation files’ in the ‘Writing R
Extensions’ manual.
* checking examples ... NONE
* checking for unstated dependencies in ‘tests’ ... OK
* checking tests ...
  Running ‘testthat.R’
 ERROR
Running the tests in ‘tests/testthat.R’ failed.
Last 13 lines of output:
  8: stop(readString(conn))
  
  testthat results ================================================================
  OK: 13 SKIPPED: 0 FAILED: 8
  1. Error: load features from GTF (@test_adamContext.R#34) 
  2. Error: load features from BED (@test_adamContext.R#40) 
  3. Error: load features from interval_list (@test_adamContext.R#53) 
  4. Error: load coverage (@test_adamContext.R#57) 
  5. Error: round trip gtf (@test_featureRdd.R#29) 
  6. Error: round trip bed (@test_featureRdd.R#38) 
  7. Error: round trip narrowpeak (@test_featureRdd.R#47) 
  8. Error: round trip interval list (@test_featureRdd.R#56) 
  
  Error: testthat unit tests failed
  Execution halted
* checking PDF version of manual ... WARNING
LaTeX errors when creating PDF version.
This typically indicates Rd problems.
* checking PDF version of manual without hyperrefs or index ... OK
* DONE

Status: 1 ERROR, 2 WARNINGs, 4 NOTEs
...
@fnothaft
Copy link
Member Author

@fnothaft fnothaft commented Jul 20, 2017

Can you pull the latest copy of this branch and retry? There had been a rebase-gone-bad that was apparently causing some tests to fail earlier this week. I've fixed that now.

@devin-petersohn
Copy link
Member

@devin-petersohn devin-petersohn commented Jul 20, 2017

[INFO] Reactor Summary:
[INFO] 
[INFO] ADAM_2.11 .......................................... SUCCESS [  6.707 s]
[INFO] ADAM_2.11: Avro-to-Dataset codegen utils ........... SUCCESS [  5.733 s]
[INFO] ADAM_2.11: Core .................................... SUCCESS [01:26 min]
[INFO] ADAM_2.11: APIs for Java ........................... SUCCESS [  5.504 s]
[INFO] ADAM_2.11: CLI ..................................... SUCCESS [ 18.365 s]
[INFO] ADAM_2.11: Assembly ................................ SUCCESS [  7.772 s]
[INFO] ADAM_2.11: R APIs .................................. SUCCESS [ 44.116 s]

Thank you @fnothaft for your help.

Copy link
Member

@devin-petersohn devin-petersohn left a comment

Looks great! Thanks again @fnothaft for help in getting ADAM's R API built.

methods
Depends:
R (>= 3.0),
SparkR (>= 1.6.0)

This comment has been minimized.

@devin-petersohn

devin-petersohn Jul 21, 2017
Member

Should this be >= 2.0.0? Even in the Markdown docs it is mentioned as Spark 2 or greater.

This comment has been minimized.

@fnothaft

fnothaft Jul 21, 2017
Author Member

Good catch! It should be >= 2.1.0 actually. Let me fix this.

@@ -0,0 +1,25 @@
Package: bdg.adam
Type: Package
Version: 0.23.0

This comment has been minimized.

@heuermh

heuermh Jul 21, 2017
Member

Should we add something to update this in the release process?

This comment has been minimized.

@fnothaft

fnothaft Jul 21, 2017
Author Member

I've added a note for this to scripts/release/README.md.

This comment has been minimized.

@heuermh

heuermh Jul 21, 2017
Member

Thanks!


## Building for R {#r-build}

ADAM supports SparkR, for Spark 2.0.0 and onwards. To build and test [ADAM's R

This comment has been minimized.

@heuermh

heuermh Jul 21, 2017
Member

2.0.0 → 2.1.0 here as well?

Resolves #882. Adds an R API that binds around the ADAM GenomicRDD APIs.
Supports Spark 2.x and onwards, as accessible SparkR functionality for binding
to Java libraries was added in Spark 2.0.0.
@fnothaft fnothaft force-pushed the fnothaft:issues/882-r branch from abcf028 to d3975ba Jul 21, 2017
@coveralls
Copy link

@coveralls coveralls commented Jul 21, 2017

Coverage Status

Coverage remained the same at 84.743% when pulling d3975ba on fnothaft:issues/882-r into 7652536 on bigdatagenomics:master.

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Jul 21, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2253/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1397/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 73cb211e6e0b24257cd5ee6bb4a3645fe21ac5fc # timeout=10Checking out Revision 73cb211e6e0b24257cd5ee6bb4a3645fe21ac5fc (origin/pr/1397/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 73cb211e6e0b24257cd5ee6bb4a3645fe21ac5fc > /home/jenkins/git2/bin/git rev-list 6212a218fbcd2f05e5434def603f8a521677215b # timeout=10Triggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.10,2.1.0,centosTriggering ADAM-prb ? 2.6.0,2.10,2.1.0,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,2.1.0,centosTriggering ADAM-prb ? 2.3.0,2.11,2.1.0,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,2.1.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,2.1.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,2.1.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.11,2.1.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@heuermh
Copy link
Member

@heuermh heuermh commented Jul 21, 2017

Jenkins, retest this please.

@coveralls
Copy link

@coveralls coveralls commented Jul 21, 2017

Coverage Status

Coverage remained the same at 84.743% when pulling d3975ba on fnothaft:issues/882-r into 7652536 on bigdatagenomics:master.

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Jul 21, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2255/
Test PASSed.

@heuermh heuermh merged commit 640c44b into bigdatagenomics:master Jul 21, 2017
3 checks passed
3 checks passed
codacy/pr Good work! A positive pull request.
Details
coverage/coveralls Coverage remained the same at 84.743%
Details
default Merged build finished.
Details
@heuermh
Copy link
Member

@heuermh heuermh commented Jul 21, 2017

Thank you, @fnothaft!

@fnothaft fnothaft mentioned this pull request Jul 30, 2017
@heuermh heuermh moved this from Triage to Completed in Release 0.23.0 Jan 4, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Linked issues

Successfully merging this pull request may close these issues.

None yet

6 participants
You can’t perform that action at this time.