New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generate docs from R API for distribution #1672

Closed
heuermh opened this Issue Aug 18, 2017 · 12 comments

Comments

Projects
2 participants
@heuermh
Member

heuermh commented Aug 18, 2017

No description provided.

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Aug 30, 2017

Member

@pbashyal-nmdp Did you have a suggestion for this? I'm reading through the Writing R Extensions doc and am not really finding what I'm looking for (how to generate docs from source code comments).

Member

heuermh commented Aug 30, 2017

@pbashyal-nmdp Did you have a suggestion for this? I'm reading through the Writing R Extensions doc and am not really finding what I'm looking for (how to generate docs from source code comments).

@pbashyal-nmdp

This comment has been minimized.

Show comment
Hide comment
@pbashyal-nmdp

pbashyal-nmdp Aug 30, 2017

Contributor

The R documentation is written in a format called Roxygen similar to JavaDoc for Java. This documentation needs to be turned into .Rd format for inclusion in the tar.gz build. Easiest way to do that is to use the devtools package to do the conversion.

Here are the steps needed to include documentation in the distribution:

Prereq:

Install SparkR package.

R CMD INSTALL $SPARK_HOME/R/lib/SparkR

Install testthat, devtools and roxygen packages:

R -e "install.packages('testthat', repos='http://cran.rstudio.com/')"
R -e "install.packages('roxygen2', repos='http://cran.rstudio.com/')"
R -e "install.packages('devtools', repos='http://cran.rstudio.com/')"

Build Help(Rd) files

cd adam-r/bdg.adam
R -e "library(devtools);document()"

You should see .Rd files under man directory:

$ ls -la man
total 328
drwxr-xr-x  43 pbashyal  827464065  1462 Aug 30 10:17 ./
drwxr-xr-x  13 pbashyal  827464065   442 Aug 29 13:10 ../
-rw-r--r--   1 pbashyal  827464065   437 Aug 30 10:17 ADAMContext-class.Rd
-rw-r--r--   1 pbashyal  827464065   655 Aug 30 10:17 aggregatedCoverage-CoverageRDD-method.Rd
-rw-r--r--   1 pbashyal  827464065   636 Aug 30 10:17 collapse-CoverageRDD-method.Rd
-rw-r--r--   1 pbashyal  827464065   598 Aug 30 10:17 countKmers-AlignmentRecordRDD-numeric-method.Rd
-rw-r--r--   1 pbashyal  827464065   627 Aug 30 10:17 coverage-CoverageRDD-method.Rd
-rw-r--r--   1 pbashyal  827464065   831 Aug 30 10:17 flankAdjacentFragments-NucleotideContigFragmentRDD-numeric-method.Rd
-rw-r--r--   1 pbashyal  827464065   400 Aug 30 10:17 flatten-CoverageRDD-method.Rd
-rw-r--r--   1 pbashyal  827464065   923 Aug 30 10:17 loadAlignments-ADAMContext-character-method.Rd
-rw-r--r--   1 pbashyal  827464065   816 Aug 30 10:17 loadContigFragments-ADAMContext-character-method.Rd
-rw-r--r--   1 pbashyal  827464065  1023 Aug 30 10:17 loadCoverage-ADAMContext-character-method.Rd
-rw-r--r--   1 pbashyal  827464065   944 Aug 30 10:17 loadFeatures-ADAMContext-character-method.Rd
-rw-r--r--   1 pbashyal  827464065   837 Aug 30 10:17 loadFragments-ADAMContext-character-method.Rd
-rw-r--r--   1 pbashyal  827464065   593 Aug 30 10:17 loadGenotypes-ADAMContext-character-method.Rd
-rw-r--r--   1 pbashyal  827464065   587 Aug 30 10:17 loadVariants-ADAMContext-character-method.Rd
-rw-r--r--   1 pbashyal  827464065   488 Aug 30 10:17 markDuplicates-AlignmentRecordRDD-method.Rd
-rw-r--r--   1 pbashyal  827464065   460 Aug 30 10:17 markDuplicates-FragmentRDD-method.Rd

...

Build the tar.gz distribution file

This produces the bdg.adam_0.23.0.tar.gz distribution file

cd ..
R CMD build bdg.adam

Build and install the tar.gz distribution file

Install bdg.adam package from bdg.adam_0.23.0.tar.gz

R CMD INSTALL bdg.adam_0.23.0.tar.gz

The help should be available from R. You have to specify the whole class as well as most of them are class methods.

$ R
> library(bdg.adam)
Loading required package: SparkR

Attaching package: ‘SparkR’

The following objects are masked from ‘package:stats’:

    cov, filter, lag, na.omit, predict, sd, var, window

The following objects are masked from ‘package:base’:

    as.data.frame, colnames, colnames<-, drop, endsWith, intersect,
    rank, rbind, sample, startsWith, subset, summary, transform, union


Attaching package: ‘bdg.adam’

The following object is masked from ‘package:SparkR’:

    transform

The following objects are masked from ‘package:base’:

    pipe, save, sort, transform

> ?`loadVariants,ADAMContext,character-method`

loadVariants,ADAMContext,character-methodpackage:bdg.adamR Documentation

Load variants into a VariantRDD.

Description:

     If the path name has a .vcf/.vcf.gz/.vcf.bgz extension, load as
     VCF format. Else, fall back to Parquet + Avro.

Usage:

     ## S4 method for signature 'ADAMContext,character'
     loadVariants(ac, filePath)

Arguments:

      ac: The ADAMContext.

filePath: The path to load the file from.

Value:

     Returns an RDD containing variants.


> ?`transform,GenomicRDD,function-method`

transform,GenomicRDD,function-method package:bdg.adam  R Documentation

Applies a function that transforms the underlying DataFrame into a new
DataFrame using the Spark SQL API.

Description:

     Applies a function that transforms the underlying DataFrame into a
     new DataFrame using the Spark SQL API.

Usage:

     ## S4 method for signature 'GenomicRDD,`function`'
     transform(ardd, tFn)

Arguments:

     tFn: A function that transforms the underlying RDD as a DataFrame.

Value:

     A new RDD where the RDD of genomic data has been replaced, but the
     metadata (sequence dictionary, and etc) is copied without
     modification.

Contributor

pbashyal-nmdp commented Aug 30, 2017

The R documentation is written in a format called Roxygen similar to JavaDoc for Java. This documentation needs to be turned into .Rd format for inclusion in the tar.gz build. Easiest way to do that is to use the devtools package to do the conversion.

Here are the steps needed to include documentation in the distribution:

Prereq:

Install SparkR package.

R CMD INSTALL $SPARK_HOME/R/lib/SparkR

Install testthat, devtools and roxygen packages:

R -e "install.packages('testthat', repos='http://cran.rstudio.com/')"
R -e "install.packages('roxygen2', repos='http://cran.rstudio.com/')"
R -e "install.packages('devtools', repos='http://cran.rstudio.com/')"

Build Help(Rd) files

cd adam-r/bdg.adam
R -e "library(devtools);document()"

You should see .Rd files under man directory:

$ ls -la man
total 328
drwxr-xr-x  43 pbashyal  827464065  1462 Aug 30 10:17 ./
drwxr-xr-x  13 pbashyal  827464065   442 Aug 29 13:10 ../
-rw-r--r--   1 pbashyal  827464065   437 Aug 30 10:17 ADAMContext-class.Rd
-rw-r--r--   1 pbashyal  827464065   655 Aug 30 10:17 aggregatedCoverage-CoverageRDD-method.Rd
-rw-r--r--   1 pbashyal  827464065   636 Aug 30 10:17 collapse-CoverageRDD-method.Rd
-rw-r--r--   1 pbashyal  827464065   598 Aug 30 10:17 countKmers-AlignmentRecordRDD-numeric-method.Rd
-rw-r--r--   1 pbashyal  827464065   627 Aug 30 10:17 coverage-CoverageRDD-method.Rd
-rw-r--r--   1 pbashyal  827464065   831 Aug 30 10:17 flankAdjacentFragments-NucleotideContigFragmentRDD-numeric-method.Rd
-rw-r--r--   1 pbashyal  827464065   400 Aug 30 10:17 flatten-CoverageRDD-method.Rd
-rw-r--r--   1 pbashyal  827464065   923 Aug 30 10:17 loadAlignments-ADAMContext-character-method.Rd
-rw-r--r--   1 pbashyal  827464065   816 Aug 30 10:17 loadContigFragments-ADAMContext-character-method.Rd
-rw-r--r--   1 pbashyal  827464065  1023 Aug 30 10:17 loadCoverage-ADAMContext-character-method.Rd
-rw-r--r--   1 pbashyal  827464065   944 Aug 30 10:17 loadFeatures-ADAMContext-character-method.Rd
-rw-r--r--   1 pbashyal  827464065   837 Aug 30 10:17 loadFragments-ADAMContext-character-method.Rd
-rw-r--r--   1 pbashyal  827464065   593 Aug 30 10:17 loadGenotypes-ADAMContext-character-method.Rd
-rw-r--r--   1 pbashyal  827464065   587 Aug 30 10:17 loadVariants-ADAMContext-character-method.Rd
-rw-r--r--   1 pbashyal  827464065   488 Aug 30 10:17 markDuplicates-AlignmentRecordRDD-method.Rd
-rw-r--r--   1 pbashyal  827464065   460 Aug 30 10:17 markDuplicates-FragmentRDD-method.Rd

...

Build the tar.gz distribution file

This produces the bdg.adam_0.23.0.tar.gz distribution file

cd ..
R CMD build bdg.adam

Build and install the tar.gz distribution file

Install bdg.adam package from bdg.adam_0.23.0.tar.gz

R CMD INSTALL bdg.adam_0.23.0.tar.gz

The help should be available from R. You have to specify the whole class as well as most of them are class methods.

$ R
> library(bdg.adam)
Loading required package: SparkR

Attaching package: ‘SparkR’

The following objects are masked from ‘package:stats’:

    cov, filter, lag, na.omit, predict, sd, var, window

The following objects are masked from ‘package:base’:

    as.data.frame, colnames, colnames<-, drop, endsWith, intersect,
    rank, rbind, sample, startsWith, subset, summary, transform, union


Attaching package: ‘bdg.adam’

The following object is masked from ‘package:SparkR’:

    transform

The following objects are masked from ‘package:base’:

    pipe, save, sort, transform

> ?`loadVariants,ADAMContext,character-method`

loadVariants,ADAMContext,character-methodpackage:bdg.adamR Documentation

Load variants into a VariantRDD.

Description:

     If the path name has a .vcf/.vcf.gz/.vcf.bgz extension, load as
     VCF format. Else, fall back to Parquet + Avro.

Usage:

     ## S4 method for signature 'ADAMContext,character'
     loadVariants(ac, filePath)

Arguments:

      ac: The ADAMContext.

filePath: The path to load the file from.

Value:

     Returns an RDD containing variants.


> ?`transform,GenomicRDD,function-method`

transform,GenomicRDD,function-method package:bdg.adam  R Documentation

Applies a function that transforms the underlying DataFrame into a new
DataFrame using the Spark SQL API.

Description:

     Applies a function that transforms the underlying DataFrame into a
     new DataFrame using the Spark SQL API.

Usage:

     ## S4 method for signature 'GenomicRDD,`function`'
     transform(ardd, tFn)

Arguments:

     tFn: A function that transforms the underlying RDD as a DataFrame.

Value:

     A new RDD where the RDD of genomic data has been replaced, but the
     metadata (sequence dictionary, and etc) is copied without
     modification.

@pbashyal-nmdp

This comment has been minimized.

Show comment
Hide comment
@pbashyal-nmdp

pbashyal-nmdp Aug 30, 2017

Contributor

Here's how it looks in RStudio

bdg.adam Index Help Page

adam-help-index

bdg.adam loadVariants Help Page

adam-help-loadvariants

Contributor

pbashyal-nmdp commented Aug 30, 2017

Here's how it looks in RStudio

bdg.adam Index Help Page

adam-help-index

bdg.adam loadVariants Help Page

adam-help-loadvariants

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Aug 30, 2017

Member

Thanks!

Trying to follow along, stuck at

$ R -e "install.packages('roxygen2', repos='http://cran.rstudio.com/')"
...
clang++ -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/usr/local/opt/gettext/lib -L/usr/local/opt/readline/lib -L/usr/local/lib -L/usr/local/Cellar/r/3.4.1_2/lib/R/lib -L/usr/local/opt/gettext/lib -L/usr/local/opt/readline/lib -L/usr/local/lib -o xml2.so RcppExports.o connection.o xml2_doc.o xml2_init.o xml2_namespace.o xml2_node.o xml2_output.o xml2_schema.o xml2_url.o xml2_xpath.o -L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/usr/lib -lxml2 -lz -lpthread -licucore -lm -L/usr/local/Cellar/r/3.4.1_2/lib/R/lib -lR -lintl -Wl,-framework -Wl,CoreFoundation
ld: file not found: /usr/lib/system/libsystem_symptoms.dylib for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [xml2.so] Error 1
ERROR: compilation failed for package ‘xml2’
* removing ‘/usr/local/lib/R/3.4/site-library/xml2’
* installing *source* package ‘desc’ ...
** package ‘desc’ successfully unpacked and MD5 sums checked
** R
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (desc)
ERROR: dependency ‘xml2’ is not available for package ‘roxygen2’
* removing ‘/usr/local/lib/R/3.4/site-library/roxygen2’

The downloaded source packages are in
	‘/private/var/folders/m6/4yqn_4q129lbth_dq3qzj_8h0000gn/T/Rtmpcyo9eZ/downloaded_packages’
Warning messages:
1: In install.packages("roxygen2", repos = "http://cran.rstudio.com/") :
  installation of package ‘xml2’ had non-zero exit status
2: In install.packages("roxygen2", repos = "http://cran.rstudio.com/") :
  installation of package ‘roxygen2’ had non-zero exit status
> 
> 
Member

heuermh commented Aug 30, 2017

Thanks!

Trying to follow along, stuck at

$ R -e "install.packages('roxygen2', repos='http://cran.rstudio.com/')"
...
clang++ -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/usr/local/opt/gettext/lib -L/usr/local/opt/readline/lib -L/usr/local/lib -L/usr/local/Cellar/r/3.4.1_2/lib/R/lib -L/usr/local/opt/gettext/lib -L/usr/local/opt/readline/lib -L/usr/local/lib -o xml2.so RcppExports.o connection.o xml2_doc.o xml2_init.o xml2_namespace.o xml2_node.o xml2_output.o xml2_schema.o xml2_url.o xml2_xpath.o -L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/usr/lib -lxml2 -lz -lpthread -licucore -lm -L/usr/local/Cellar/r/3.4.1_2/lib/R/lib -lR -lintl -Wl,-framework -Wl,CoreFoundation
ld: file not found: /usr/lib/system/libsystem_symptoms.dylib for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [xml2.so] Error 1
ERROR: compilation failed for package ‘xml2’
* removing ‘/usr/local/lib/R/3.4/site-library/xml2’
* installing *source* package ‘desc’ ...
** package ‘desc’ successfully unpacked and MD5 sums checked
** R
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (desc)
ERROR: dependency ‘xml2’ is not available for package ‘roxygen2’
* removing ‘/usr/local/lib/R/3.4/site-library/roxygen2’

The downloaded source packages are in
	‘/private/var/folders/m6/4yqn_4q129lbth_dq3qzj_8h0000gn/T/Rtmpcyo9eZ/downloaded_packages’
Warning messages:
1: In install.packages("roxygen2", repos = "http://cran.rstudio.com/") :
  installation of package ‘xml2’ had non-zero exit status
2: In install.packages("roxygen2", repos = "http://cran.rstudio.com/") :
  installation of package ‘roxygen2’ had non-zero exit status
> 
> 
@pbashyal-nmdp

This comment has been minimized.

Show comment
Hide comment
@pbashyal-nmdp

pbashyal-nmdp Aug 31, 2017

Contributor

I missed the xml2 dependencies. You can install it separately

R -e "install.packages('xml2', repos='http://cran.rstudio.com/')"

or install roxygen with dependencies set to TRUE.

R -e "install.packages('roxygen2', repos='http://cran.rstudio.com/', dependencies = TRUE)"
Contributor

pbashyal-nmdp commented Aug 31, 2017

I missed the xml2 dependencies. You can install it separately

R -e "install.packages('xml2', repos='http://cran.rstudio.com/')"

or install roxygen with dependencies set to TRUE.

R -e "install.packages('roxygen2', repos='http://cran.rstudio.com/', dependencies = TRUE)"
@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Aug 31, 2017

Member

Thanks, I tried that, the error is due to some missing lib on OSX at compile time.

Member

heuermh commented Aug 31, 2017

Thanks, I tried that, the error is due to some missing lib on OSX at compile time.

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Sep 6, 2017

Member

Looks like I either have to upgrade to OSX 10.12 or figure out how to
"The easiest way is just installing the binary package from CRAN."

Nope, there aren't any binary packages available for version 3.4, apparently

$ R -e "install.packages('xml2', repos='http://cran.rstudio.com/', type='mac.binary')"

R version 3.4.1 (2017-06-30) -- "Single Candle"
Copyright (C) 2017 The R Foundation for Statistical Computing
Platform: x86_64-apple-darwin15.6.0 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> install.packages('xml2', repos='http://cran.rstudio.com/', type='mac.binary')
Installing package into ‘/usr/local/lib/R/3.4/site-library’
(as ‘lib’ is unspecified)
Warning: unable to access index for repository http://cran.rstudio.com/bin/macosx/contrib/3.4:
  cannot open URL 'http://cran.rstudio.com/bin/macosx/contrib/3.4/PACKAGES'

   package ‘xml2’ is available as a source package but not as a binary

Warning message:
package ‘xml2’ is not available (as a binary package for R version 3.4.1) 
Member

heuermh commented Sep 6, 2017

Looks like I either have to upgrade to OSX 10.12 or figure out how to
"The easiest way is just installing the binary package from CRAN."

Nope, there aren't any binary packages available for version 3.4, apparently

$ R -e "install.packages('xml2', repos='http://cran.rstudio.com/', type='mac.binary')"

R version 3.4.1 (2017-06-30) -- "Single Candle"
Copyright (C) 2017 The R Foundation for Statistical Computing
Platform: x86_64-apple-darwin15.6.0 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> install.packages('xml2', repos='http://cran.rstudio.com/', type='mac.binary')
Installing package into ‘/usr/local/lib/R/3.4/site-library’
(as ‘lib’ is unspecified)
Warning: unable to access index for repository http://cran.rstudio.com/bin/macosx/contrib/3.4:
  cannot open URL 'http://cran.rstudio.com/bin/macosx/contrib/3.4/PACKAGES'

   package ‘xml2’ is available as a source package but not as a binary

Warning message:
package ‘xml2’ is not available (as a binary package for R version 3.4.1) 
@pbashyal-nmdp

This comment has been minimized.

Show comment
Hide comment
@pbashyal-nmdp

pbashyal-nmdp Sep 6, 2017

Contributor

Hmm.. What OS version are you running?

I'm running MacOS X 10.10.5 and R 3.3.

Contributor

pbashyal-nmdp commented Sep 6, 2017

Hmm.. What OS version are you running?

I'm running MacOS X 10.10.5 and R 3.3.

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Sep 6, 2017

Member

10.11, see issue 127 at https://github.com/r-lib/xml2. I'm currently upgrading to 10.12...

Member

heuermh commented Sep 6, 2017

10.11, see issue 127 at https://github.com/r-lib/xml2. I'm currently upgrading to 10.12...

@pbashyal-nmdp

This comment has been minimized.

Show comment
Hide comment
@pbashyal-nmdp

pbashyal-nmdp Sep 6, 2017

Contributor

lucky you ;)

Contributor

pbashyal-nmdp commented Sep 6, 2017

lucky you ;)

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Nov 7, 2017

Member

Fixed in #1765.

Member

heuermh commented Nov 7, 2017

Fixed in #1765.

@heuermh heuermh closed this Nov 7, 2017

@heuermh heuermh reopened this Nov 7, 2017

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Nov 7, 2017

Member

@pbashyal-nmdp can you review whether #1765 works for you (now merged to git head)? While testing #1651 it doesn't look to me like the R docs were included in the tarball.

Member

heuermh commented Nov 7, 2017

@pbashyal-nmdp can you review whether #1765 works for you (now merged to git head)? While testing #1651 it doesn't look to me like the R docs were included in the tarball.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment