Add soils-related CRAN packages to R conda env file

[emiliom]

@lsetiawan & ocefpaf, I'm anticipating wanting to add two R packages dealing with soils data by the end of tomorrow. Both are from @dylanbeaudette (or he is one of the main contributors).

• aqp. From Dylan: "over the weekend I pushed the latest version of the "aqp" package to CRAN."
• soilDB. From Dylan: "The latest soilDB package may make it to CRAN today or tomorrow."

I have no clue if including them in the conda env file involves just a reference on the env yml file and other magic @lsetiawan is doing, or if it's much more involved and involves setting up new conda-forge recipes. If it's the latter and it involves a lot of work, it's probably not worth rushing it for the workshop.

Regardless, it's not a priority for today. Let's see what we learn today and where things stand by tomorrow morning.

@dylanbeaudette, please let us know via this github issue when soilDB has been updated on CRAN.

Thanks, all!

[dylanbeaudette]

Howdy,

Here is what I generally recommend:

1. install aqp, soilDB, and sharpshootR (AQP suite) from CRAN (this will satisfy most dependencies)
2. install latest version of soilDB from GitHub if it isn't on CRAN today:

devtools::install_github("ncss-tech/soilDB", dependencies=FALSE, upgrade_dependencies=FALSE)

3. install latest sharpshootR from GitHub (latest version won't be on CRAN this week):

devtools::install_github("ncss-tech/sharpshootR", dependencies=FALSE, upgrade_dependencies=FALSE)

4. just in case, install the following packages from CRAN which are used by the AQP suite:

xml2, curl, dismo, rgdal, jsonlite, RODBC, httr, rgeos, rvest, e1071

These should be picked up by the dependency-resolver used by install.packages(), but some may have slipped through the cracks as we are mixing stable / development versions here.

[emiliom]

Thanks, @dylanbeaudette

@lsetiawan and @ocefpaf: As you know I have no clue how much work this is. No rush, specially today. @dylanbeaudette doesn't actually need it for his presentation -- it's just "nice" to have.

I suspect we'll want to make our lives easier for now and just focus on deploying packages that are on CRAN (ie, skip sharpshooterR this week, or load the old version from CRAN).

[dylanbeaudette]

Correct. My presentation and any demos that I do from my machine will be self-contained. Many of the tutorials on the AQP website depend on "recent" versions of the packages. There has been so much churn over the last year that we hadn't got around to submitting anything to CRAN! Before the end of the month, a "modern" version of all three packages will be on CRAN.

[lsetiawan]

@emiliom and @dylanbeaudette I have successfully installed aqp, soilDB, and sharpshootR packages into our jupyterhub from CRAN. If you could try it out, it would be awesome. Thanks!

[dylanbeaudette]

Hmm, I just tried this:

library(aqp)
Error in library(aqp): there is no package called ‘aqp’
Traceback:

1. library(aqp)
2. stop(txt, domain = NA)

Any ideas?

[lsetiawan]

@dylanbeaudette Hmm.. I just logged into your instance, and all seems well. Go ahead and try again tomorrow. Thanks.

[dylanbeaudette]

That is good. I had tried it from the Jupyter notebook. Perhaps I am missing a critical step?

[lsetiawan]

You can try shutting down the server, then click the logout button on top right, this will force the server to recreate the environment.

[emiliom]

@emiliom and @dylanbeaudette I have successfully installed aqp, soilDB, and sharpshootR packages into our jupyterhub from CRAN. If you could try it out, it would be awesome.

Thanks, @lsetiawan! Wow. But I won't try it out yet. I don' t know how -- no R skills here 😞 But seriously, my thought is that we'll only try it after @dylanbeaudette's presentation.

I do have one question, though (again, my conda R ignorance here). The R conda env file shows aqp, soilDB, etc, in commented lines. How can those lines be doing anything? Or is all the magic actually in the Docker file, which is only used in the JupyterHub server setting? ie, a user who tries to create a local conda env from renvironment.yml won't actually succeed?

Again, this is NOT a priority right now! But we do want to enable local deployments, if possible (i.e., if it's fairly easy), by Wednesday afternoon. If not doable, so be it, it's not a big problem. I just want to understand what we have.

[emiliom]

FYI, I've confirmed just now that the following statements work on my JupyterHub using the R conda environment:

library(aqp)
library(soilDB)
library(sharpshootR)

Or at least they don't produce errors that are obvious to an R non-user like me.

[ocefpaf]

@emiliom what @lsetiawan did, manually installing the packages, is probably the only way forward if people need those software in the jupyterhub instance. I created the conda packages but that won't help much b/c they are based on released version and not the latest GitHub code.

PS: unfortunately CRAN has a policy that is quite odd in the OSS world. They penalized people package with a higher frequency of releases, forcing most users to resort to GitHub installs, unfortunately that also hurt reproducibility b/c it makes installation harder to track and demands more expertise from the users to do so... Sorry for the rant 😬

[lsetiawan]

@emiliom what @lsetiawan did, manually installing the packages, is probably the only way forward if people need those software in the jupyterhub instance. I created the conda packages but that won't help much b/c they are based on released version and not the latest GitHub code.

Wow! You're fast @ocefpaf. This is great! I actually installed from CRAN latest release rather than latest GitHub code 😛 So it's similar to your packages on conda. I might try to install from latest GitHub code later.

Also, while you're at it, could you put WaterML into conda please? 😸 https://cran.r-project.org/package=WaterML Thanks!

[ocefpaf]

I actually installed from CRAN latest release rather than latest GitHub code 😛 So it's similar to your packages on conda.

Yep. But if people need the latest package, which for r that is usually true, the'll need the GitHub dev install.

Also, while you're at it, could you put WaterML into conda please?

See conda-forge/staged-recipes#4416 😉

[emiliom]

Thanks, guys. Yikes, I didn't know about issues with CRAN. Again, not an R person.

@lsetiawan, I think @ocefpaf answered my question, but can you confirm this:

I do have one question, though (again, my conda R ignorance here). The R conda env file shows aqp, soilDB, etc, in commented lines. How can those lines be doing anything? Or is all the magic actually in the Docker file, which is only used in the JupyterHub server setting? ie, a user who tries to create a local conda env from renvironment.yml won't actually succeed?

And gosh, I guess I won't tire of repeating this: we're not trying to create a perfect setup for R users. None of us in the core ODM2 gang use R, and we don't have the resources to make it a priority. We're only trying to provide something reasonably useful with a reasonable/small amount of work, at least for this week. (and afterwards BiGCZ funds end, so it's even sketchier).

[ocefpaf]

Yikes, I didn't know about issues with CRAN. Again, not an R person.

Yep. They are against RERO 😒

And gosh, I guess I won't tire of repeating this: we're not trying to create a perfect setup for R users. None of us in the core ODM2 gang use R, and we don't have the resources to make it a priority.

The addition to conda-forge is easy and makes @lsetiawan lifes easier by reducing the number of steps in the jupyterhub setup and consolidating everything in the env file. So no reason not to do it 😄

We're only trying to provide something reasonably useful with a reasonable/small amount of work, at least for this week. (and afterwards BiGCZ funds end, so it's even sketchier).

Acknowledged! BTW I don't think pursuing the GitHub install are worth it at this point.

[lsetiawan]

Acknowledged! BTW I don't think pursuing the GitHub install are worth it at this point.

I actually did it already, only took few seconds since @dylanbeaudette already gave me the command on how to install it on #14 (comment).

The addition to conda-forge is easy and makes @lsetiawan lifes easier by reducing the number of steps in the jupyterhub setup and consolidating everything in the env file. So no reason not to do it

This is true. Thanks again for the help.

The R conda env file shows aqp, soilDB, etc, in commented lines.

The commented lines are no magic, I am listing them there to show R user to install those things manually.

[emiliom]

The addition to conda-forge is easy and makes @lsetiawan lifes easier by reducing the number of steps in the jupyterhub setup and consolidating everything in the env file. So no reason not to do it

This is true. Thanks again for the help.

Got it, thanks.

The R conda env file shows aqp, soilDB, etc, in commented lines.

The commented lines are no magic, I am listing them there to show R user to install those things manually.

Ah ok. I doubt anyone would look in the env file for manual instructions, but for now that's a reference for us. Thanks. That answers my question.

@ocefpaf, thanks also for the background on CRAN policy vs github dev. Interesting. I didn't know that.

[ocefpaf]

The commented lines are no magic, I am listing them there to show R user to install those things manually.

Indeed. If they are "pure" r it is no big deal. If they require compilation then it is quite hard.

[dylanbeaudette]

Thanks for working on all of this. I suggested using the development versions of soilDB and sharpshootR because my talk and online examples require these versions.