Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Non-R system requirements #20

Open
wlandau opened this issue Jun 22, 2019 · 11 comments
Open

Non-R system requirements #20

wlandau opened this issue Jun 22, 2019 · 11 comments

Comments

@wlandau
Copy link

@wlandau wlandau commented Jun 22, 2019

holepunch is magical! I am using it for https://github.com/wlandau/learndrake, and it makes the workshop materials so much more accessible.

The setup was mostly smooth, but I hit a snag because of the dependency on TensorFlow. rocker/binder omits TensorFlow (and rightly so) so learndrake's own Dockerfile borrows heavily from rocker/tensorflow. It works now, but it took some micromanaging to go through apt-get and pip3 and make sure to invoke the correct user for each.

I an wondering if we can or should do anything in holepunch's API to abstract away this sort of thing. Maybe have arguments in holepunch::write_dockerfile() to supply apt and python dependencies? Or maybe a .holepunch.yml file that borrows from .travis.yml?

@karthik

This comment has been minimized.

Copy link
Owner

@karthik karthik commented Jun 22, 2019

👋@wlandau!
Thanks for the kind comments!

The setup was mostly smooth, but I hit a snag because of the dependency on TensorFlow. rocker/binder omits TensorFlow (and rightly so) so learndrake's own Dockerfile borrows heavily from rocker/tensorflow. It works now, but it took some micromanaging to go through apt-get and pip3 and make sure to invoke the correct user for each.

Supporting Tensorflow is very much on the radar. For now I'm focusing on getting the first version up before working on more complex use cases such as ML and geospatial.

I an wondering if we can or should do anything in holepunch's API to abstract away this sort of thing. Maybe have arguments in holepunch::write_dockerfile() to supply apt and python dependencies? Or maybe a .holepunch.yml file that borrows from .travis.yml?

I was waiting to introduce support for apt/pip in the next release but you nudged me earlier with this request 😀. I have now pushed support for write_apt() and write_requirements(). Both are as self explanatory as they sound. So your workflow can now be:

write_compendium_description
write_apt(apt_input = "
texlive-latex-base
texlive-latex-recommended
texlive-science
texlive-latex-extra
texlive-fonts-recommended
dvipng
ghostscript")

# and then 

write_requirements(requirements = "
matplotlib==3.*
seaborn==0.8.1
")

write_dockerfile(...)

Can you install the latest (0.1.13.9000) and let me know if that improves the workflow?

karthik added a commit that referenced this issue Jun 22, 2019
Now a user can run: `write_apt` to install Debian dependencies, and `write_requirements` to install Python packages, before creating a Dockerfile that replies on these. Still hasn’t been tested with a real workflow. #20
karthik added a commit that referenced this issue Jun 22, 2019
@karthik

This comment has been minimized.

Copy link
Owner

@karthik karthik commented Jun 22, 2019

Also worth noting that the Rocker team is planning to support Tensorflow dailies soon so a end user might not have to mess too much with apt and requirements.

@karthik

This comment has been minimized.

Copy link
Owner

@karthik karthik commented Jun 22, 2019

I spoke too soon. Having a Dockerfile will likely just override anything in apt and requirements. 😞

I'm hoping to introduce reference environments in a future release, where a downstream user will not have to build their own Dockerfile/Binder setup. They can point to someone else's binder [for compute environment] and their own repo [for code].

karthik added a commit that referenced this issue Jun 22, 2019
@wlandau

This comment has been minimized.

Copy link
Author

@wlandau wlandau commented Jun 23, 2019

Glad to hear you are working on easier configuration! I have added a new branch to learndrake for testing holepunch, and this is my latest attempt. I look forward to more tinkering.

@betatim

This comment has been minimized.

Copy link
Contributor

@betatim betatim commented Jul 27, 2019

@wlandau might be worth to look at using repo2docker directly which supports apt.txt and requirements.txt. This is the tool that powers mybinder.org and you can use it locally as well to build a docker image from your repo. A list of all the config files supported

If you install repo2docker directly from GitHub rebuilds of your repo that don't change the dependencies should be instant. This is a new feature for the upcoming release!

@wlandau

This comment has been minimized.

Copy link
Author

@wlandau wlandau commented Jul 28, 2019

Awesome! I will check it out when I get the chance.

@pat-s

This comment has been minimized.

Copy link

@pat-s pat-s commented Aug 20, 2019

Just letting you know that the direct way via repo2docker works by providing a apt.txt file in combination with runtime.txt. Thanks @betatim!
See giscience-fsu/daad_summerschool_hyperspectral for an example.

The downside is that the R version is not configurable right now and the current default is R 3.4.4.

@betatim

This comment has been minimized.

Copy link
Contributor

@betatim betatim commented Aug 20, 2019

jupyter/repo2docker#661 is waiting for who ever wants to see being able to change R versions enough that they tackle it ;) The point being that mybinder.org and everything that goes into it might look like a professional effort but in reality there are two or three people (and their spare time) behind it. This means the bottleneck is time to discuss and come to consensus and then implement things -> all help even at a low level is welcome.

@nuest

This comment has been minimized.

Copy link

@nuest nuest commented Sep 19, 2019

@karthik (as you likely know) The containerit R package creates the required RUN statements to install the apt packages needed by R packages as part of the Dockerfile, based on sysreqs. Wholepunch could do the same copying some code, or we could figure out how to expose the required functions from containerit.

With the new support for R versions in runtime.txt, did you consider not generating the Dockerfile but generating the different configuration files supported by repo2docker (install.R, runtime.txt, apt.txt) ?

@karthik

This comment has been minimized.

Copy link
Owner

@karthik karthik commented Oct 18, 2019

With the new support for R versions in runtime.txt, did you consider not generating the Dockerfile but generating the different configuration files supported by repo2docker (install.R, runtime.txt, apt.txt) ?

Both options are currently possible in holepunch as outlined in the readme and vignettes. One can either go for the Dockerfile approach, or the install + runtime + apt approach. The downside of the latter is just the long installation time.

I could certainly fold the apt.txt into the Dockerfile if both are present.

@nuest

This comment has been minimized.

Copy link

@nuest nuest commented Oct 24, 2019

Great, thanks for the clarification - I need to play around more with wholepunch obviously.

So what I'd suggest is to streamline something like this in a way that works for the given DESCRIPTION file but also for all required R packages:

> holepunch::write_apt(apt_input = sysreqs::sysreqs(desc = file.path(path.package(package = "openssl"), "DESCRIPTION"), platform="linux-x86_64-ubuntu-gcc"))
> readLines(".binder/apt.txt")
[1] "libssl-dev"             "libssl-dev"             "git-core"               "pandoc pandoc-citeproc" "pandoc pandoc-citeproc" "libsodium-dev"    

In case of write_dockerfile(..), sysreqs can deliver the actual command:

> sysreqs::sysreq_commands(desc = file.path(path.package(package = "openssl"), "DESCRIPTION"), platform="linux-x86_64-ubuntu-gcc")
[1] "export DEBIAN_FRONTEND=noninteractive; apt-get -y update && apt-get install -y libssl-dev git-core pandoc pandoc-citeproc libsodium-dev"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.