Specifying multiple languages in a single environment

[BoPeng]

The SoS kernel allows the use of multiple Jupyter kernels in a single Jupyter notebook. When I tried to set up an environment that contains multiple languages (e.g. Julia and R) for a binder example (binder-examples/jupyter-sos), I noticed that runtime.txt is ignored with the presence of REQUIRE. It turned out that I could use configuration files for simple cases (e.g. environment.yml and REQUIRE for python-julia, and environment.yml and runtime.txt for python-r), but I had to use a Dockerfile to provide more kernels.

It would be nice if repo2docker could allow the use of more configuration files simultaneously in a single environment. There can be compatibility issues between different languages, but perhaps certain level of isolation (e.g. put interpreters and their dependencies to their own directories) could help.

[betatim]

Nice to see your work on multi-languaging!

I think the current "leader" for most different languages in one image is https://github.com/binder-examples/multi-language-demo. I'd look how things are solved there and see what you can learn and/or contribute to that example.

I am -1 on making repo2docker (the software) more complex to facilitate extreme multi-languaging in one repository. If there are some simple, low maintenance changes we can make to help with multiple languages we should discuss those.

Going beyond two or maybe three languages in one repository doesn't seem to be something a large number of people do in practice. Keeping things simple and mostly "zero config" for the common use cases at the expense of having to use a Dockerfile for the rarer "all the languages in one repo" case seems like the right trade off.

It could be that I am missing something here, if so please tell me!

[BoPeng]

The multi-language-demo essentially uses an ipython kernel and a number of cross-language modules (e.g. rpy) and magics to talk to other languages. Basically it uses an environment with a single kernel, multiple interpreters and magics installed through apt, conda etc. I am having some trouble figuring out how R was installed though, because there is no runtime.txt or install.R, and the only thing relevant seems to be rpy2 in environment.yml.

The SoS approach allows the use of multiple kernels in one notebook, orchestrated by the SoS kernel, There is no need to learn multiple inter-language modules because SoS provides a universal way to exchange variables between live kernels of supported languages. This means multiple kernels need to be set up to be used in a SoS environment.

I understand the difficulties to provide extreme multi-languaging in one repository. However, given that repo2docker currently only supports Julia, R, and Python natively, and the python+Julia and python+r combinations already work, we are talking about one specific question here: why runtime.txt is ignored in the presence of REQUIRE? It could be fixed if it is a bug, or be documented if it is a limitation by design.

[choldgraf]

To be fair, composability is one of the primary design principles of repo2docker. From that design page:

Although other projects, like s2i, exist to convert source to Docker images, repo2docker provides the additional functionality to support composable environments. We want to easily have an image with Python3+Julia+R-3.2 environments, rather than just one single language environment. While generally one language environment per container works well, in many scientific / datascience computing environments you need multiple languages working together to get anything done. So all buildpacks are composable, and need to be able to work well with other languages.

[betatim]

we are talking about one specific question here: why runtime.txt is ignored in the presence of REQUIRE?

We try build packs in turn until one of them matches. Currently the Julia build pack gets tried before the R one. A build pack can inherit from others and call its parents classes methods to collect scripts/steps for assembling the image. Both the Julia and R build packs inherit from the conda build pack, this means neither is a parent of the other.

Experimenting with changing how things are composed by inheritance would not change how users specify things so it is worth experimenting with.

[kkmann]

+1 to making R+Julia work without resorting to postBuild.

edit: conda doesn't cover cases where you need to install a development verion of a package...