Running Pluto in Docker Container

[lungben]

Hi,

I have just seen Pluto and it looks very nice!
Here is a simple Dockerfile and startup script which allows to run Pluto in a Docker container.

[fonsp]

Awesome, thanks for sharing! Look forward to try it out tomorrow :)

[lungben]

I am just going through the samples. Probably it would be good to include PlutoUI.jl and Plots.jl directly into the Container to have them readily available.
What do you think?

[fonsp]

Sorry for the late response, JuliaCon is going on!

I did try it out yesterday and it worked wonderfully! I also found that Pluto works with binder, and I have earlier managed to get it working on heroku. So maybe we should have a deployment folder, with this one as a subfolder? Right now heroku and binder would mostly be some instruction .md files.

What do you think?

[lungben]

Great JuliaCon talk :-)

That sounds good!

Currently, only PlutoUI and Plots are included into the Docker container. In principle, additional packages like CSV and DataFrames could be included, too, but this would make the container image larger.
Assuming that you cannot directly access Julia inside the container, is possible / advised to install packages in Pluto notebooks directly using Pkg (sorry, I do not have a machine at hand to test it)?

In principle, a sysimage could be created for the Docker image with PackageCompiler.jl to reduce the "1st time to plot". But I do not have any experience with it and do not know if Pluto benefits from it.

[fonsp]

Pluto would definitely benefit! I have worked with PackageCompiler once, we can probably figure it out together!

Docker images can stack right? Maybe there are some "Data Science" docker images already out there - we can layer Pluto on top of it.

I will come back to the package question.

As for this PR - I recently found a number of other cool reasons to create a package PlutoUtils.jl, including
https://github.com/vdayanand/Jupyter2Pluto.jl
and
JuliaCLI/PlutoCLI.jl#2
and the deployments I mentioned before.

[Roger-luo]

I have found out how to build and depoly system images on github actions (using it for IonCLI), so users don't have to build them by themselves but just download it, I find this make installation faster. I could set this up once PlutoUtils.jl is created.

[elbosso]

Hi,

I have just seen Pluto and it looks very nice!
Here is a simple Dockerfile and startup script which allows to run Pluto in a Docker container.

Damn, you beat me to it!

[fonsp]

@elbosso Is your solution the same? What do you think about preloading packages?

[elbosso]

@fonsp No, actually I try to hone my docker abilities at the moment and I put off learning julia - this project looks as though I no longer have any excuses for putting things off in that regard. So I only had started writing my own Dockerfile and some bolts and nuts around it. But as I tested the contents of this pull request (and it worked wonderfully!) the only two cents I can add at the moment would be a docker-compose-file with preparations made for traefik2:

version: '2'

services:
  pluto_jl:
    build: .
#    ports:
#      - "1234:1234"
    restart: unless-stopped
    labels:
      - "traefik.backend=pluto_jl"
      - "traefik.enable=true"
      - "traefik.http.routers.pluto_jl.rule=Host(`pluto-jl.docker.lab`)"
      - "traefik.http.services.pluto_jl.loadbalancer.server.port=1234"
      - "traefik.docker.network=traefik_proxy"
    networks:
      - traefik_proxy

networks:
  traefik_proxy:
    external:
      name: traefik_proxy

[elbosso]

As an afterthought: the mentioned idea about layering pluto on top of other, more specialized docker containers seems like a really good and important one: One could write a section in the README detailing the "how to" for interested users and maybe build a network with other users/institutions making such docker images available.

[lungben]

I agree that it would be helpful to have an example Docker-Compose file alogside.
A volume mount should be added to it for transfering and persistence of notebooks.

Having an official Docker image created in the CI pipeline woud be great!

Regarding layers: in principle this should work fine, a derived image could just Pkg.add any additional requirements.

Julia Sysimages on the other hand would not stack well: if a sysimage is created both in the base image and in the derived image both of them would be in the final Docker image.

But we could offer 2 Docker images - one with Sysimage (including Pluto, PlutoUI, Plots) for usage "out-of-the-box" and one without sysimage if the user wants to create its own.

For the 1st image variant, we could add other packages frequently used in data analysis, etc., like CSV.jl and DataFrames.jl.

What do you think?

[fonsp]

Thanks!

Pluto and PlutoUI are both fairly small packages, so there is only a small (max 10 seconds) benefit in including them in the sysimage. The benefit of sysimaging Plots, DiffEq, etc, is much greater.

So how about we get an existing science-docker image with these popular packages that precompiles a sysimage (does anyone know of one?) and we just Pkg.add Pluto on top of it?

EDIT:
A quick google led to this one:
https://github.com/xiaodaigh/julia-data-science-base-docker-img
It includes the mentioned packages, it precompiles a sysimage.

It uses jupyter/scipy-notebook as base image, but maybe you can overwrite this to be scratch using a build argument?

[lungben]

The Jupyter Scipy Notebook image has 1.2 GB, for our purposes only the Debian part would be needed, but no Python, Conda, etc.
In addition, most of the Dockerfile content in this repository deals with Jupyter-specific stuff.
Therefore, I suggest to use the official Julia base image.

The list of packages, however, looks like a good start for me. For the "batteries included" Docker image maybe the JuMP and DifferentialEquations/SciML ecosystems should be added, too?

If Pluto does not significantly benefit from sysimages, we could do the following:

1. Pluto base image with Pluto, PlutoUI, Plots, based on official Julia (Debian) image
2. Pluto Datascience image, based on Pluto base image, plus additional packages and using a custom sysimage

[fonsp]

Right! The project

https://github.com/xiaodaigh/julia-data-science-base-docker-img

does not install anything Jupyter/python related, right? Maybe we can create a fork that just uses scratch instead of jupyter/scipy-notebook as its base image.

You suggested layering common science packages on top of a base Pluto image. I was thinking about doing it the other way around. Is there a benefit in either way? Is building sysimages easier? I release Pluto updates quite frequently - does that matter?

[lungben]

We need at least an OS, therefore rather use Debian (Slim) than Scratch as base image.

Lines 30-36 in the Dockerfile of the repository you linked (https://github.com/xiaodaigh/julia-data-science-base-docker-img/blob/master/Dockerfile) are essentially installing Julia, similar to the official Julia Docker image:
https://github.com/docker-library/julia/blob/master/1.5/buster/Dockerfile

Lines 38-90 are doing Jupyter-related stuff and are not required for Pluto (except Pkg.update and Pkg.precompile).

Afterwards, additional packages are installed and the sysimage is created, steps we would also need for Pluto. However, I would merge the RUN statements calling Julia into a single one to reduce the number of Docker image layers (giving slightly smaller and more efficient images).

Regarding releases:
If you tag a new official Pluto release, a newly built container (either by you or by someone else) will use this automatically (it is essentially just a Pkg.add).
If you provide an official Pluto image at Dockerhub, this needs to be re-built (which should be automatized).

[lungben]

One more point: the standard Julia image lets the user run on ROOT (and I have not changed it in my Dockerfile in this PR due to lazyness). From security perspective it would be better to create a non-root user in the Dockerfile and switch to it (similar to the jovyan user in the Jupyer Dockerfiles).

[fonsp]

I don't know how to do that :) Do you?

[lungben]

I just updated the PR accordingly :-)

It took me a while that I need to change the JULIA_DEPOT_PATH manually so that the .julia directory is created for the right user.

[fonsp]

Oops sorry I merged thinking that this was PlutoUtils.jl, so I reverted again. Can you create a PR there?

[lungben]

Done

[shafdog]

Is it possible for the Pluto docker container support other architectures? Precompiled docker tags are all for x86_64.

Not a docker expert myself, but using docker's "buildx" generally "cross-compiles" these things from a Mac (x86_64) to ARMv7. But when I try it on the PlutoUtils.jl docker sources, I get some nasty errors and it doesn't build a ARM Pluto docker contains:

default git:(master) ✗ docker buildx build --platform linux/arm/v7 -t a2m0/plutorb .
[+] Building 17.4s (9/10)                                                       
 => [internal] load build definition from Dockerfile                       0.0s
 => => transferring dockerfile: 32B                                        0.0s
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 2B                                            0.0s
 => [internal] load metadata for docker.io/library/julia:latest            0.0s
 => CACHED [1/6] FROM docker.io/library/julia:latest                       0.0s
 => [internal] load build context                                          0.0s
 => => transferring context: 425B                                          0.0s
 => [2/6] RUN useradd -m -d /home/pluto pluto     && mkdir /home/pluto/no  0.4s
 => [3/6] COPY prestartup.jl /home/pluto/                                  0.0s
 => [4/6] COPY startup.jl /home/pluto/                                     0.0s
 => ERROR [5/6] RUN julia /home/pluto/prestartup.jl     && chown -R plut  16.9s
------                                                                          
 > [5/6] RUN julia /home/pluto/prestartup.jl     && chown -R pluto /home/pluto: 
#9 4.083   Installing known registries into `/home/pluto/.julia`                
#9 16.55        Added                                                           
#9 16.55 signal (6): Aborted                                                    
#9 16.55 in expression starting at /home/pluto/prestartup.jl:2                  
#9 16.56 unknown function (ip: 0xff6a5746)
#9 16.56 gsignal at /lib/arm-linux-gnueabihf/libc.so.6 (unknown line)
#9 16.56 Allocations: 3945490 (Pool: 3945086; Big: 404); GC: 14
#9 16.56 qemu: uncaught target signal 6 (Aborted) - core dumped
#9 16.60 Aborted
------
error: failed to solve: executor failed running [/bin/sh -c julia ${USER_HOME_DIR}/prestartup.jl     && chown -R ${USER} ${USER_HOME_DIR}]: exit code: 134

The base Julia docker does support "linux/arm/v7" docker platform, but something go amiss in the Pkg.add() or Pkg.precompile() for this "Pluto-enabled" one when using docker buildx.

[elbosso]

I just tested https://github.com/fonsp/PlutoUtils.jl successfully on a Rockchip RK3288 (aarch64): building as well as running worked out of the box - the only change i had to make was instead of directory docker I had to run the command for building it inside directory default . Afterwards I spun up the container that I built and it did work (after a fashion, the NanoPi Neo 3 is somewhat underpowered for Pluto.jl). So if you have access to ARM hardware - why not build the image yourself on this hardware and the use it?

[shafdog]

Thanks @elbosso – trying to do something similar as your Neo3, but on a Mikrotik router. They recently added docker support and wanted to get Pluto.jl/Julia on it – I've be experimenting with using a notebooks for device configuration/management/reporting, want to try Pluto with these Mikrotik routers. Unfortunately, the Mikrotik devices have no real shell (bash/etc) or package loading, so uploading a saved Docker build is the only option. Mikrotik Docker Docs.

Just little odd there was an apparent core dump someplace in the process – but don't know if/where docker would stashes that. Although not hard to imagine something could go wrong with Pkg.precompile() in some QEMU context on Mac+homebrew within Docker's cross-compiling build... Going to try to spin up a fuller Ubuntu docker on the Mikrotik ARM device to try to bootstrap this and avoid thinking about [pre-/cross-/]compiling happening between IntelMac/ARM32 for both Docker&Julia. But have seen "buildx" work for a couple other things.

Perhaps suggesting a feature enhancement be Docker's multiplatform support for PlutoUtils.jl io the Docker Hub repo.

[elbosso]

I actually once proposed my boss a scheme where we would find opensource projects without arm versions of their docker images and build it for them so that the company could get more visibility or dare I say PR and we could give a little something back to the open source community - i would have bought a small arm system to do that. But my boss is - sad to say - averse to the whole idea of giving to the open source community.
So I only do it privately and mostly blog about it...