Building a Livebook container with our application's code in it

[mbklein]

My team is responsible for a large, staff-facing Elixir/Phoenix application called Meadow. In order to make support, troubleshooting, prototyping, and data wrangling easier, we decided it would be beneficial to be able to call Meadow functions from a Livebook notebook. After some limited success with a two-container setup, using Livebook's “attached node” runtime option as well as the standalone runtime with remote execution smart cells, we felt we could do better if Meadow were running right inside the same runtime as the notebook.

However, given the size of our application's codebase, Mix.installing it and waiting for it to compile inside the notebook wasn't practical. The solution presented below has been a complete game changer for us – rock solid, giving us the ability to work with Meadow's codebase and live data with all the same autocomplete, inline documentation, smart cells, and other features that make Livebook such a great tool. It allows us to investigate issues and bugs while documenting them and their solutions all in one place, and also to create repeatable workflows for processes that aren't (yet) among Meadow's user-facing features.

I'm sure there are things we could be doing better, but so far this has been an incredible improvement in the way we work.

The four main pieces of the solution are:

Dockerfile

A custom Dockerfile that uses the latest Livebook image as a base and builds Meadow inside it, which ensures that Meadow is built using the same version of Elixir/OTP that the notebook will use for its runtime.

# Stage 1: Build meadow inside Livebook image
FROM ghcr.io/livebook-dev/livebook:latest AS builder

ENV MIX_ENV=prod \
  LANG=C.UTF-8

WORKDIR /build/meadow

COPY ./app/mix.exs ./app/mix.lock ./app/config ./

RUN mix deps.get --only prod \
 && mix deps.compile

# Copy the source and build
COPY ./app/ ./
RUN mix compile

RUN find _build/prod/lib -type l | while read -r link; do \
      target=$(readlink -f "$link") && \
      rm "$link" && \
      cp -a "$target" "$link"; \
    done

# Stage 2: Final image with compiled meadow code
FROM ghcr.io/livebook-dev/livebook:latest
ENV LIVEBOOK_AWS_CREDENTIALS=true
ENV LIVEBOOK_DATA_PATH=/data
ENV LIVEBOOK_DISTRIBUTION=name
ENV LIVEBOOK_HOME=${LIVEBOOK_DATA_PATH}/books
ENV LIVEBOOK_IDENTITY_PROVIDER=custom:MeadowLivebookAuth
ENV LIVEBOOK_IP=0.0.0.0
ENV LIVEBOOK_TOKEN_ENABLED=false

RUN mkdir -p ${LIVEBOOK_DATA_PATH}/books

# Copy compiled _build/prod/lib into /meadow/lib in final image
COPY --from=builder /build/meadow/_build/prod/lib /meadow/lib
COPY --from=builder /build/meadow/config /meadow/config
COPY ./livebook/bootstrap.exs /meadow/bootstrap.exs
COPY ./livebook/startup.livemd /data/books/_meadow.livemd
COPY ./livebook/extensions/* /app/user/extensions/

# Optional: set MEADOW_PATH environment variable
ENV MEADOW_PATH=/meadow/lib

/meadow/bootstrap.exs

An Elixir runtime script that loads Meadow into the current runtime, configures it, and starts its main application.

Path.wildcard("/meadow/lib/*/ebin")
|> Enum.each(&:code.add_path(String.to_charlist(&1)))

defmodule Meadow.Bootstrap do
  require Logger

  def bootstrap do
    case Application.started_applications() |> Enum.find(fn {app, _, _} -> app == :meadow end) do
      nil ->
        load_meadow()

      {app, _, _} ->
        Logger.warning("Meadow is already loaded: #{inspect(app)}")
        :ok
    end
  end

  defp load_meadow do
    Application.ensure_all_started(:mix)

    with config <- Config.Reader.read!("/meadow/config/config.exs") do
      Enum.map(config, fn {app, _} ->
        {app, Application.get_all_env(app)}
      end)
      |> Config.Reader.merge(config)
      |> Application.put_all_env()
    end

    Application.load(:meadow)
    if :code.is_loaded(Mix), do: Meadow.Config.Runtime.configure!()
    Logger.configure(level: :info)
    Application.ensure_all_started(:meadow)
  end
end

Starter Notebook (_meadow.livemd)

This livemd file, containing just the code needed to bootstrap Meadow, is copied to the LIVEBOOK_HOME directory, making it easy to fork and create a Meadow support notebook.

Code.require_file("/meadow/bootstrap.exs")
Meadow.Bootstrap.bootstrap()
Mix.install([
  # Add additional dependencies here
])

Identity Provider Module

A custom Livebook Identity Provider module that calls back to the running Meadow application to make sure that the Livebook user is a valid Meadow superuser, allowing seamless integration between Meadow and Livebook auth. I haven't included the code here because it's highly specific to Meadow and therefore not as generally useful as the other pieces.

Here's a screen recording showing it in action, with a couple very simple cells at the bottom just to show off autocompletion and documentation:

[jonatanklosko]

Great, thank you for sharing!