Calhoun

Jupyter notebook preview service

Background

This project is part of the Terra platform of service APIs. It is primarily intended to provide preview versions of notebooks and R markdown files for Terra-based web applications, such as the Terra workbench research environment.

Description

Your web-app needs to display read-only versions of Jupyter notebooks and R markdown files.

Under the hood, Jupyter .ipynb files are JSON documents with a particular format. Jupyter provides a commandline tool / library nbconvert for interacting with these documents and converting them to various formats, in particular HTML.

Similarly, R Markdown .Rmd files are documents with a particular format, and the rmarkdown package provides mechanisms for converting R Markdown documents to HTML.

This project is essentially nbconvert and rmarkdown as a REST service. It takes an input notebook or R Markdown document and returns HTML that can be embedded in your application.

API

A swagger-ui page is available at /swagger-ui/ on any running instance. For existing instances, those are:

• dev: https://calhoun.dsde-dev.broadinstitute.org/swagger-ui/
• alpha: https://calhoun.dsde-alpha.broadinstitute.org/swagger-ui/
• perf: https://calhoun.dsde-perf.broadinstitute.org/swagger-ui/
• staging: https://calhoun.dsde-staging.broadinstitute.org/swagger-ui/
• prod: https://calhoun.dsde-prod.broadinstitute.org/swagger-ui/

Framework

This project uses the Flask Python web framework.

Managing dependencies

We use Poetry to manage our dependencies. From their website:

Poetry is a tool for dependency management and packaging in Python. It allows you to declare the libraries your project depends on and it will manage (install/update) them for you. Poetry offers a lockfile to ensure repeatable installs, and can build your project for distribution.

Install Poetry

curl -sSL https://install.python-poetry.org | python3 -

If you need to change any dependency versions:

• update the pyproject.toml file
• run the following to update the lock file

poetry lock

To install dependencies

poetry install

To update dependencies

poetry update

Running locally

Run a local containerized server:

docker image build . -t calhoun-test:0
docker kill t1
docker run -e FLASK_DEBUG=1 --rm -itd --name t1 -p 8080:8080 calhoun-test:0

This will start a Calhoun server at localhost:8080.

Access the application locally:

• http://localhost:8080/status
• http://localhost:8080/swagger-ui

Alternative (no Docker container)

You can skip the container and run a local app with Flask:

python3 -m venv env
source env/bin/activate
pip install Flask
export FLASK_DEBUG=1

Dependencies for running containerless

Install Pandoc and R

brew install pandoc
brew install R

Install R packages

R
> install.packages(c("rmarkdown", "stringi", "tidyverse", "Seurat", "ggforce"))

Install Poetry and project dependencies

curl -sSL https://install.python-poetry.org
poetry install

Write a dev config file

cp config.py config.dev.py

Ensure etc/hosts file has the following record:

127.0.0.1       local.dsde-dev.broadinstitute.org

Once complete, copy vault read secret/dsde/firecloud/dev/common/server.crt to /etc/ssl/certs and vault read secret/dsde/firecloud/dev/common/server.key to /etc/ssl/private.

Serve the containerless app

Run a local server

DEVELOPMENT='true' SAM_ROOT='https://sam.dsde-dev.broadinstitute.org' python3 main.py

Running locally with terra-ui

• point the calhoun URL in the terra-ui dev config to your local url http://localhost:8080
• run a local terra-ui
• Look at previews!

Automated testing

Run unit tests locally

./scripts/unit-test.sh

Run automation tests locally

gcloud auth login <any-terra-dev-user>
RUN_AUTHENTICATED_TEST=1 ./scripts/automation-test.sh

If you add a new test case, make sure it is imported and added to test_cases in unit_test.py.

Deployment

Upon merging a change to dev:

• The build github workflow builds the new image
• Then it automatically updates the calhoun version in terra-helmfile