DIT4C is a scalable platform for providing containerized web-based programming and data analysis environments to researchers.
- Client == Modern web browser
- No local credentials: use your GitHub or AAF account
- Based on rkt - run ACIs or Docker images on bare metal or cloud computing
All authentication is via federated identity providers - all a user needs is a modern web browser.
Some of the current environments available are:
- Base - web-based TTY sessions and basic file management
- Jupyter Notebook + Base
- NLTK + Jupyter Notebook + Base
- OpenRefine + Base
- RStudio + Base
- Apache Zeppelin + Base
- X11 - Base + X11 sessions via HTML5 VNC client
- Octave + X11
- QGIS + Jupyter + RStudio + X11 - QGIS with supporting Python & R environments
DIT4C is focused on meeting two needs:
- Training sessions - having a working install right from the beginning means training participants start programming sooner, and do so in a consistent environment.
- Reproducible research - container sharing and export allows complete working environments to be exchanged and archived.
DIT4C separates the portal environment which manages user access and containers from the compute nodes that provide them.
Core services:
- portal - user-facing UI and scheduler coordination
- scheduler - manages compute clusters and schedules containers on individual nodes
Additional services:
- image server - necessary for saving and exporting instances
- dit4c-imageserver-filesystem - store images on a filesystem (possibly mounted via NFS)
- dit4c-imageserver-swift - store images in OpenStack Swift object storage
Auxiliary "helper" container images:
- dit4c-helper-listener-*
- dit4c-helper-listener-ngrok2 (DEFAULT) - development image that exposes containers via ngrok.com (don't use this in production)
- dit4c-helper-listener-ssh - expose containers using SSH reverse port forwarding and reverse-proxy server. When used with dit4c-routingserver-ssh it supports HTTP & HTTPS.
- dit4c-helper-listener-ngrok1 (DEPRECATED) - expose containers via your own ngrok1 servers. Supports HTTP, HTTPS & individual TCP ports.
- dit4c-helper-auth-portal - proxies container services behind portal-provided auth
- dit4c-helper-upload-webdav - uploads saved images to a webdav server
See https://dit4c.github.io/ for further archtecture details.
All container instances are issued an OpenPGP key prior to starting which is convertible to a JSON Web Key (JWK) or SSH key. This allows container helpers to independently contact the portal to update and retrieve information using a signed JSON Web Token (JWT).
The portal also provides keys via a public registry, which will allow future helpers to authenticate independently to other services or retrieve encrypted content. This is still a work in progress.
While DIT4C could be run on a single server for development purposes, a secure installation will attempt to segregate the portal, scheduler and compute onto separate physical or virtual machines.
A valid minimal configuration would include:
- portal/image server (public ports exposed: HTTPS)
- nghttpx/nginx as HTTPS reverse-proxy
- portal
- image server
- Cassandra database for portal
- scheduler (public ports exposed: none)
- scheduler
- Cassandra database for scheduler
- compute node (public ports exposed: none)
- no installed software required
- CoreOS recommended
- SSH port must be accessible by scheduler
HTTPS should be used with the portal, image server (recommended, but optional) and routing server. Let's Encrypt is sufficient for the portal and image server, but all HTTPS routing server implementations require a wildcard certificate. While highly discouraged, a self-signed certificate can be used for the routing server if necessary (presumably while a valid wildcard certificate is being sourced).
See https://dit4c.github.io/ for installation instructions.