Skip to content
A web application reproducing ReproZip packages in the cloud. Runs on Kubernetes
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
builder
common/common
runner Fix runner not reporting correct exit status Jun 15, 2018
scripts Add scripts/psql.sh Jul 19, 2017
web
.env Add docker-compose file Oct 21, 2018
.gitignore Generate Kubernetes configuration via doit May 18, 2017
.gitlab-ci.yml Fix flake8 Oct 15, 2018
.gitmodules Add ReproZip as a submodule May 12, 2017
LICENSE.txt Add BSD license May 18, 2017
README.md Add submodule instructions to README Oct 30, 2017
aws_policy.json Add AWS policy Jun 17, 2017
config.dist.yml Add nginx proxy checklist to config.dist.yml Nov 12, 2018
deployment.dot Deployment details in the README May 25, 2017
deployment.png Deployment details in the README May 25, 2017
docker-compose.yml Add docker-compose file Oct 21, 2018
dodo.py Fix flake8 Oct 15, 2018
doit.cfg Introduce .tarignore May 12, 2017
k8s-ingress.yml Add k8s ingress config Aug 23, 2017
k8s-secrets.yml
k8s-services.yml
k8s-volumes.yml
k8s.tpl.yml

README.md

Matrix

ReproServer

Goals

  • Import something we can build a Docker image from (currently only a ReproZip package)
  • Build a Docker image from it
  • Allow the user to change experiment parameters and input files
  • Run the experiment
  • Show the log and output files to the user

Components

Frontend

Web application allowing users to select or upload an experiment, edit parameters, upload input files. After running, shows the log and lets the user download output files.

Builder

From the experiment file, builds a Docker image and caches it on a private Docker registry for the runners.

Runner

From the cached Docker image, input files, and parameters, runs the experiment and stores the results.

How to setup locally for development

This doesn't use docker-compose, because it has serious limitations. To have better control over the build, and to have efficient & automatic builds, pydoit is used to drive Docker. It can also generate the Kubernetes configuration for you.

Don't forget to check out the submodule using git submodule init && git submodule update.

Using doit build will build the images (reproserver-*). Using doit start will start all the containers locally.

The ports are:

  • 8000 is the frontend web server;
  • 8080 is RabbitMQ's web interface;
  • 9000 is Minio's web interface;
  • 5000 is the Docker registry;
  • 5432 is PostgreSQL, but using the ORM from the web container is probably easier (docker exec -ti reproserver-web python).

Deployment on Kubernetes

You can use doit to generate the Kubernetes configuration from the template k8s.tpl.yml.

First, you need to create the secret holding the user/password used by RabbitMQ, PostgreSQL, and Minio: kubectl create secret generic reproserver-secret-prod --from-literal=user=USERNAMEHERE --from-literal=password=PASSWORDHERE. Then, you need to create the persistent volume (and claim) for PostgreSQL and Minio. You can tweak the k8s-volumes.yml configuration file then apply it with kubectl apply -f k8s-volumes.yml.

You can then generate the configuration file for the app with doit k8s tier=prod registry=vidanyu tag=0.3 (this will pull images vidanyu/reproserver-{web,builder,runner}:0.3`).

You can run multiple versions of the app alongside each other by changing the tier= parameter (example: tier=staging). If you don't want to create volumes for each of those, you can use postgres_no_volume=1 minio_no_volume=1.

You will need to expose the web container to the Internet in some way. In my setup, it is exposed as a node port (see k8s-services.yml) and an NGINX proxy sits in front of it.

Deployment overview

You can’t perform that action at this time.