Skip to content
Permalink
Browse files

[web] Container maker documentation.

  • Loading branch information...
bilke committed Feb 11, 2019
1 parent 2f5cbb7 commit 87e2efe7baec89c858aa8cc1a1c7fb20068d6e32
@@ -11,29 +11,31 @@ weight = 1034

## Introduction

> [Docker](https://www.docker.com/whatisdocker) containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in.
> Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in.
>
> <cite>[www.docker.com/whatisdocker](https://www.docker.com/whatisdocker)</cite>

See the [docs](https://docs.docker.com) for installation instructions.

## Images

Docker images can be created by [Dockerfiles](https://docs.docker.com/reference/builder/) which can be stacked and thus form a directed graph. OGS-6 image definitions can be found at the [ufz/dockerfiles](https://github.com/ufz/dockerfiles)-repo. Built images can be found at [Docker Hub](https://hub.docker.com/u/ogs6/).
Docker images can be created by [Dockerfiles](https://docs.docker.com/reference/builder/) which can be stacked and thus form a directed graph. OGS-6 image definitions are created with [ufz/ogs-container-maker](https://github.com/ufz/ogs-container-maker). Built images can be found at [Docker Hub](https://hub.docker.com/u/ogs6/).

To build an image by yourself create a Dockerfile:

```
FROM ubuntu:14.04
MAINTAINER Jon Doe <jon.doe@foo.com>
FROM ubuntu:17.10

RUN ...
```

Run the `build` command:

```bash
$ docker build -t repo/image_name path/to/directory
$ docker build --rm -t repo/image_name path/to/directory
```

- `--rm` Cleans up after exiting the container
- `-t` specifies a name for the image, can be arbitrary chosen (but should match the corresponding image on Docker Hub if there is one)
- The path should specify the directory where the Dockerfile is located

@@ -44,37 +46,38 @@ Now you can see your build image with `$ docker images`.
To run commands inside a container:

```bash
$ docker run -t image_name command_to_run
$ docker run --rm -t image_name command_to_run
```

To run an interactive shell add the `-i`-switch:

```bash
$ docker run -i -t image_name /bin/bash
$ docker run --rm -i -t image_name
```

It is useful to mount folders from the host operating system in the Docker container, e.g. to edit source code on your host with your favorite editor:

```bash
$ docker run -i -t -v /host/directory:/container/directory image_name /bin/bash
$ docker run --rm -i -t -v /host/directory:/container/directory image_name
```

## Prebuilt OGS-6 Docker images

There are docker images provided on [Docker Hub](https://hub.docker.com/u/ogs6/) which include everything necessary to build OGS-6.

There are images for `gcc`, `clang` and `mingw`.
There are images for `gcc` and `clang`.

E.g. to start developing OGS:

```bash
docker pull ogs6/gcc
docker run -t -i -v /path/to/ogs/source/on/host:/usr/src/ogs ogs6/gcc
docker run --rm -t -i -v /path/to/ogs/source/on/host:/tmp/ogs ogs6/gcc
```

You are now in the Docker container the ogs source code from your host computer mounted at `/usr/src/ogs`. Now you can build and run OGS:

```bash
cd /tmp
mkdir build; cd build
cmake ../ogs
make ogs
@@ -86,4 +89,5 @@ make ogs
The following images are used by Jenkins:

- `ogs6/gcc`, defined in [scripts/docker/Dockerfile.gcc.full](https://github.com/ufz/ogs/blob/master/scripts/docker/Dockerfile.gcc.full)
- `ogs6/gcc:gui`, defined in [scripts/docker/Dockerfile.gcc.gui](https://github.com/ufz/ogs/blob/master/scripts/docker/Dockerfile.gcc.gui)
- `ogs6/clang`, defined in [scripts/docker/Dockerfile.clang.full](https://github.com/ufz/ogs/blob/master/scripts/docker/Dockerfile.clang.full)
@@ -0,0 +1,54 @@
+++
date = "2018-09-21T11:00:13+01:00"
title = "Singularity"
author = "Lars Bilke"
weight = 1034

[menu]
[menu.devguide]
parent = "advanced"
+++

## Introduction

[Singularity](https://www.sylabs.io) is a Linux container runtime similar to Docker. Key advantages over Docker are

- Container don't run with root privileges
- You are the same user with the same privileges inside the container as on the host
- Container can run on HPC systems and seamlessly integrate with resource managers and MPI
- Container can leverage NVidia GPUs

Singularity per default mounts your home directory and also passes your current working directory when starting a container. Therefore it is easy to use it for development.

### Prerequisites

- Linux only!
- [Install Git with LFS]({{< ref "prerequisites" >}}/#step-install-git)
- [Install Singularity](https://www.sylabs.io/guides/2.6/user-guide/quick_start.html#quick-installation-steps)

### Build OGS inside a container

```bash
[git clone ogs]
singularity pull docker://ogs6/gcc # Downloads the image to gcc.simg
singularity shell gcc.simg
[Now inside the container]
mkdir build; cd build
cmake ../ogs -DCMAKE_BUILD_TYPE=Release
make
./bin/ogs
```

### Run OGS inside a Container (called from outside)

Once ogs executable is built it can be called from **outside** the container:

```bash
singularity exec gcc.simg build/bin/ogs some/path/project.prj
```

This starts the container, mounts your home directory inside the container, passes the current working directory and runs the ogs executable (which is in your home directory which is mounted inside the container) with the passed project file. Everything works as expected and is transparent to the user. When ogs finishes the container stops and you returns to the host system.

## Container generator

You can download a prebuilt container from Docker Hub as shown above (e.g. `singularity pull docker://ogs6/gcc`). But we also provide a [container generator]({{< ref "container.pandoc" >}}) to create a specific container for your needs.
@@ -20,7 +20,8 @@ The following explanation is taken from an [in-depth article](https://www.atlass
> The main advantage of the Forking Workflow is that contributions can be integrated without the need for everybody to push to a single central repository. Developers push to their own server-side repositories, and only the project maintainer can push to the official repository. This allows the maintainer to accept commits from any developer without giving them write access to the official codebase.
>
> The result is a distributed workflow that provides a flexible way for large, organic teams (including untrusted third-parties) to collaborate securely. This also makes it an ideal workflow for open source projects.
> <cite> https://www.atlassian.com/git/tutorials/comparing-workflows#!workflow-forking </cite>
>
> <cite><a href="https://www.atlassian.com/git/tutorials/comparing-workflows#!workflow-forking">www.atlassian.com/git/tutorials/comparing-workflows#!workflow-forking</a> </cite>
>

The workflow is summarized in the following image from the [GitHub blog](https://github.com/blog/2042-git-2-5-including-multiple-worktrees-and-triangular-workflows):
@@ -0,0 +1,64 @@
+++
date = "2018-11-14T15:00:13+01`:00"
title = "Running OGS in a container"
author = "Lars Bilke"
weight = 10

[menu]
[menu.userguide]
parent = "basics"
+++

## With Singularity

### Prerequisites

* Linux (you may run Singularity with [Vagrant](https://app.vagrantup.com/sylabs) on macOS or Windows as well)
* A running [installation](https://www.sylabs.io/guides/2.6/user-guide/quick_start.html#quick-installation-steps) of Singularity 2.6

### Get a container image

TODO: Download prebuilt images

Build images with [ogs-container-maker](https://github.com/ufz/ogs-container-maker) on [Jenkins](https://jenkins.opengeosys.org/job/ufz/job/ogs-container-maker/job/master/build) (login required, ask Lars Bilke for access):

![](../ogs-container-maker-params.png){width=450px}

It will build all combinations of `format`, `openmpi_versions` and `pm`. When the build finishes you can download the container (`*.simg` files) at the jobs page under `Build Artifacts`.

### Run OGS inside a Container (called from outside)

```bash
singularity exec ogs-openmpi-2.1.1-conan.simg ogs some/path/project.prj
```

This starts the container, mounts your home directory inside the container, passes the current working directory and runs the ogs executable (which is in your home directory which is mounted inside the container) with the passed project file. Everything works as expected and is transparent to the user. When ogs finishes the container stops and you returns to the host system.

```bash
# Create output directories
mkdir -p _out _out_mpi
# Run serial benchmark
singularity exec ogs-openmpi-2.1.1-conan.simg ogs -o _out [ogs-sources]/Tests/Data/Mechanics/Linear/disc_with_hole.prj
# Run serial benchmark with output validation (via vtkdiff)
singularity exec ogs-openmpi-2.1.1-conan.simg -o _out -r [ogs-sources]/Tests/Data/Mechanics/Linear [ogs-sources]/Tests/Data/Mechanics/Linear/disc_with_hole.prj
# Run parallel benchmark with MPI
mpirun -np 4 singularity exec ogs-openmpi-2.1.1-conan.simg -o _out_mpi [ogs-sources]/Tests/Data/Mechanics/Linear/disc_with_hole.prj
```

You can run other contained executables, e.g. `vtkdiff`:

```bash
singularity exec ogs-openmpi-2.1.1-conan.simg vtkdiff --help
```

You can interactively explore the container with `singularity shell` (you can see that you are **in** the container because of the `Singularity [container image file]:...>` prefix of the shell):

```bash
# Shell into container
singularity shell ogs-openmpi-2.1.1-conan.simg
# List files in the container
Singularity ogs-openmpi-2.1.1-conan.simg:...> ls /usr/local/ogs/bin
... ogs tetgen vtkdiff
# Exit the container and get back to your hosts shell
Singularity ogs-openmpi-2.1.1-conan.simg:...> exit
```

0 comments on commit 87e2efe

Please sign in to comment.
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.