Patches (#155)

* rm travis yml * add extras for testing reqs * update examples * rewrite setup with more setup args * only get git version if not release * minor json updates * update help * add json output * change c3d to convert3d * add codecov * ignore coverage artifacts * update badges * rm coverage shield * rm coverage shield * add petpvc support * minimize dockerfile * do not upload test artifacts * standardize values * fix petpvc tests * fix binary name * update examples * add workdir and add_to_entrypoint * fix typo * add freesurfer url 6.0.1 * rm todo * fix another typo
ReproNim · Apr 23, 2018 · 934566d · 934566d
1 parent ea5425b
commit 934566d
Show file tree

Hide file tree

Showing 19 changed files with 261 additions and 270 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -17,15 +17,16 @@ _setup_env: &setup_kwds
     conda update -yq --all
 
     pip install -q --no-cache-dir -U pip
-    pip install -q --no-cache-dir dropbox pytest-cov reprozip
-    pip install -q --no-cache-dir -e ~/neurodocker
+    pip install -q --no-cache-dir reprozip codecov
+    pip install -q --no-cache-dir -e ~/neurodocker[dev]
 
 
 version: 2
 jobs:
   test_docker:
     machine:
       *machine_kwds
+    working_directory: ~/neurodocker
     steps:
       - checkout:
           *checkout_kwds
@@ -41,7 +42,8 @@ jobs:
           no_output_timeout: 360m
           command: |
             source ~/.bashrc
-            pytest -k 'test_docker' ~/neurodocker/neurodocker
+            pytest --cov -k 'test_docker' neurodocker
+            codecov
       - save_cache:
           key: dfs-v0-{{ .Branch }}-{{ .Revision }}
           when: always
@@ -51,6 +53,7 @@ jobs:
   test_singularity:
     machine:
       *machine_kwds
+    working_directory: ~/neurodocker
     steps:
       - checkout:
           *checkout_kwds
@@ -76,7 +79,8 @@ jobs:
           no_output_timeout: 360m
           command: |
             source ~/.bashrc
-            pytest -k 'test_singularity' ~/neurodocker/neurodocker
+            pytest --cov -k 'test_singularity' neurodocker
+            codecov
       - save_cache:
           key: srs-v0-{{ .Branch }}-{{ .Revision }}
           when: always
@@ -86,6 +90,7 @@ jobs:
   test_others:
     machine:
       *machine_kwds
+    working_directory: ~/neurodocker
     steps:
       - checkout:
           *checkout_kwds
@@ -96,7 +101,8 @@ jobs:
           no_output_timeout: 30m
           command: |
             source ~/.bashrc
-            pytest -k 'not test_docker and not test_singularity' ~/neurodocker/neurodocker
+            pytest --cov -k 'not test_docker and not test_singularity' neurodocker
+            codecov
 
 
 workflows:

diff --git a/.dockerignore b/.dockerignore
@@ -3,11 +3,17 @@ __pycache__
 **/.DS_Store
 .DS_Store
 /scraper
+tests
+**/tests
 .cache
+.pytest_cache
 .ipynb_checkpoints
 dockerfile_tests
 *.egg-info
 
+.coverage
+coverage.xml
+
 *.ipynb
 *.pyc
 *.tar

diff --git a/.gitignore b/.gitignore
@@ -2,10 +2,14 @@ __pycache__
 .cache
 .DS_Store
 .ipynb_checkpoints
+.pytest_cache
 dockerfile_tests
 /scraper
 *.egg-info
 
+.coverage
+coverage.xml
+
 *.ipynb
 *.pyc
 *.tar

diff --git a/.travis.yml b/.travis.yml
diff --git a/Dockerfile b/Dockerfile
@@ -2,14 +2,15 @@ FROM alpine:3.7
 
 LABEL maintainer="Jakub Kaczmarzyk <jakubk@mit.edu>"
 
-COPY . /opt/neurodocker
-
 RUN tmp_pkgs="curl gcc musl-dev python3-dev sqlite-dev" \
-    && apk add --update --no-cache git python3 rsync $tmp_pkgs \
+    && apk add --update --no-cache git python3 py3-yaml rsync $tmp_pkgs \
     && curl -fsSL https://bootstrap.pypa.io/get-pip.py | python3 - \
     && pip install --no-cache-dir reprozip \
-    && pip install --no-cache-dir -e /opt/neurodocker \
-    && neurodocker --help \
-    && apk del $tmp_pkgs
+    && apk del $tmp_pkgs \
+    && rm -rf /var/cache/apk/* ~/.cache/pip/*
+
+COPY . /opt/neurodocker
+RUN pip install --no-cache-dir -e /opt/neurodocker \
+    && neurodocker --help
 
 ENTRYPOINT ["neurodocker"]
diff --git a/README.md b/README.md
@@ -1,35 +1,30 @@
 # Neurodocker
 
-[![Build Status](https://travis-ci.org/kaczmarj/neurodocker.svg?branch=master)](https://travis-ci.org/kaczmarj/neurodocker)
-[![codecov](https://codecov.io/gh/kaczmarj/neurodocker/branch/master/graph/badge.svg)](https://codecov.io/gh/kaczmarj/neurodocker)
+[![build status](https://img.shields.io/circleci/project/github/kaczmarj/neurodocker/master.svg)](https://circleci.com/gh/kaczmarj/neurodocker/tree/master)
 
-
-_Neurodocker_ is a Python project that generates custom Dockerfiles for neuroimaging and minifies existing Docker images (using [ReproZip](https://www.reprozip.org/)). The package can be used from the command-line or within a Python script. The command-line interface generates Dockerfiles and minifies Docker images, but interaction with the Docker Engine is left to the various `docker` commands. Within a Python script, however, _Neurodocker_ can generate Dockerfiles, build Docker images, run commands within resulting containers (using the [`docker` Python package](https://github.com/docker/docker-py)), and minify Docker images. The project is used for regression testing of [Nipype](https://github.com/nipy/nipype/) interfaces.
+_Neurodocker_ is a command-line program that generates custom Dockerfiles and Singularity recipes for neuroimaging and minifies existing containers.
 
 Examples:
-  - [Generate Dockerfile](#generate-dockerfile)
-  - [Generate Dockerfile (full)](#generate-dockerfile-full)
+  - [Canonical examples](#canonical-examples)
+    - [Docker](#docker)
+    - [Singularity](#singularity)
+  - [Assorted examples](./examples)
   - [Minimize existing Docker image](#minimize-existing-docker-image)
   - [Example of minimizing Docker image for FreeSurfer recon-all](https://github.com/freesurfer/freesurfer/issues/70#issuecomment-316361886)
 
 
-# Note to users
-
-This software is still in the early stages of development. If you come across an issue or a way to improve _Neurodocker_, please submit an issue or a pull request.
-
-
 # Installation
 
 Use the _Neurodocker_ Docker image:
 
 ```
-docker run --rm kaczmarj/neurodocker:v0.3.1 --help
+docker run --rm kaczmarj/neurodocker:0.4.0 --help
 ```
 
 Note: it is not yet possible to minimize Docker containers using the _Neurodocker_ Docker image.
 
 
-# Supported Software
+# Supported software
 
 | software | argument | description |
 | -------- | -------- | ----------- |
@@ -79,6 +74,9 @@ Note: it is not yet possible to minimize Docker containers using the _Neurodocke
 | **NeuroDebian** | os_codename* | Codename of the operating system (e.g., stretch, zesty). |
 |                 | server* | Server to download NeuroDebian packages from. Choose the one closest to you. See `neurodocker generate docker --help` for the full list of servers. |
 |                 | full | If true (default), use non-free sources. If false, use libre sources. |
+| **PETPVC** | version* | 1.2.2, 1.2.1, 1.2.0-b, 1.2.0-a, 1.1.0, 1.0.0 |
+|            | method | binaries (default) |
+|            | install_path | Installation path. Default `/opt/petpvc-{version}`. |
 | **SPM12** | version* | r7219, r6914, r6685, r6472, r6225 |
 |           | install_path | Installation path. Default `/opt/spm12-{version}`. |
 |           |              | _Note: Matlab Compiler Runtime is installed when SPM12 is installed._ |
@@ -92,14 +90,28 @@ Note: it is not yet possible to minimize Docker containers using the _Neurodocke
 
 Please see the [examples](examples) directory.
 
-## Canonical example
+## Canonical examples
+
+The canonical examples install ANTs version 2.2.0 on Ubuntu 18.04.
 
-Generate a Dockerfile which will install ANTs on Ubuntu 17.04. The result can be piped to `docker build` to build the Docker image.
+### Docker
 
 ```shell
-docker run --rm kaczmarj/neurodocker:v0.3.2 generate -b ubuntu:17.04 -p apt --ants version=2.2.0
+$ docker run --rm kaczmarj/neurodocker:0.4.0 generate \
+    --base ubuntu:18.04 --pkg-manager apt --ants version=2.2.0
 
-docker run --rm kaczmarj/neurodocker:v0.3.2 generate -b ubuntu:17.04 -p apt --ants version=2.2.0 | docker build -
+# Build image by piping Dockerfile to `docker build`
+$ docker run --rm kaczmarj/neurodocker:0.4.0 generate \
+    --base ubuntu:18.04 --pkg-manager apt --ants version=2.2.0 | docker build -
+```
+
+### Singularity
+
+Install ANTs on Ubuntu 18.04.
+
+```shell
+$ docker run --rm kaczmarj/neurodocker:v0.4.0 generate singularity \
+    --base ubuntu:18.04 --pkag-manager apt --ants version=2.2.0
 ```
 
 
@@ -117,18 +129,17 @@ In the following example, a Docker image is built with ANTs version 2.2.0 and a
 
 ```shell
 # Create a Docker image with ANTs, and download a functional scan.
-download_cmd="RUN curl -sSL -o /home/func.nii.gz http://psydata.ovgu.de/studyforrest/phase2/sub-01/ses-movie/func/sub-01_ses-movie_task-movie_run-1_bold.nii.gz"
-neurodocker generate -b centos:7 -p yum --ants version=2.2.0 --instruction="$download_cmd" | docker build -t ants:2.2.0 -
+$ download_cmd="curl -sSL -o /home/func.nii.gz http://psydata.ovgu.de/studyforrest/phase2/sub-01/ses-movie/func/sub-01_ses-movie_task-movie_run-1_bold.nii.gz"
+$ neurodocker generate docker -b centos:7 -p yum --ants version=2.2.0 --run="$download_cmd" | docker build -t ants:2.2.0 -
 
 # Run the container.
-docker run --rm -it --name ants-reprozip-container --security-opt=seccomp:unconfined ants:2.2.0
+$ docker run --rm -itd --name ants-reprozip-container --security-opt=seccomp:unconfined ants:2.2.0
 
-# (in a new terminal window)
 # Output a ReproZip pack file in ~/neurodocker-reprozip-output with the files
 # necessary to run antsMotionCorr.
 # See https://github.com/stnava/ANTs/blob/master/Scripts/antsMotionCorrExample
-cmd="antsMotionCorr -d 3 -a /home/func.nii.gz -o /home/func_avg.nii.gz"
-neurodocker reprozip-trace ants-reprozip-container "$cmd"
-
-reprounzip docker setup neurodocker-reprozip.rpz test
+$ cmd="antsMotionCorr -d 3 -a /home/func.nii.gz -o /home/func_avg.nii.gz"
+$ neurodocker reprozip-trace ants-reprozip-container "$cmd"
+# Create a Docker container with the contents of ReproZip's trace.
+$ reprounzip docker setup neurodocker-reprozip.rpz test
 ```
diff --git a/neurodocker/generators/common.py b/neurodocker/generators/common.py
@@ -77,13 +77,13 @@ def clear_memory(cls):
 
 def _get_json_spec_str(specs):
     """Return instruction to write out specs dictionary to JSON file."""
-    json_specs = json.dumps(specs, indent=2)
-    json_specs = json_specs.replace('\\n', '__TO_REPLACE_NEWLINE__')
-    json_specs = "\n\\n".join(json_specs.split("\n"))
+    js = json.dumps(specs, indent=2)
+    js = js.replace('\\n', '__TO_REPLACE_NEWLINE__')
+    js = "\n\\n".join(js.split("\n"))
     # Escape newline characters that the user provided.
-    json_specs = json_specs.replace('__TO_REPLACE_NEWLINE__', '\\\\n')
+    js = js.replace('__TO_REPLACE_NEWLINE__', '\\\\n')
     # Workaround to escape single quotes in a single-quoted string.
     # https://stackoverflow.com/a/1250279/5666087
-    json_specs = json_specs.replace("'", """'"'"'""")
-    cmd = "echo '{string}' > {path}".format(string=json_specs, path=SPEC_FILE)
+    js = js.replace("'", """'"'"'""")
+    cmd = "echo '{string}' > {path}".format(string=js, path=SPEC_FILE)
     return cmd
diff --git a/neurodocker/generators/singularity.py b/neurodocker/generators/singularity.py
@@ -4,6 +4,8 @@
 import copy
 import inspect
 
+from neurodocker.generators.common import _add_to_entrypoint
+from neurodocker.generators.common import _get_json_spec_str
 from neurodocker.generators.common import _installation_implementations
 from neurodocker.generators.common import _install
 from neurodocker.generators.common import _Users
@@ -16,7 +18,7 @@ def __init__(self, singularity_recipe_object):
         self._singobj = singularity_recipe_object
 
     def add_to_entrypoint(self, cmd):
-        self._singobj._runscript.insert(0, cmd)
+        self._singobj._post.append(_add_to_entrypoint(cmd))
 
     def base(self, base):
         if base.startswith('docker://'):
@@ -39,11 +41,14 @@ def install(self, pkgs, pkg_manager, opts=None):
         self._singobj._post.append(_install(pkgs, pkg_manager))
 
     def entrypoint(self, entrypoint):
-        self._singobj._runscript.append(entrypoint)
+        self._singobj._runscript = entrypoint
 
     def env(self, d):
         self._singobj._environment.update(**d)
 
+    def run(self, s):
+        self._singobj._post.append(s)
+
     def user(self, user):
         user_cmd = "su - {}".format(user)
         add_user_cmd = _Users.add(user)
@@ -53,6 +58,9 @@ def user(self, user):
             cmd = user_cmd
         self._singobj._post.append(cmd)
 
+    def workdir(self, path):
+        self._singobj._post.append("cd {}".format(path))
+
 
 class SingularityRecipe:
 
@@ -65,7 +73,7 @@ def __init__(self, specs):
         self._post = []
         self._environment = OrderedDict()
         self._files = []
-        self._runscript = ['/neurodocker/startup.sh "$@"']
+        self._runscript = '/neurodocker/startup.sh "$@"'
         self._test = []
         self._labels = []
 
@@ -89,6 +97,7 @@ def __init__(self, specs):
         self._parts_filled = False
         _Users.clear_memory()
         self._add_neurodocker_header()
+        self._add_json()
 
     def render(self):
         def _render_one(section):
@@ -125,7 +134,7 @@ def _render_files(self):
             + "\n".join("{} {}".format(*f) for f in self._files))
 
     def _render_runscript(self):
-        return "%runscript\n" + "\n".join(self._runscript)
+        return "%runscript\n" + self._runscript
 
     def _render_test(self):
         return "%test\n" + "\n".join(self._test)
@@ -160,3 +169,7 @@ def _fill_parts(self):
         if not self._runscript:
             self._runscript.append(NEURODOCKER_ENTRYPOINT)
         self._parts_filled = True
+
+    def _add_json(self):
+        jsonstr = _get_json_spec_str(self._specs)
+        self._specs['instructions'].append(("run", jsonstr))