Skip to content
Container plugin for Slurm Workload Manager
C Makefile
Branch: master
Clone or download
lukeyeager and flx42 Overload the --container-image option to allow squashfs path (#11)
Co-authored-by: Felix Abecassis <fabecassis@nvidia.com>
Signed-off-by: Felix Abecassis <fabecassis@nvidia.com>
Latest commit 84a31ce Nov 22, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
debian
.gitignore Initial release Aug 22, 2019
.travis.yml
CONTRIBUTING.md
LICENSE
Makefile More verbose logging (#7) Nov 20, 2019
README.md Overload the --container-image option to allow squashfs path (#11) Dec 3, 2019
common.c Use strcat instead of strncat to remove a warning with gcc 9.2.1 Nov 20, 2019
common.h
pyxis_slurmd.c Fix capitalization of "Slurm" in code comments Oct 3, 2019
pyxis_slurmstepd.c Overload the --container-image option to allow squashfs path (#11) Dec 3, 2019
seccomp_filter.h Initial release Aug 22, 2019

README.md

Pyxis

Pyxis is a SPANK plugin for the Slurm Workload Manager. It allows unprivileged cluster users to run containerized tasks through the srun command.

A pyxis is an ancient small box or container.

Benefits

  • Seamlessly execute the user's task in an unprivileged container.
  • Simple command-line interface.
  • Fast Docker image download with support for layers caching and layers sharing across users.
  • Supports multi-node MPI jobs through PMI2 or PMIx (requires Slurm support).
  • Allows users to install packages inside the container.
  • Works with shared filesystems.
  • Does not require cluster-wide management of subordinate user/group ids.

Installation

Pyxis requires the enroot container utility to be installed.

$ # Option 1: quick install from sources
$ sudo make install
$ sudo ln -s /usr/local/share/pyxis/pyxis.conf /etc/slurm-llnl/plugstack.conf.d/pyxis.conf
$ sudo systemctl restart slurmd

$ # Option 2: generate a deb package and install it
$ make orig
$ make deb
$ sudo dpkg -i ../nvslurm-plugin-pyxis_*_amd64.deb
$ sudo ln -s /usr/share/pyxis/pyxis.conf /etc/slurm-llnl/plugstack.conf.d/pyxis.conf
$ sudo systemctl restart slurmd

Usage

Pyxis being a SPANK plugin, the new command-line arguments it introduces are directly added to srun.

$ srun --help
...
      --container-image=[USER@][REGISTRY#]IMAGE[:TAG]|PATH
                              [pyxis] the image to use for the container
                              filesystem. Can be either a docker image given as
                              an enroot URI, or a path to a squashfs file on the
                              remote host filesystem.

      --container-mounts=SRC:DST[:FLAGS][,SRC:DST...]
                              [pyxis] bind mount[s] inside the container. Mount
                              flags are separated with "+", e.g. "ro+rprivate"

      --container-workdir=PATH
                              [pyxis] working directory inside the container
      --container-name=NAME   [pyxis] name to use for saving and loading the
                              container on the host. Unnamed containers are
                              removed after the slurm task is complete; named
                              containers are not. If a container with this name
                              already exists, the existing container is used and
                              the import is skipped.
      --container-mount-home  [pyxis] bind mount the user's home directory.
                              System-level enroot settings might cause this
                              directory to be already-mounted.

      --no-container-mount-home
                              [pyxis] do not bind mount the user's home
                              directory
      --container-remap-root  [pyxis] ask to be remapped to root inside the
                              container. Does not grant elevated system
                              permissions, despite appearances. (default)

      --no-container-remap-root
                              [pyxis] do not remap to root inside the container

Examples

$ # Run a command on a worker node
$ srun grep PRETTY /etc/os-release
PRETTY_NAME="Ubuntu 18.04.2 LTS"

$ # run the same command, but now inside of a container
$ srun --container-image=centos grep PRETTY /etc/os-release
PRETTY_NAME="CentOS Linux 7 (Core)"

$ # mount a file from the host and run the command on it, from inside the container
$ srun --container-image=centos --container-mounts=/etc/os-release:/host/os-release grep PRETTY /host/os-release
PRETTY_NAME="Ubuntu 18.04.2 LTS"

Copyright and License

This project is released under the Apache License Version 2.0.

Issues and Contributing

Reporting Security Issues

When reporting a security issue, do not create an issue or file a pull request.
Instead, disclose the issue responsibly by sending an email to psirt<at>nvidia.com.

You can’t perform that action at this time.