Skip to content
Local runner for Valohai projects
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.circleci
tests
valohai_local_run
.coveragerc
.gitignore
README.md
requirements-dev.in
requirements-dev.txt
setup.cfg
setup.py

README.md

valohai-local-run

CircleCI Codecov PyPI MIT License

This utility allows you to run experiments designed for the Valohai IaaS Machine Learning Platform on local Linux hardware.

Unlike the platform, this tool does not offer any reproducibility, bookkeeping and collaboration features. It is meant for "pre-flight" testing of code packaged for Valohai, and local runs in environments not reachable via the Internet.

Requirements

  • Python 3.4+ with Pip
  • Git
  • Docker (configured to be available for the user running valohai-local-run)

Installation

valohai-local-run is available on the Python PyPI package registry. To install the latest released version, simply run pip install valohai-local-run (optionally with the --user flag to allow for non-root installs). You may also use a Python virtualenv to install in.

You can also install a development version by cloning the repository and running pip install -e . in the working copy directory.

Usage

The syntax of valohai-local-run mostly mirrors that of the Valohai CLI's exec run command.

Assuming the valohai-local-run command is available within your shell's PATH (by way of installation or virtualenv activation, or otherwise – if you installed with pip --user, it may be in ~/.local/bin), you can simply run

$ valohai-local-run step-name --help

where step-name is the name of a step, as described in the valohai.yaml file, to see the syntax for the parameters and inputs, e.g. for the Tensorflow example:

parameters for "Train model":
  --max-steps INTEGER   Number of steps to run the trainer
  --learning-rate FLOAT
                        Initial learning rate
  --dropout FLOAT       Keep probability for training dropout

inputs for "Train model":
  --training-set-images URL
                        Input "training-set-images"
  --training-set-labels URL
                        Input "training-set-labels"
  --test-set-images URL
                        Input "test-set-images"
  --test-set-labels URL
                        Input "test-set-labels"

Other arguments supported by vh exec run are also available; see the full --help output.

The metadata, logs and output files resulting from a run are saved into a timestamped directory. By default the directory is created within valohai-local-outputs in the working directory. This output root path may be changed with the --output-root argument.

Input syntax

Inputs may be HTTP/HTTPS URLs if the requests package is available.
Paths to directories and files are always supported. When all paths are files, multiple repetitions of a single input argument is accepted, and the files are mounted with their original names within the /valohai/inputs/input-name virtual directory.

When a path is a directory, that directory is assumed to be the entirety of that input, i.e. the directory is mounted as /valohai/inputs/input-name. All inputs are mounted read-only.

Running with GPU support

There is tentative support for running on local GPU devices. This relies on the nvidia-docker package; please follow its installation instructions first.

Once nvidia-docker is installed, you can direct valohai-local-run to use it with either the --docker-command or --docker-add-args arguments, as follows:

  • If you are using nvidia-docker 1.x, add --docker-command=nvidia-docker.
  • If you are using nvidia-docker 2.x, add --docker-add-args=--runtime=nvidia.
You can’t perform that action at this time.