Skip to content

Commit

Permalink
Changes 'experiment' by 'pipeline'
Browse files Browse the repository at this point in the history 
fixes #183
  • Loading branch information
@ivotron
ivotron committed Dec 4, 2017
1 parent 2e0d70c commit 871a050
Show file tree
Hide file tree
Showing 9 changed files with 114 additions and 152 deletions.
95 changes: 42 additions & 53 deletions docs/ci/popperci.md
View file
Original file line number Diff line number Diff line change
@@ -1,20 +1,20 @@
# Popper and CI systems

By following a convention for structuring the files of a project,
experiment execution and validation can be automated without the need
for manual intervention. In addition to this, the status of an
experiment (integrity over time) can be tracked by a CI service. In
this section we describe the workflow that one follows in order to
make an experiment suitable for automation on CI systems.
pipeline execution and validation can be automated without the need
for manual intervention. In addition to this, the status of a pipeline
(integrity over time) can be tracked by a CI service. In this section
we describe the workflow that one follows in order to make a pipeline
suitable for automation on CI systems.

## Experiment Folder Structure
## Pipeline Folder Structure

A minimal experiment folder structure for an experiment is shown
below:
A minimal pipeline folder structure for an experiment or analysis is
shown below:

```{#lst:repo .bash caption="Basic structure of a Popper repository."}
$> tree -a paper-repo/experiments/myexp
paper-repo/experiments/myexp/
$> tree -a paper-repo/pipelines/myexp
paper-repo/pipelines/myexp/
|-- README.md
|-- .popper.yml
|-- run.sh
Expand All @@ -23,34 +23,33 @@ paper-repo/experiments/myexp/
|-- teardown.sh
```

Every experiment has `setup.sh`, `run.sh` and `validate.sh` scripts
that serve as the interface to the experiment. All these return
non-zero exit codes if there's a failure. In the case of
`validate.sh`, this script should print to standard output one line
per validation, denoting whether a validation passed or not. In
general, the form for validation results is `[true|false] <statement>`
(see examples below).
Every pipeline has `setup.sh`, `run.sh` and `validate.sh` scripts that
serve as the interface to the pipeline. All these return non-zero exit
codes if there's a failure. In the case of `validate.sh`, this script
should print to standard output one line per validation, denoting
whether a validation passed or not. In general, the form for
validation results is `[true|false] <statement>` (see examples below).

```{#lst:validations .bash caption="Example output of validations."}
[true] algorithm A outperforms B
[false] network throughput is 2x the IO bandwidth
```

The [PopperCLI](https://github.com/systemslab/popper/popper) tool
includes an `experiment init` subcommand that can be executed to
scaffold an experiment with the above structure. The syntax of this
includes an `pipeline init` subcommand that can be executed to
scaffold an pipeline with the above structure. The syntax of this
command is the following:

```bash
popper experiment init <name>
popper pipeline init <name>
```

Where `<name>` is the name of the experiment to initialize.
Where `<name>` is the name of the pipeline to initialize.

## CI System Configuration

In this section we describe how to configure a CI system so that
Popper experiments can be continuously validated. The next section
Popper pipelines can be continuously validated. The next section
describes the multiple steps that are executed as part of this
validation process.

Expand Down Expand Up @@ -94,67 +93,57 @@ We then can trigger an execution by pushing to GitHub:
git push
```

After this, one can go to the TravisCI website to see your experiments
After this, one can go to the TravisCI website to see your pipelines
being executed.

## CI Functionality

The following is the list of steps that are verified when validating
an experiment:
an pipeline:

<!--
1. Ensure that every versioned dependency is healthy. For example,
ensure that external repos can be cloned correctly.
2. Check the integrity of every special subfolder (see previous
subsection).
-->

1. For every experiment, trigger an execution (invoke `setup.sh`
1. For every pipeline, trigger an execution (invoke `setup.sh`
followed by `run.sh`).
2. After the experiment finishes, execute validations on the output
2. After the pipeline finishes, execute validations on the output
(invoke `validate.sh`).
3. Keep track of every experiment and report their status.
3. Keep track of every pipeline and report their status.
4. Execute `teardown.sh`

There are three possible statuses for every experiment: `FAIL`, `PASS`
There are three possible statuses for every pipeline: `FAIL`, `PASS`
and `GOLD`. There are two possible values for the status of a
validation, `FAIL` or `PASS`. When the experiment status is `FAIL`,
this list of validations is empty since the experiment execution has
failed and validations are not able to execute at all. When the
experiment status' is `GOLD`, the status of all validations is `PASS`.
When the experiment runs correctly but one or more validations fail
(experiment's status is `PASS`), the status of one or more validations
is `FAIL`.
validation, `FAIL` or `PASS`. When the pipeline status is `FAIL`, this
list of validations is empty since the pipeline execution has failed
and validations are not able to execute at all. When the pipeline
status' is `GOLD`, the status of all validations is `PASS`. When the
pipeline runs correctly but one or more validations fail (pipeline's
status is `PASS`), the status of one or more validations is `FAIL`.

## Popper Badges

We maintain a badging service that can be used to keep track of the
status of an experiment. In order to enable this, the
`--enable-badging` flag has to be passed to the `popper ci`
subcommand.
status of a pipeline. In order to enable this, the `--enable-badging`
flag has to be passed to the `popper ci` subcommand.

![Badging service.](/figures/cibadges.png)

Badges are commonly used to denote the status of a software project
with respect to certain aspect, e.g. whether the latest version can be
built without errors, or the percentage of code that unit tests cover
(code coverage). Badges available for Popper are shown in the above
figure. If badging is enabled, after the execution of an experiment,
the status of an experiment is recorded in the badging server, which
keeps track of statuses for every revision of every experiment.
figure. If badging is enabled, after the execution of a pipeline, the
status of a pipeline is recorded in the badging server, which keeps
track of statuses for every revision of ever pipeline.

Users can include a link to the badge in the `README` page of an
experiment, which can be displayed on the web interface of the version
Users can include a link to the badge in the `README` page of a
pipeline, which can be displayed on the web interface of the version
control system (GitHub in this case). The CLI tool can generate links
for experiments:
for pipelines:

```bash
popper badge <exp>
```

Which prints to `stdout` the text that should be added to the `README`
file of the experiment.
file of the pipeline.

## Testing Locally

Expand Down
49 changes: 11 additions & 38 deletions docs/protocol/getting_started.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
_Popper_ is a convention for organizing an academic article's
artifacts following a [DevOps](https://en.wikipedia.org/wiki/DevOps)
approach, with the goal of making it easy for others (and yourself!)
to repeat an experiment.
to repeat an experiment or analysis pipeline.

## Quickstart Guide

Expand Down Expand Up @@ -42,9 +42,9 @@ git add .
git commit -m 'adds .popper.yml file'
```

### New experiment
### New pipeline

Initialize experiment using `init` (scaffolding):
Initialize pipeline using `init` (scaffolding):

```bash
popper init myexp
Expand All @@ -53,43 +53,16 @@ popper init myexp
Show what this did:

```bash
ls -l experiments/myexp
ls -l pipelines/myexp
```

Commit the "empty" experiment:
Commit the "empty" pipeline:

```bash
git add experiments/myexp
git add pipelines/myexp
git commit -m 'adding myexp scaffold'
```

### Add existing experiment

List available experiment templates:

```bash
popper search
```

Show information about an experiment:

```bash
popper info blis
```

Import an available experiment:

```bash
popper add blis
```

Commit the new experiment:

```bash
git add experiments/blis
git commit -m 'adding blis baseline'
```

### Popper check

Run popper check:
Expand All @@ -100,14 +73,14 @@ popper check

> **NOTE:** By default, `popper check` runs all commands directly on
the host. We recommend running an isolated environment. In order to do
this, one can create an experiment using the `--env` flag of the
`popper init` command. For example, `popper init <exp>
--env=alpine-3.4` runs a command inside an `alpine-3.4` container.
this, one can create a pipeline using the `--env` flag of the `popper
init` command. For example, `popper init <pipeline> --env=alpine-3.4`
runs a command inside an `alpine-3.4` container.

Once an experiment is checked, one can show the logs:
Once a pipeline is checked, one can show the logs:

```bash
ls -l experiments/blis/popper_logs
ls -l pipelines/myexp/popper_logs
```

### Adding Project to GitHub
Expand Down
16 changes: 8 additions & 8 deletions popper/_check/check.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ def execute(stage, timeout):
return p.poll()


def check_experiment(skip, timeout):
def check_pipeline(skip, timeout):
check_output('rm -rf popper_logs/ popper_status', shell=True)
check_output('mkdir -p popper_logs/', shell=True)

Expand Down Expand Up @@ -97,12 +97,12 @@ def __getattr__(self, attr):
args = parser.parse_args()
sys.stdout = Unbuffered(sys.stdout)

if path.isdir('./experiments'):
for f in os.listdir('experiments'):
if not f.startswith('.') and path.isdir('experiments/' + f):
print('\nChecking experiment ' + f)
os.chdir('experiments/' + f)
check_experiment(args.skip, int(args.timeout))
if path.isdir('./pipelines'):
for f in os.listdir('pipelines'):
if not f.startswith('.') and path.isdir('pipelines/' + f):
print('\nChecking pipeline ' + f)
os.chdir('pipelines/' + f)
check_pipeline(args.skip, int(args.timeout))
os.chdir('../../')
else:
check_experiment(args.skip, int(args.timeout))
check_pipeline(args.skip, int(args.timeout))
18 changes: 9 additions & 9 deletions popper/add.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ import (
"github.com/spf13/cobra"
)

func addExperiment(templateType string, templateName string, folder string) {
func addPipeline(templateType string, templateName string, folder string) {
checkTemplateFolderExists(templateType)

if sh.Test("dir", folder) {
Expand All @@ -24,8 +24,8 @@ func addExperiment(templateType string, templateName string, folder string) {
}

var addCmd = &cobra.Command{
Use: "add <experiment> [<name>]",
Short: "Add an experiment to the project",
Use: "add <pipeline> [<name>]",
Short: "Add a pipeline to the project",
Long: ``,
Run: func(cmd *cobra.Command, args []string) {
expname := ""
Expand All @@ -34,23 +34,23 @@ var addCmd = &cobra.Command{
} else if len(args) == 2 {
expname = args[1]
} else {
log.Fatalln("Incorrect number of arguments, type 'popper experiment add --help'")
log.Fatalln("Incorrect number of arguments, type 'popper add --help'")
}
if !sh.Test("dir", ".git") {
log.Fatalln("Can't find .git folder. Are you on the root folder of project?")
}

if strings.HasPrefix(args[0], "paper-") {
addExperiment("paper", args[0], "paper/")
addPipeline("paper", args[0], "paper/")
} else {
// create experiments folder if it doesn't exist
if err := sh.Command("mkdir", "-p", "experiments/").Run(); err != nil {
// create pipelines folder if it doesn't exist
if err := sh.Command("mkdir", "-p", "pipelines/").Run(); err != nil {
log.Fatalln(err)
}
addExperiment("experiments", args[0], "experiments/"+expname)
addPipeline("pipelines", args[0], "pipelines/"+expname)
}

fmt.Println("Added " + expname + " to experiments/ folder.")
fmt.Println("Added " + expname + " to pipelines/ folder.")
},
}

Expand Down
14 changes: 7 additions & 7 deletions popper/check.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ func runInDocker(checkFlags []string, checkEnv string) {
}

func runCheck() {
expName, err := getExperimentName()
expName, err := getPipelineName()
if err != nil {
log.Fatalln(err)
}
Expand Down Expand Up @@ -86,11 +86,11 @@ func runCheck() {

var checkCmd = &cobra.Command{
Use: "check",
Short: "Run experiment and report on its status",
Long: `Executes an experiment in its corresponding environment (host or docker). If using docker,
Short: "Run pipeline and report on its status",
Long: `Executes an pipeline in its corresponding environment (host or docker). If using docker,
environment variables and folders can be made available inside the container by using -e
and -v flags respectively. These flags are passed down to the 'docker run' command. The
experiment folder is bind-mounted. If the environment is 'host', the -v and -e flags are
pipeline folder is bind-mounted. If the environment is 'host', the -v and -e flags are
ignored.`,
Run: func(cmd *cobra.Command, args []string) {
if len(args) != 0 {
Expand All @@ -103,11 +103,11 @@ ignored.`,
func init() {
RootCmd.AddCommand(checkCmd)

checkCmd.Flags().StringSliceVarP(&environment, "environment", "e", []string{}, `Environment variable available to the experiment. Can be
checkCmd.Flags().StringSliceVarP(&environment, "environment", "e", []string{}, `Environment variable available to the pipeline. Can be
given multiple times. This flag is ignored when the environment
is 'host'.`)
checkCmd.Flags().StringSliceVarP(&volume, "volume", "v", []string{}, `Volume available to the experiment. Can be given multiple times
checkCmd.Flags().StringSliceVarP(&volume, "volume", "v", []string{}, `Volume available to the pipeline. Can be given multiple times
This flag is ignored when the environment is 'host'.`)
checkCmd.Flags().StringVarP(&skip, "skip", "s", "", "Comma-separated list of stages to skip.")
checkCmd.Flags().StringVarP(&timeout, "timeout", "t", "36000", "Timeout limit for experiment in seconds.")
checkCmd.Flags().StringVarP(&timeout, "timeout", "t", "36000", "Timeout limit for pipeline in seconds.")
}

0 comments on commit 871a050

Please sign in to comment.