This package contains the infrastructure powering the @nanosoldier CI bot used by the Julia language.
Submitting CI jobs via comments
If you're a collaborator in the JuliaLang/julia repository, you can submit CI jobs to the Julia Lab's Nanosoldier cluster at MIT by commenting on commits or pull requests. The @nanosoldier bot looks for a special "trigger phrase" in your comment, and if the trigger phrase is found, it is parsed by the bot to configure and submit a CI job.
The trigger phrase syntax is:
@nanosoldier `command(args..., kwargs...)`
Here's an example of a comment that might trigger a benchmarking job from a commit:
Let's see whether the commit I'm commenting on is faster than the master branch for the linear algebra benchmarks: @nanosoldier `runbenchmarks("linalg", vs = ":master")`
When a job is completed, @nanosoldier will reply to your comment to tell you how the job went and link you to any relevant results.
Note that only one job can be triggered per comment.
Available job types
CI jobs are implemented in this package as subtypes of
Nanosoldier.AbstractJob. See here for a description of the interface new job types need to implement.
BenchmarkJob has the following execution cycle:
- Pull in the JuliaLang/julia repository and build the commit specified by the context of the trigger phrase.
- Using the new Julia build, fetch the
nanosoldierbranch of the BaseBenchmarks repository and run the benchmarks specified by the trigger phrase.
- If the trigger phrase specifies a commit to compare against, build that version of Julia and perform step 2 using the comparison build.
- Upload a markdown report to the BaseBenchmarkReports repository.
BenchmarkJob is triggered with the following syntax:
@nanosoldier `runbenchmarks(tag_predicate, vs = "ref")`
vs keyword argument is optional, and is used to determine whether or not the comparison step (step 3 above) is performed.
The tag predicate is used to decide which benchmarks to run, and supports the syntax defined by the tagging system implemented in the BenchmarkTools package. Additionally, you can run all benchmarks by using the keyword
vs keyword argument takes a reference string which can points to a Julia commit to compare against. The following syntax is supported for reference string:
":branch": the head commit of the branch named
branchin the current repository (
"@sha": the commit specified by
shain the current repository (
"owner/repo:branch": the head commit of the branch named
branchin the repository
"owner/repo@sha": the commit specified by
shain the repository
BenchmarkJob is complete, the results are uploaded to the
BaseBenchmarkReports repository. Each job
has its own directory for results. This directory contains the following items:
report.mdis a markdown report that summarizes the job results
data.tar.gzcontains raw timing data in JSON format. To untar this file, run
tar -xzvf data.tar.gz. You can analyze this data using the BenchmarkTools package.
logsis a directory containing the build logs and benchmark execution logs for the job.
Here are some examples of comments that trigger a
BenchmarkJob in various contexts:
I want to run benchmarks tagged "array" on the current commit. @nanosoldier `runbenchmarks("array")` If this comment is on a specific commit, benchmarks will run on that commit. If it's in a PR, they will run on the head/merge commit of the PR. If it's on a diff, they will run on the commit associated with the diff.
I want to run benchmarks tagged "array" on the current commit, and compare the results with the results of running benchmarks on commit 858dee2b09d6a01cb5a2e4fb2444dd6bed469b7f. @nanosoldier `runbenchmarks("array", vs = "@858dee2b09d6a01cb5a2e4fb2444dd6bed469b7f")`
I want to run benchmarks tagged "array", but not "simd" or "linalg", on the current commit. I want to compare the results against those of the release-0.4 branch. @nanosoldier `runbenchmarks("array" && !("simd" || "linalg"), vs = ":release-0.4")`
I want to run all benchmarks on the current commit. I want to compare the results against a commit on my fork. @nanosoldier `runbenchmarks(ALL, vs = "ararslan/julia@c70ab26bb677c92f0d8e0ae41c3035217a4b111f")`
I want to run all benchmarks on the current commit. I want to compare the results against the head commit of my fork's branch. @nanosoldier `runbenchmarks(ALL, vs = "ararslan/julia:mybranch")`
The development of the Nanosoldier benchmarking platform was supported in part by the US Army Research Office through the Institute for Soldier Nanotechnologies under Contract No. W911NF-07-D0004.