45216 continuousbenchmarking 20160926 asb 4

[ashawnbandy-te-tfb]

The work in this PR introduces a basic facility for continuous benchmarking. run-continuously is essentially a while-loop that makes 'life-cycle' calls in addition to initiating a benchmarking run. Additionally, an example of an upstart process configuration is included.

Below are some relevant notes from a recent commit:

toolset/run-continuously.sh is simply a while loop that
removes and rebuilds the framework benchmark suite and
then runs the benchmarks.

There are five environment variables to be set:

• TFB_REPOPARENT the absolute path to the folder containing
  the repository
• TFB_REPONAME the name of the repository folder
• TFB_REPOURI the URI of the repository
• TFB_REPOBRANCH the branch of the repository to use
• TFB_LOGSFOLDER the folder to archive log files

There are four life cycle stages in run-continuously:

• tear-down-environment.sh called upon to remove the existing
  environment
• rebuild-environment.sh called upon to rebuild the environment
• pre-run-tests scripts in this folder are called
  before the benchmarks are run
• post-run-tests scripts in this folder are called
  after the benchmarks are run
• Notes -

run-continuously.sh generally assumes that a clone of the
appropriate repo and branch exist and have the list
scripts available and an appropriate copy of benchmark.cfg
in place. Some effort is made to support starting states
that differ but those are not intended to be the general
case for use.

At the end of each run (post-run-tests) there are two python scripts. One zips the results.json file and sends it to a specified email address. The other makes copies of the logs for each frameworks (each are independently zipped while being copied).

Changes made following the previous PR for this work:

1. Added @knewmanTE's work towards tagging the results with a commit.
2. The mailing list environment variables are now split into two, on for TO: and one for FROM:
3. I removed a duplicated block of code in email-results.py.

[ashawnbandy-te-tfb]

Also, I will defer the work related to @knewmanTE's PR #2283 until after it has been merged into master.

[ashawnbandy-te-tfb]

Also, I will defer the work related to @knewmanTE's PR #2283 until after it has been merged into master.

[knewmanTE]

Merge'd!

[knewmanTE]

Merge'd!