This is a very, very crude step-by-step recipe for setting up a full
CPAN smoke for multiple branches of perl and generating a regression
report from the results. There are various known issues. Among the top
ones are that parallelization and restarting comes at a very high cost
in terms of redoing work (for all dependencies of the dists to test),
and that the whole thing is all but user friendly. --Steffen

TODO: Validate that the CPAN config works without host perl config being
properly set up. This matters only for bootstrapping. (Update: It does
not unless you have at least CPAN::Mini installed.)

User Guide
==========

- Create a new smoke configuration file (YAML). You can find examples
  in the config/ subdirectory.
  See the definition of the various fields below the User Guide.
  For the sake of this guide, assume that there's two perl's configured
  that have the names 'reference' and 'test'.
  Suppose also, that this software is available in ~/cpanbranchtest.

- Get a minicpan: (CPAN::Mini required, see next step)
  $ minicpan -l ~/minicpan/ -r http://your_favourite_cpan_mirror
  It makes sense NOT to modify/update this minicpan during the test run!

- Install all the required smoker modules in an unrelated perl. Can be the system perl,
  your own, whatever.
  => Either use the slightly questionable (because of the sudo)
     $ sudo perl bin/install_prereqs.pl --into-host-perl=`which perl` --config=myconfig.yml"
  => Or manually install the host_perl modules listed in bin/install_prereqs.pl.

- Build the two perls to compare (and install smoking modules into each):
  $ cd ~/cpanbranchtest
  $ perl bin/prep_a_perl.pl --config=myconfig.yml
  You can use the --perl-name=... option to pick one or more perls to build at a time
  (several --perl-name=... are allowed in one run).

- Get a list of distributions to smoke:
  $ cd ~/cpanbranchtest; perl bin/get_dists.pl --config=myconfig.yml
  By default, these will be taken from the configured source minicpan and written
  to the smoke_distributions.txt file of the smoke report output directory (see
  configuration).
  You can manually specify dist files or module names in that file instead.

- The big, slow step: Run the smoker for each perl:
  $ cd ~/cpanbranchtest
  # Ideally one in a screen session each:
  $ perl bin/smoker.pl --config=myconfig.yml --perlname=reference
  $ perl bin/smoker.pl --config=myconfig.yml --perlname=test

  At (presumably) any point during the run, you can kill the smoker process
  and restart it. It won't reprocess distributions for which a report exists.
  If you need to start from scratch, use the --restart-from-scratch smoker option.
  Be aware that restarting will require re-building the distributions that are
  depended on by those still outstanding, so it can lead to quite some duplicate
  work. The same gotcha applies to parallelization, btw.

- Compare the output
  $ perl bin/compare-report-dirs.pl --output-dir=html_out --html
  (This may take a while...)
  $ firefox html_out/index.html
  
  You can also already run the comparison script before both smokers are done.
  To avoid lots of "missing" vs. "pass" result differences, pass the
  --skip-missing option to compare-report-dirs.pl.

- The smoker will run for a few days.
  My test run: Two perls, 4-core/8-thread Xeon, 14 smoker processes => 4 days.
  Slightly overcommitting the machine is not a bad idea since some distributions
  may block on random stuff such as user input and we don't want our precious
  CPU time wasted with timeouts, do we?

- For setting this up on a new host, check out the bin/cron_* scripts which are
  an enormously hacky and somewhat hardcoded way of regularly pushing machine
  status, smoke progress, and the current smoke report to the some web-accessible
  place.

Configuration File
==================

Here's a complete one:

---
name: 'magicflags2'
cpan-mirror: 'file:///home/tsee/perl/minicpan'
perl-git-remote: 'file:///home/tsee/bleadperl/perl-ssh'
perl-install-base: '/tmp/testperls'
smoke-report-output-base: '/home/tsee/cpan_smoke_out'
smoke-processes-per-perl: 7
tmpdir: /tmp
perls:
  - perl-name: 'reference'
    smoke-branch: 'a3f424e51e225fc2e216af0ee9134dbcd2eaf300'
    grindperl-opt: ~
  - perl-name: 'magicflags2'
    smoke-branch: 'chip/magicflags2'
    grindperl-opt: ~

Should be straight-forward, but let's walk through the fields:

- 'name' indicates the name of the smoke run. It will be used in path
  names, so don't use funny characters.
- 'cpan-mirror' should point at a minicpan or local CPAN mirror that
  must not change during the smoke run. I generally create a fresh
  minicpan per smoke run.
- 'perl-git-remote' is the URL of a git remote to clone from for
  building the test perls.
- 'perl-install-base' is the base directory where all of the test
  perls will be installed.
- 'smoke-report-output-base' is the base directory under which the
  directories for the CPAN tester report files will be created.
  This is the per-module reports, not the final smoke report.
- 'smoke-processes-per-perl' indicates the number of processes to
  dedicate to smoking a single perl. A slight overcommittment of the
  hardware may make sense since there's still some modules that wait
  on STDIN until killed by the timeout.
- 'tmpdir' (defaults to File::Spec->tmpdir) can be pointed at a custom
  scratch directory to use for the (large!) work & build directories.
- 'perls' is an array of configurations for perls to smoke.
  Right now, it's only been tested with two -- the reporting script
  doesn't handle more. TODO
=> 'perl-name' the name to use to refer to this test perl. This name
   is used both in paths and on the command line of various tools which
   operate on a single smoke-perl.
=> 'smoke-branch' is the commit SHA1 or branch name of the perl to smoke.
=> 'grindperl-opt' is an array reference (or undef) of command line
   options to pass to the 'grindperl' script when building the test
   perls. Something as simple as:
   grindperl-opt:
     - '-Dusethreads'
   should work, but is untested.
=> 'executable' is an alternative to specifying the 'smoke-branch'. If
   the 'executable' option is present, the given perl will be used for
   smoking. Obviously, perls with specified executable won't be picked
   up by the prep_a_perl script.