Scripts to smoke a full copy of CPAN against a perl branch
Fetching latest commit…
Cannot retrieve the latest commit at this time
This is a very, very crude step-by-step recipe for setting up a full CPAN smoke for multiple branches of perl and generating a regression report from the results. There are various known issues. Among the top ones are that parallelization and restarting comes at a very high cost in terms of redoing work (for all dependencies of the dists to test), and that the whole thing is all but user friendly. --Steffen TODO: Validate that the CPAN config works without host perl config being properly set up. This matters only for bootstrapping. (Update: It does not unless you have at least CPAN::Mini installed.) User Guide ========== - Create a new smoke configuration file (YAML). You can find examples in the config/ subdirectory. See the definition of the various fields below the User Guide. For the sake of this guide, assume that there's two perl's configured that have the names 'reference' and 'test'. Suppose also, that this software is available in ~/cpanbranchtest. - Get a minicpan: (CPAN::Mini required, see next step) $ minicpan -l ~/minicpan/ -r http://your_favourite_cpan_mirror It makes sense NOT to modify/update this minicpan during the test run! - Install all the required smoker modules in an unrelated perl. Can be the system perl, your own, whatever. => Either use the slightly questionable (because of the sudo) $ sudo perl bin/install_prereqs.pl --into-host-perl=`which perl` --config=myconfig.yml" => Or manually install the host_perl modules listed in bin/install_prereqs.pl. - Build the two perls to compare (and install smoking modules into each): $ cd ~/cpanbranchtest $ perl bin/prep_a_perl.pl --config=myconfig.yml You can use the --perl-name=... option to pick one or more perls to build at a time (several --perl-name=... are allowed in one run). - Get a list of distributions to smoke: $ cd ~/cpanbranchtest; perl bin/get_dists.pl --config=myconfig.yml By default, these will be taken from the configured source minicpan and written to the smoke_distributions.txt file of the smoke report output directory (see configuration). You can manually specify dist files or module names in that file instead. - The big, slow step: Run the smoker for each perl: $ cd ~/cpanbranchtest # Ideally one in a screen session each: $ perl bin/smoker.pl --config=myconfig.yml --perlname=reference $ perl bin/smoker.pl --config=myconfig.yml --perlname=test At (presumably) any point during the run, you can kill the smoker process and restart it. It won't reprocess distributions for which a report exists. If you need to start from scratch, use the --restart-from-scratch smoker option. Be aware that restarting will require re-building the distributions that are depended on by those still outstanding, so it can lead to quite some duplicate work. The same gotcha applies to parallelization, btw. - Compare the output $ perl bin/compare-report-dirs.pl --output-dir=html_out --html (This may take a while...) $ firefox html_out/index.html You can also already run the comparison script before both smokers are done. To avoid lots of "missing" vs. "pass" result differences, pass the --skip-missing option to compare-report-dirs.pl. - The smoker will run for a few days. My test run: Two perls, 4-core/8-thread Xeon, 14 smoker processes => 4 days. Slightly overcommitting the machine is not a bad idea since some distributions may block on random stuff such as user input and we don't want our precious CPU time wasted with timeouts, do we? - For setting this up on a new host, check out the bin/cron_* scripts which are an enormously hacky and somewhat hardcoded way of regularly pushing machine status, smoke progress, and the current smoke report to the some web-accessible place. Configuration File ================== Here's a complete one: --- name: 'magicflags2' cpan-mirror: 'file:///home/tsee/perl/minicpan' perl-git-remote: 'file:///home/tsee/bleadperl/perl-ssh' perl-install-base: '/tmp/testperls' smoke-report-output-base: '/home/tsee/cpan_smoke_out' smoke-processes-per-perl: 7 tmpdir: /tmp perls: - perl-name: 'reference' smoke-branch: 'a3f424e51e225fc2e216af0ee9134dbcd2eaf300' grindperl-opt: ~ - perl-name: 'magicflags2' smoke-branch: 'chip/magicflags2' grindperl-opt: ~ Should be straight-forward, but let's walk through the fields: - 'name' indicates the name of the smoke run. It will be used in path names, so don't use funny characters. - 'cpan-mirror' should point at a minicpan or local CPAN mirror that must not change during the smoke run. I generally create a fresh minicpan per smoke run. - 'perl-git-remote' is the URL of a git remote to clone from for building the test perls. - 'perl-install-base' is the base directory where all of the test perls will be installed. - 'smoke-report-output-base' is the base directory under which the directories for the CPAN tester report files will be created. This is the per-module reports, not the final smoke report. - 'smoke-processes-per-perl' indicates the number of processes to dedicate to smoking a single perl. A slight overcommittment of the hardware may make sense since there's still some modules that wait on STDIN until killed by the timeout. - 'tmpdir' (defaults to File::Spec->tmpdir) can be pointed at a custom scratch directory to use for the (large!) work & build directories. - 'perls' is an array of configurations for perls to smoke. Right now, it's only been tested with two -- the reporting script doesn't handle more. TODO => 'perl-name' the name to use to refer to this test perl. This name is used both in paths and on the command line of various tools which operate on a single smoke-perl. => 'smoke-branch' is the commit SHA1 or branch name of the perl to smoke. => 'grindperl-opt' is an array reference (or undef) of command line options to pass to the 'grindperl' script when building the test perls. Something as simple as: grindperl-opt: - '-Dusethreads' should work, but is untested. => 'executable' is an alternative to specifying the 'smoke-branch'. If the 'executable' option is present, the given perl will be used for smoking. Obviously, perls with specified executable won't be picked up by the prep_a_perl script.