Dist::Zilla and Moose are widely used by Perl modules and take ages to install. Is there any possibility we could get them preinstalled?
We have tried it before. It takes forever to install and often times out or fails for obscure reasons ("bailing out …" is about as much as cpanm reports). Sorry, it just does not work for us.
What we would much rather do at this point is hosting our network-local CPAN mirror. Disk space is not an issue. Would you be interested in helping us with that (we only need someone knowledgeable to explain what we should do and why)? It will speed up installation of much more than Dist::Zilla.
I'm one of the maintainers of http://metacpan.org - I'll chat to some of the guys in that project who know more about this than I do and get back to you
OK - it's easy :)
You can setup a local mirror with instant mirroring: https://github.com/perlorg/cpanorg/wiki/Instant-update-mirroring
(ignore the bit about "just in testing" - it's been live for years)
Then to make cpanm use the local mirror, you can create an alias, eg:
alias travispan='cpanm --mirror http://mycompany.example.com/CPAN --mirror-only'
See https://metacpan.org/module/cpanm for more config options.
Let me know if anything isn't working or if you need more info
If we don't pass --mirror-only, will main cpan and project-specific mirrors be used? That's how our Maven mirror works, it merges global defaults (that point to maven.travis-ci.org for a few popular repositories) and whatever may be in the project build file.
Rereading the docs, you probably want --mirror http://mycompany.example.com/CPAN --mirror http://search.cpan.org/CPAN instead.
--mirror http://mycompany.example.com/CPAN --mirror http://search.cpan.org/CPAN
The --mirror-only option is for offline use or if you have a private CPAN. Without this option, cpanm will use http://cpanmetadb.plackperl.org/ to find the package name which it needs to install.
By providing two --mirror options, it'll try your local mirror first, then failover to search.cpan.org
You might need a --cascade-search option as well. BTW, I had no problem installing Dist::Zilla using:
cpanm --quiet --notest Dist::Zilla
# only needed on a per-distro basis
dzil authordeps | cpanm --quiet --notest
Once we put our mirror in operation, I hope we can preinstall Dist::Zilla without any issues. By the way, we would definitely appreciate some help with testing it before we instruct Travis CI ENV to use it by setting PERL_CPANM_OPT.
Sure, no problem testing - just let us know when the mirror is setup and what we need to do to try it out.
@clintongormley http://cpan.mirrors.travis-ci.org is up and it takes 56 seconds to install Dist::Zilla from it for me locally. On travis-ci.org worker machines, it should be a few times faster over LAN. Thank you so much for helping us figure out CPAN mirroring.
We are not familiar with the Perl ecosystem, what modules Moose consists of? Anything else you want us to preinstall (like commonly used testing libraries or anything else CI related)?
Preinstall Moose, per discussion in #72
That's brilliant! Thanks so much. My test times dropped from 22 min to 8 min! (which is good for you resources too)
PERL_CPANM_OPT="--mirror http://cpan.mirrors.travis-ci.org --mirror http://cpan.cpantesters.org/ --mirror http://search.cpan.org/CPAN --cascade-search --notest --force --skip-satisfied"
I've been thinking about what modules it would be useful to have preinstalled, and Moose, Dist::Zilla, Test::More/Exception/Deep/Moose all spring to mind.
The problem is keeping them up to date without causing you guys a massive maintenance headache. Perhaps the way to do it is for me/us to maintain a single module on CPAN called Task::TravisCI. Then all you need to do is to install that module regularly, and it will pull in the latest versions of anything we think is useful.
How does that sound?
I don't think that would change anything. Those packages will get out of date between our VM image upgrades but that's typically a few days or maybe a week or two max. Even if we preinstall a single meta module that depends on those you've listed, they still will get outdated, as far as I understand?
Shouldn't cpanm check for updates and install them if needed?
Yes it will. I just thought it'd be easier for you to target a single module rather than giving you a long list of modules to add to your recipe. The ones I listed above are probably the most frequently used, although I'd possibly also add Module::Builder which can be quite heavy to install.
@clintongormley if you want to tweak our list of preinstalled modules, just edit this line and submit a PR.
As far as the caching/updating issue goes, if cpanm does not check for updates, I am afraid preinstalling anything may be problematic. But we also can solve problems as they arise and go with Dist::Zilla, Moose and a few popular test packages,
then see how it goes.
Mirroring now makes most of the difference anyway.
By the way, new images with PERL_CPANM_OPT set and travis-build/travis-worker updates will be deployed within 24 hours.
OK great. To be clear: cpanm WILL check for new versions.
Yeah, I wouldn't recommend that full string for the PERL_CPANM_OPT line:
There's also --skip-satisfied. This will basically skip any module if any sort of version is installed. This might be a good thing, actually, since it saves time, and the "mostly up-to-date" version will probably be installed, anyway. Otherwise, --skip-installed will make sure that it's the latest version before it skips.
I have a question about how PERL_CPANM_OPT is used when you also pass switches to cpanm on the command line. Are two sets of switches merged or cpanm will only use those passed explicitly?
We have both PERL_CPANM_OPT set and our Perl builder also passes some options to install: and test: commands.
PERL_CPANM_OPT is merged in. There are --no-* switches to cancel out the env switches as well. I'd recommend:
PERL_CPANM_OPT="--mirror http://cpan.mirrors.travis-ci.org --mirror http://cpan.cpantesters.org/ --mirror http://search.cpan.org/CPAN --cascade-search --skip-satisfied"
Update PERL_CPANM_OPT per discussion in #72
@michaelklishin Thanks for the big improvment!
Some useful modules to add to the base environment: Test::Pod, Test::Pod::Coverage, Test::Exception, Test::Kwalitee.
Preinstall a few more popular and/or heavyweight modules per discussi…
…on in #72
@SineSwiper suggested above to use --skip-satisfied in the PERL_CPANM_OPT.
This is a good idea if the aim is to speed-up the build. However the aim of Travis-CI is integration testing. So in this context I think it would be better to always test against the latest versions of the CPAN, which is cpanm's default without this flag.
So it would be better to remove the --skip-satisfied flag.
That option was never added. We set PERL_CPANM_OPT to only includes mirrors now. Thank you for the clarification, though!
@dolmen: I disagree, since a CI should test with various different environments of modules and also test that Prereqs are accurate. One of the things I'm going to be pushing with Dist::Zilla::Plugins::TravisCI is MVDT, or "Minimum Version Dependency Testing", as is it very common to just list prereqs as "any version", which would break for certain sets of modules. Some Debian Lenny releases still exist, and they might contain either the minimum within Perl core or modules that were release in 2007-8.
This may be a moot point with the pre-installed modules on the Perl Brew list (which are mostly just needs for the builder), but if it fails on 5.10 because the core module is too old, you should really know about it.
Barring that, if we can't agree with the --skip-satisfied option, then at least the --skip-installed option should be put into PERL_CPANM_OPT. That will make sure that it's not going to re-download the latest version if that is already installed.
Given that any user of travis can set their own PERL_CPANM_OPT I wouldn't make it the default. Let people decide what they want to add
@michaelklishin The --skip-satisfied option is used in /ci_environment/perlbrew/files/default/perlbrew.sh.
However I don't know yet which environment is impacted by this file. Is is just for installing the base modules, or is it also the environment in which user commands (from travis.yml) are run?
@clintongormley: We are also discussing how the option impacts which modules are installed in the base system, so which versions of modules above the core are installed before the user commands may be involved. If some recent versions from the CPAN are installed over core modules and the user wants to override this, he will have to downgrade those modules. So the balance is between a base system that have the latest modules from the CPAN and a base system that has only the missing things from the perl core. I'm in favor of the first option: I want the fixes of the perl core modules that have been published on the CPAN.