Preinstall Dist::Zilla and Moose for Perl #72

Closed
clintongormley opened this Issue Jun 24, 2012 · 25 comments

Comments

Projects
None yet
4 participants
Contributor

clintongormley commented Jun 24, 2012

Hiya

Dist::Zilla and Moose are widely used by Perl modules and take ages to install. Is there any possibility we could get them preinstalled?

ta

Contributor

michaelklishin commented Jun 25, 2012

We have tried it before. It takes forever to install and often times out or fails for obscure reasons ("bailing out …" is about as much as cpanm reports). Sorry, it just does not work for us.

Contributor

michaelklishin commented Jun 25, 2012

What we would much rather do at this point is hosting our network-local CPAN mirror. Disk space is not an issue. Would you be interested in helping us with that (we only need someone knowledgeable to explain what we should do and why)? It will speed up installation of much more than Dist::Zilla.

Contributor

clintongormley commented Jun 25, 2012

I'm one of the maintainers of http://metacpan.org - I'll chat to some of the guys in that project who know more about this than I do and get back to you

Contributor

clintongormley commented Jun 25, 2012

OK - it's easy :)

You can setup a local mirror with instant mirroring: https://github.com/perlorg/cpanorg/wiki/Instant-update-mirroring
(ignore the bit about "just in testing" - it's been live for years)

Then to make cpanm use the local mirror, you can create an alias, eg:

alias travispan='cpanm --mirror http://mycompany.example.com/CPAN --mirror-only'

See https://metacpan.org/module/cpanm for more config options.

Let me know if anything isn't working or if you need more info

Contributor

michaelklishin commented Jun 25, 2012

If we don't pass --mirror-only, will main cpan and project-specific mirrors be used? That's how our Maven mirror works, it merges global defaults (that point to maven.travis-ci.org for a few popular repositories) and whatever may be in the project build file.

Contributor

clintongormley commented Jun 25, 2012

Rereading the docs, you probably want --mirror http://mycompany.example.com/CPAN --mirror http://search.cpan.org/CPAN instead.

The --mirror-only option is for offline use or if you have a private CPAN. Without this option, cpanm will use http://cpanmetadb.plackperl.org/ to find the package name which it needs to install.

By providing two --mirror options, it'll try your local mirror first, then failover to search.cpan.org

You might need a --cascade-search option as well. BTW, I had no problem installing Dist::Zilla using:

cpanm --quiet --notest Dist::Zilla
# only needed on a per-distro basis
dzil authordeps | cpanm --quiet --notest
Contributor

michaelklishin commented Jul 14, 2012

Once we put our mirror in operation, I hope we can preinstall Dist::Zilla without any issues. By the way, we would definitely appreciate some help with testing it before we instruct Travis CI ENV to use it by setting PERL_CPANM_OPT.

Contributor

clintongormley commented Jul 14, 2012

Sure, no problem testing - just let us know when the mirror is setup and what we need to do to try it out.

Contributor

michaelklishin commented Jul 21, 2012

@clintongormley http://cpan.mirrors.travis-ci.org is up and it takes 56 seconds to install Dist::Zilla from it for me locally. On travis-ci.org worker machines, it should be a few times faster over LAN. Thank you so much for helping us figure out CPAN mirroring.

We are not familiar with the Perl ecosystem, what modules Moose consists of? Anything else you want us to preinstall (like commonly used testing libraries or anything else CI related)?

Contributor

clintongormley commented Jul 21, 2012

That's brilliant! Thanks so much. My test times dropped from 22 min to 8 min! (which is good for you resources too)

http://travis-ci.org/#!/clintongormley/Elastic-Model/builds

I used:

PERL_CPANM_OPT="--mirror http://cpan.mirrors.travis-ci.org  --mirror http://cpan.cpantesters.org/ --mirror http://search.cpan.org/CPAN --cascade-search --notest --force --skip-satisfied" 

I've been thinking about what modules it would be useful to have preinstalled, and Moose, Dist::Zilla, Test::More/Exception/Deep/Moose all spring to mind.

The problem is keeping them up to date without causing you guys a massive maintenance headache. Perhaps the way to do it is for me/us to maintain a single module on CPAN called Task::TravisCI. Then all you need to do is to install that module regularly, and it will pull in the latest versions of anything we think is useful.

How does that sound?

Contributor

michaelklishin commented Jul 21, 2012

I don't think that would change anything. Those packages will get out of date between our VM image upgrades but that's typically a few days or maybe a week or two max. Even if we preinstall a single meta module that depends on those you've listed, they still will get outdated, as far as I understand?

Shouldn't cpanm check for updates and install them if needed?

Contributor

clintongormley commented Jul 21, 2012

Yes it will. I just thought it'd be easier for you to target a single module rather than giving you a long list of modules to add to your recipe. The ones I listed above are probably the most frequently used, although I'd possibly also add Module::Builder which can be quite heavy to install.

Contributor

michaelklishin commented Jul 21, 2012

@clintongormley if you want to tweak our list of preinstalled modules, just edit this line and submit a PR.

As far as the caching/updating issue goes, if cpanm does not check for updates, I am afraid preinstalling anything may be problematic. But we also can solve problems as they arise and go with Dist::Zilla, Moose and a few popular test packages,
then see how it goes.

Mirroring now makes most of the difference anyway.

Contributor

michaelklishin commented Jul 21, 2012

By the way, new images with PERL_CPANM_OPT set and travis-build/travis-worker updates will be deployed within 24 hours.

Contributor

clintongormley commented Jul 21, 2012

OK great. To be clear: cpanm WILL check for new versions.

Yeah, I wouldn't recommend that full string for the PERL_CPANM_OPT line:

  • --notest This is good for installing certain prereqs (like Dist::Zilla or Module::Build), but you don't want this as default, since that would invalidate the point of the CI testing. Take it out of here and let the user choose when to use it.
  • --force This will force an install, even if the test fails. Might as well just stop the process if it fails, since that saves time.

There's also --skip-satisfied. This will basically skip any module if any sort of version is installed. This might be a good thing, actually, since it saves time, and the "mostly up-to-date" version will probably be installed, anyway. Otherwise, --skip-installed will make sure that it's the latest version before it skips.

Contributor

michaelklishin commented Jul 21, 2012

I have a question about how PERL_CPANM_OPT is used when you also pass switches to cpanm on the command line. Are two sets of switches merged or cpanm will only use those passed explicitly?

We have both PERL_CPANM_OPT set and our Perl builder also passes some options to install: and test: commands.

PERL_CPANM_OPT is merged in. There are --no-* switches to cancel out the env switches as well. I'd recommend:

PERL_CPANM_OPT="--mirror http://cpan.mirrors.travis-ci.org --mirror http://cpan.cpantesters.org/ --mirror http://search.cpan.org/CPAN --cascade-search --skip-satisfied" 

dolmen commented Jul 22, 2012

@michaelklishin Thanks for the big improvment!
Some useful modules to add to the base environment: Test::Pod, Test::Pod::Coverage, Test::Exception, Test::Kwalitee.
Also Dist::Zilla::Plugin::Bootstrap::lib.

dolmen commented Jul 23, 2012

@SineSwiper suggested above to use --skip-satisfied in the PERL_CPANM_OPT.
This is a good idea if the aim is to speed-up the build. However the aim of Travis-CI is integration testing. So in this context I think it would be better to always test against the latest versions of the CPAN, which is cpanm's default without this flag.
So it would be better to remove the --skip-satisfied flag.

Contributor

michaelklishin commented Jul 23, 2012

That option was never added. We set PERL_CPANM_OPT to only includes mirrors now. Thank you for the clarification, though!

@dolmen: I disagree, since a CI should test with various different environments of modules and also test that Prereqs are accurate. One of the things I'm going to be pushing with Dist::Zilla::Plugins::TravisCI is MVDT, or "Minimum Version Dependency Testing", as is it very common to just list prereqs as "any version", which would break for certain sets of modules. Some Debian Lenny releases still exist, and they might contain either the minimum within Perl core or modules that were release in 2007-8.

This may be a moot point with the pre-installed modules on the Perl Brew list (which are mostly just needs for the builder), but if it fails on 5.10 because the core module is too old, you should really know about it.

Barring that, if we can't agree with the --skip-satisfied option, then at least the --skip-installed option should be put into PERL_CPANM_OPT. That will make sure that it's not going to re-download the latest version if that is already installed.

Contributor

clintongormley commented Jul 23, 2012

Given that any user of travis can set their own PERL_CPANM_OPT I wouldn't make it the default. Let people decide what they want to add

dolmen commented Jul 24, 2012

@michaelklishin The --skip-satisfied option is used in /ci_environment/perlbrew/files/default/perlbrew.sh.
However I don't know yet which environment is impacted by this file. Is is just for installing the base modules, or is it also the environment in which user commands (from travis.yml) are run?

@clintongormley: We are also discussing how the option impacts which modules are installed in the base system, so which versions of modules above the core are installed before the user commands may be involved. If some recent versions from the CPAN are installed over core modules and the user wants to override this, he will have to downgrade those modules. So the balance is between a base system that have the latest modules from the CPAN and a base system that has only the missing things from the perl core. I'm in favor of the first option: I want the fixes of the perl core modules that have been published on the CPAN.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment