Add perl package

[hartzell]

This needs feedback, see my questions below (thanks!)

Add perl package, based on work by justintoo. He had too many things lumped into his pull request, this just adds a perl package.

W.r.t. extentions: I need to understand Spack's extension mechanism and see if it makes sense for Perl.

This supports the current releases on the past three minor branches.

It runs perl's tests before installing.

It installs cpanm alongside the core (which makes building on top of this perl much simpler). Can be disabled via a variant.

Questions:

1. This is my first crack at a "new" (though based on other work) package file. What does it need in the header/comments.
2. I want to install a known version of App::cpanminus alongside the core. I've currently hardcoded the version. I thought I could use a non-boolean variant, but that change looks like it's still waiting to be merged. Do I have other alternatives?
3. Running perl's tests takes a while, but it's the only real way to know if you have a functional build (until I added it, I thought that 5.18.4 was working; turns out that release won't build w/ gcc 5...). Is there a tradition around enabling "fast" or "skip the tests" builds that I should follow?

[becker33]

Non-boolean variants should be working, they were mixed in with the cflags support.

[adamjstewart]

Non-boolean variants should be working, they were mixed in with the cflags support.

@becker33 Really? I don't see anything in the documentation and I haven't heard anything about non-boolean variants. If they existed, I imagine most of the MPI packages would be rewritten immediately haha.

This is my first crack at a "new" (though based on other work) package file. What does it need in the header/comments.

Did you run spack create http://www.cpan.org/src/5.0/perl-5.22.2.tar.gz? If you did, it would generate a package.py with the appropriate header and comments. Just copy the licensing stuff from any other package.py and you should be good.

I want to install a known version of App::cpanminus alongside the core. I've currently hardcoded the version. I thought I could use a non-boolean variant, but that change looks like it's still waiting to be merged. Do I have other alternatives?

If you want to keep track of the version or be able to change the version, I think your best bet is to add it as a separate package that depends on perl.

Running perl's tests takes a while, but it's the only real way to know if you have a functional build (until I added it, I thought that 5.18.4 was working; turns out that release won't build w/ gcc 5...). Is there a tradition around enabling "fast" or "skip the tests" builds that I should follow?

@davydden added a --run-tests flag to the install subcommand. Until #1186 is complete, the easiest way to make the tests optional would be to use:

if self.run_tests:
    make('test')

[adamjstewart]

W.r.t. extentions: I need to understand Spack's extension mechanism and see if it makes sense for Perl.

The way I understand it, extends('perl') is equivalent to depends_on('perl'), but adds one extra feature. You can run spack activate perl-module and it symlinks that module into the default directory that Perl searches for extensions. See the Python package for more information.

[becker33]

They are working. Apparently I forgot about them when I wrote the documentation. I’ll look at the documentation and find a place for them.

[becker33]

They’re in the documentation. Check the basic_usage guide.

[adamjstewart]

Interesting. I'll open a separate issue for this, but I would be interested in implementing this for MPI, since most of the variants are mutually exclusive.

[hartzell]

@becker33 -- I saw the example in the basic usage guide (first para in the variants section...) but wasn't able to figure out how to fish the value of a variant out of the specification and found a related pull request that still seemed to be in progress. Sorry I mis-represented the state of the world.

I'm an unsophisticated Python programmer (for now...). I did a bit of grep and acking looking for other packages that would show me how to access it, but ....

Given [momentarily putting aside the issue of whether this should be a variant or another package or...] something like this:

variant('cpanm_version', default='1.2.3',
        description='The version of cpanm to install.')

how would I access the value of the variant?

[hartzell]

@adamjstewart -- w.r.t. header info.

I started from JustinToo's work, not something fresh.

Looking at e.g. the R/package.py, I can't use the "Copyright ... Lawrence Livermore ..." bit though, nor the "Created by Todd Gamblin". Do I need to do "contributor" paperwork, or.....

The initial file created by spack edit -f foobar-2000 does not include any header/... info.

[adamjstewart]

Looking at e.g. the R/package.py, I can't use the "Copyright ... Lawrence Livermore ..." bit though, nor the "Created by Todd Gamblin". Do I need to do "contributor" paperwork, or.....

You can use it. Every single package in all of Spack uses it, whether or not it was created by @tgamblin or someone who works at LLNL. Honestly, I don't know why we need this in every file. I feel like the LICENSE file in $SPACK_ROOT is plenty, but maybe @tgamblin can fill me in on the legalities.

The initial file created by spack edit -f foobar-2000 does not include any header/... info.

spack edit -f is broken and will eventually be removed when I get the time (see #1108).

[becker33]

To access the value of a non-boolean variant foo from the install method, use

spec.variants[‘foo’].value

To test whether a variant foo has value bar in a package.py file, use

spec.satisfies(‘foo=bar’)

Hope that helps!

[hartzell]

@adamjstewart -- w.r.t. extensions and Perl.

I'm still fairly confused about how extensions work, I don't think that they're going to work in the Perl world and I'm nearly certain that they're not the right solution to including a copy of cpanm into each Perl (part of my opinionated Perl practices).

Stacking or pooling modules that weren't installed in a coherent manner (the same interpreter, the same stack of library directories) is not reliable.

A number of perl modules involve compiled C code. Modules that have been compiled with one perl can not be counted on to work with other perl binaries. In particular, there is no ABI guarantee across minor versions (things built with 5.20.x are unlikely to run with 5.22.y) but compilers and libraries (and the phase of the moon...) enter the mix too. I assume that spack will only activate an extension for the Python that created it, not a different version. I haven't checked, perhaps it's not a problem for the other languages.

I think that Perl modules/distributions are finer grained than the other dynamic languages' equivalents. You don't just install Moose, you install all of its missing dependencies too. If you then install an isolated tree of some other unrelated distribution (perhaps Rose::DB) it will drag in all of its prerequisites. How do you merge those two trees? What happens if some extension was activated when Moose was installed but then inactivated? Where are Moose's dependencies now? Worse, there may be versioning conflicts between modules in the two deployments.

The Perl language's ability to specify versions for prerequisites is weak (just "unspecified" or "must be at least X"). Some tools (cpanm, dzil) have richer models but they're not ubiquitous. This makes sharing tricky.

Proper perl-based things are installed using perl tools. If I build and test a package with a particular perl, it's safe to assume that it will work with that perl. Installers like cpanm will rewrite the #! lines of scripts that start with #!perl to refer to the perl executable that installed them. This is generally safer than #!/usr/bin/env perl (even when using e.g. Lmod and dependencies). For that to work out though, cpanm needs to be running under the perl executable that will ultimately run the script.

There are also some historical issues and bugs in how the installers figure out whether they need to install a module (or an updated version of one) vis a vis what's in a particular perl's core.

Over the years, I've adopted a fairly uptight approach to supporting complex perl things in complex environments, no coupling.

I install and share perl+cpanm and keep it unsullied. Every project maintains it's own project specific local library with all of its non-core dependencies, accessed via the PERL5LIB environment variable. I have a "department" deployment that includes "all of the common add-ons" (loaded via an environment module) that's used by folks who just want to "run perl". Some teams maintain a common deployment (set up via modules) that all of the team members use when working on that project (or branch of....). And in some settings we maintain individual deployments of sets of modules. Generally, installations just work. Distributions that don't install easily are fixed and copies are cached locally (while waiting on the upstream).

I strongly discourage layering multiple directories onto PERL5LIB; I can't stop people, but they're stuck with the pieces when it breaks. Serious projects also vendor (cache copies of) all of their dependencies' tarballs (via Pinto or other tools) to make installations repeatable. Shared practices and machinery (e.g. dzil) minimize the overhead.

The inconvenience of each project (each developer, often...) installing {its,her,his} own copy of a package is minor, but we revel in not being taken down by someone else's changes.

TL;DR (wow, you made it this far). I'm open to seeing extensions work for Perl but want to be sure to provide access to an unsullied perl core+cpanm.

[hartzell]

@becker33 -- Thanks! (now I've learned something new today!)

[hartzell]

@adamjstewart -- r.e. --run-tests

What's the Spack best practice? Default to running the tests (my world view...) or skip them and trust to luck (I think I've been listening to too much convention coverage...)?

But seriously, my bias is to run the and perhaps give people a way to skip them. It sounds as if Spack tends the other way. Would leaving it as is block getting this included?

[adamjstewart]

@hartzell What you're describing sounds fairly similar to the way Python's extensions work. Let me try to explain how Spack handles Python and clear up any misconceptions.

A number of perl modules involve compiled C code. Modules that have been compiled with one perl can not be counted on to work with other perl binaries. In particular, there is no ABI guarantee across minor versions (things built with 5.20.x are unlikely to run with 5.22.y) but compilers and libraries (and the phase of the moon...) enter the mix too. I assume that spack will only activate an extension for the Python that created it, not a different version. I haven't checked, perhaps it's not a problem for the other languages.

Yep. So if you build a Perl extension with Spack and activate it, it can only be activated to the Perl that it was built with. Does that address your concerns? Unfortunately we don't yet hash the phase of the moon, but I've heard talk of adding that to the spec 😜

I think that Perl modules/distributions are finer grained than the other dynamic languages' equivalents. You don't just install Moose, you install all of its missing dependencies too. If you then install an isolated tree of some other unrelated distribution (perhaps Rose::DB) it will drag in all of its prerequisites. How do you merge those two trees?

If you were to create a perl-moose package that extended Perl, you would add its dependencies to the package.py.

What happens if some extension was activated when Moose was installed but then inactivated? Where are Moose's dependencies now?

Spack does not allow you to deactivate an extension without deactivating the extensions that rely on it. For example, if you activate py-scipy, it automatically activates py-numpy. If you try to deactivate py-numpy, Spack won't let you. You have to deactivate py-scipy first.

Worse, there may be versioning conflicts between modules in the two deployments.

No way around that unfortunately. The same problem could theoretically happen with Python, although I haven't yet run into it. Honestly, this could happen with any software, not just extensions.

The Perl language's ability to specify versions for prerequisites is weak (just "unspecified" or "must be at least X"). Some tools (cpanm, dzil) have richer models but they're not ubiquitous. This makes sharing tricky.

Most packages are like that. If they don't specify a version, I don't constrain it until I find one that breaks. If they do say it "must be at least X", you can add that to the dependency:

extends('perl')
depends_on('perl-moose@1.2.3:', type=nolink)

Proper perl-based things are installed using perl tools.

And proper Python-based things are installed with pip. Or easy_install. Or Spack. Or any of a thousand other package managers. The difference is that you can control what version Spack installs, but that may be more difficult with perl tools that are outside of your control.

I'm open to seeing extensions work for Perl but want to be sure to provide access to an unsullied perl core+cpanm.

We can do that for you. If you don't want to make Perl extendable yourself, then I don't think that should hold up this PR. It's kind of a lot of work to setup the first time (see the monstrous Python package), but once it's working, it's a thing of beauty. And even if someone else implements Perl extensions, there's no reason that you have to use them. You can install Perl, cpanm, and any other perl packages that get added with Spack. If you don't activate those extensions, you'll just have to add them to your PERL5LIB path by hand. Of course, doing it this way doesn't prevent your users from matching incompatible libraries like you said. If you activate the extensions, then the Perl you distribute just comes with the right packages, without any user intervention required.

[adamjstewart]

@adamjstewart -- r.e. --run-tests

What's the Spack best practice? Default to running the tests (my world view...) or skip them and trust to luck (I think I've been listening to too much convention coverage...)?

But seriously, my bias is to run the and perhaps give people a way to skip them. It sounds as if Spack tends the other way. Would leaving it as is block getting this included?

I think Spack's best practice will change a bit when #1186 is merged, but as of right now, there are two accepted practices. Always run the test (I do this a lot if the test is quick and rarely fails) or use --run-tests which defaults to False.

There is no way to make the test optional and default to True as of right now without using a variant. And we've all come to an agreement not to use variants for this purpose because it results in two different installations with the only difference being that tests were performed on one and not the other.

I think once #1186 is merged, all tests will eventually get moved to their own special function and will only be activated when --run-tests is invoked, so it may be better to start using this functionality. Maybe we can someday add a Spack config settings that lets you set --run-tests to default to True.

@davydden and @alalazo may have more opinions on this.

[adamjstewart]

I don't think cpanm should even be included in this package. If it is a separate software package with a version, it deserves it's own package. It can depend on perl.

[hartzell]

@adamjstewart

The Perl language's ability to specify versions for prerequisites is weak (just "unspecified" or "must be at least X"). Some tools (cpanm, dzil) have richer models but they're not ubiquitous. This makes sharing tricky.

Most packages are like that. If they don't specify a version, I don't constrain it until I find one that breaks. If they do say it "must be at least X", you can add that to the dependency:

extends('perl')
depends_on('perl-moose@1.2.3:', type=nolink)

Those aren't language things, they're toolchain things. Perl's tooling has been improving over the last 5 or so years (e.g. cpanm) and gaining that type of control. E.g. carton (aka Ruby's Bundler for Perl) and it's carton.lock file.

The language itself though, can only say things like this:

use Moose 2.1804

which constrains it to use version '2.1804' or newer. That might be '3.001', which includes a completely different API.

This means that tools like dzil can't e.g. walk the source tree and produce a correctly constrained list of prerequisites. It's really nice to not have to maintain your list of dependencies by hand.

A common solution seems to be to just produce a known good set of prereqs and specify them exactly.

I gather (citation needed) that there are other languages that give you more control at the "import statement" level. Sadly, I don't work with them....

[hartzell]

@adamjstewart

I'm open to seeing extensions work for Perl but want to be sure to provide access to an unsullied perl core+cpanm.

We can do that for you. If you don't want to make Perl extendable yourself, then I don't think that should hold up this PR.

I'm up for digging into it too just want to move in small steps.

[hartzell]

For what it's worth (a bit of data to go with all of the words):

• installing Pinto requires 89 distributions beyond the Perl 5.22.1 core set; and
• installing Dist::Zilla::PluginBundle::HARTZELL (my standard dzil configuration) requires 221 distributions beyond the Perl 5.22.1 core set.

After years of apologizing for Perl to people who have tried to wire up a hairball, and being told that other languages (usually Python, these days often node.js) don't have these problems, this quote in the EasyBuild docs made me smile:

Installing any Python package can be a real pain, and since EasyBuild is basically a set of Python packages glued together, installing EasyBuild may (ironically) cause some headaches.

http://easybuild.readthedocs.io/en/latest/Installation.html

The left-pad incident shook a lot of people too.

In the end, it's a hard problem and I am so thankful for tools like EasyBuild, Spack and {home,linux}brew (and their creators) for struggling with it

[hartzell]

@adamjstewart

Thanks for this:

I think Spack's best practice will change a bit when #1186 is merged, but as of right now, there are two accepted practices. Always run the test (I do this a lot if the test is quick and rarely fails) or use --run-tests which defaults to False.

Since tests can take a while, I'll use run-tests.

@everyone --

Is there a way to document something like this for a package. Best idea that I've come up with is to add prose to the package's description so that it shows up in spack info package_name. That feels wrong though.

[adamjstewart]

Nah, I wouldn't worry about documenting it in the package. It's not like it only applies to that package, it applies to all Spack packages that feature tests. After #1186 is merged, all Autotools packages will have test suites via make check (whether or not it is in the package.py). Since --run-tests is in the Spack documentation, you're good to go.

[hartzell]

I don't think cpanm should even be included in this package. If it is a separate software package with a version, it deserves it's own package. It can depend on perl.

I disagree, as do e.g. the PerlBrew [the most commonly used perl installer] team:

In short, patchperl and cpanm are so important, it is best to make them always available.

https://perlbrew.pl/Perlbrew-and-Friends.html.

(patchperl's not relevant if you're not supporting ancient Perls). This is not an uncommon technique for enabling loosely coupled perl installs.

For what it's worth, there is already an older CPAN tool in the core, cpan, it's just complicated and much harder to use correctly than cpanm.

In my years of supporting Perl I've found life is way simpler if I include it alongside the core (and I say that as someone who has come to loathe coupled things).

When I install it as a separate thing, people neglect to load it and use the cpan tool instead, or find a way to run it with a different perl. If you bring it into their environment by adding its directory onto PATH and PERL5LIB then you've introduced the "multiple directories on PERL5LIB" complexity and it stops working when they push something other directory before it (or some other perl thing stops working when someone doesn't load it). The Spack extension mechanism offers an alternative that I have yet to explore (and gain real-life experience with).

There are other perl-based "stand-alone" tools, e.g. App::Ack and Pinto that can nicely stand on their own (they work hard to divorce themselves from the external world) but cpanm is so intertwingled that I've come to make an exception for it.

Would a compromise be to leave it an option but have the variant default to False?

[hartzell]

ps. Thank you all for the discussion!

[adamjstewart]

• installing Pinto requires 89 distributions beyond the Perl 5.22.1 core set; and
• installing Dist::Zilla::PluginBundle::HARTZELL (my standard dzil configuration) requires 221 distributions beyond the Perl 5.22.1 core set.

Oh. My. God. That sounds awful. If this is a common thing, then it might be best not to make Perl extendable. Otherwise, you would have to add all 221 dependencies just to add that package. Probably best to take advantage of cpanm in that case. No sense in keeping track of our own set of dependencies and version constraints.

[hartzell]

Oh. My. God. That sounds awful [...]

Perl, the language I hate to love...

[hartzell]

I think that this is ready to be considered for merging.

• I've added the standard header (copied from antlr/package.py).
• I've convinced people (correct me if I overstep here) that the way I'm handing installing cpanm alongside the core is a perl-ish and reasonable practice. We can explore extensions later.
• I've made running perl's tests conditional, following the Spack --run-tests practice.

Once this is merged I'll update my git PR (#1345), which depends on it.

[hartzell]

Rebased to the tip of develop

[hartzell]

Why, oh why, is the "close and comment" button bigger that the "comment" button.

I did not mean to close that.....

[alalazo]

@hartzell Eheh, anyhow close and reopen is a useful trick to trigger a Travis rebuild if you don't have write permission upstream...

[adamjstewart]

I always just rebase and force push to trigger a Travis rebuild, but that works too. Looks much better now, thanks!

[becker33]

Now that you have the cpanm installer as a resource, please delete the hard-included installer from the PR.

[hartzell]

@becker33 -- done, thank you!

[becker33]

There are also a few flake8 errors in the travis output now, please fix those.

[adamjstewart]

Did you accidentally add the cabal stuff?

[hartzell]

Did you accidentally add the cabal stuff?

Yep. I'm going to sit down and have a talk with my intra-cranial QA personality. He's been slacking off this AM. Sorry for making you folks clean up after me.

[hartzell]

@becker33 -- flake 8 is clean for me. It might have been tripping up on the mistakenly committed cabal bits.

[throgg@spack-central spack]$ ./share/spack/qa/run-flake8
=======================================================
flake8: running flake8 code checks on spack.

Modified files:

  var/spack/repos/builtin/packages/perl/package.py
=======================================================
Flake8 checks were clean.