generated .travis.yml downgrades dependencies installed by other dependencies

[Lx]

(Travis CI build log for reference.)

In my case, dzil chainsmoke --mvdt is adding an install instruction for:

cpanm --verbose PHAYLON/MooseX-Types-0.01.tar.gz

All well and good so far, since my code depends on that module but doesn't specify a minimum version for it. All is working as advertised.

However, other modules before this one depend on later versions of this module, and do list a minimum version:

• https://travis-ci.org/Lx/perl5-Finance-Bank-Bankwest/jobs/10922187#L885
• https://travis-ci.org/Lx/perl5-Finance-Bank-Bankwest/jobs/10922187#L902
• https://travis-ci.org/Lx/perl5-Finance-Bank-Bankwest/jobs/10922187#L991
• https://travis-ci.org/Lx/perl5-Finance-Bank-Bankwest/jobs/10922187#L1028
• https://travis-ci.org/Lx/perl5-Finance-Bank-Bankwest/jobs/10922187#L1724
• https://travis-ci.org/Lx/perl5-Finance-Bank-Bankwest/jobs/10922187#L1850

Still all well and good, since the necessary minimum version of that module is installed. Until this line, that is:

cpanm --verbose PHAYLON/MooseX-Types-0.01.tar.gz

when cpanm actually downgrades whatever the installed version is back to 0.01.

As a result, many unrelated explosions occur and the build errors out.

I'm guessing that the line calling cpanm has to be modified such that it also has a --skip-satisfied option passed in, which would prevent downgrading. (Mind you, I'm not sure if this option would actually work when explicit filenames are passed the way they're being passed here.)

[Lx]

Looks like --skip-satisfied isn't the solution, since it still goes ahead and installs the specified version anyway.

I guess in my case, I could overcome the problem by looking for the very first error mentioning an unmet minimum version dependency in another dependency and hard-code that minimum version in my own code. This doesn't seem ideal though.

[Lx]

No, this really isn't ideal.

For each of my dependencies, I'd need to determine whether any of my other dependencies depend on that initial dependency and if so, determine the highest minimum version of each one from that. Then I'd need to repeat this every time I bump another dependency (which would even happen multiple times while determining that initial list).

[SineSwiper]

Interesting problem. It seems like ::TravisCI would need to deep dive into the dependency tree, see which other dependencies exist, and start at that minimum version first.

Unfortunately, at this point in time, MVDT is a very manual process. So, even if that issue was solved, it's still requires incremental upgrades of each version (read: guesses) and re-tests until you get a full MVDT dep map that works.

I have plans to revamp the model, using the Travis API and a local SQLite DB (or just a CSV) to keep track of test failures, and make this whole process automated. But, that's likely going to be months away, since I have a few other things on my plate.

[Lx]

I'm not too offended by having to do the version incrementing myself; it gives me the opportunity to look at exactly why failures occur with the given version set and this can potentially highlight other problems that need fixing in my own code. Still, it'd be a nice convenience later down the track.

Right now though, my locally-installed version of cpanm allows me to do this:

$ cpanm MooseX::Types~0.01
You have MooseX::Types (0.35)

i.e. a later version of that module required by a different dependency isn't overwritten with an older version.

Provided that Travis' cpanm offers the same thing, I wonder if this is an acceptable short-term fix? It'd at least get me going again on my own MVDT adventures...

[SineSwiper]

The problem with that is initially you actually want the downgrade. The stock TravisCI VM might have a version of that module above the one you want installed.

Though, I might be splitting hairs at this point. This might be a fair compromise. But, I'd like to tackle the full problem at some point.

I'll see about pushing something to the repo in the next few days.

[SineSwiper]

The latest repo has a compromise that may work better. It runs through two passes on the module installs. The first pass is the standard module install. The second pass is one with --installdeps to only install dependencies that might have been overridden by other installs. There's also some fixes for distro de-duplication, especially between phases.

Try it out and see if that works better.

[Lx]

Thanks for your efforts on this. I'd like to try out your changes but I'm not sure how to make them available on my system. Are you willing to push a new release of this module to CPAN?

[Lx]

Actually, don't worry about that--I've managed to get the repo code working:

$ cpanm --uninstall Dist::Zilla::App::Command::chainsmoke
$ PERL5LIB=$HOME/code/Dist-Zilla-TravisCI/lib dzil chainsmoke --mvdt

I was falling over dzil's --lib-inc option, which seems to only be honoured after the Dist::Zilla::App::Command namespace is loaded. Not helpful when you're trying to use --lib-inc to affect the available dzil commands. Unsure whether that should be regarded as a dzil bug or not.

Anyway, the changes so far seem to be working well for me. I'll report back if I run into any problems.

[Lx]

I can see one potential problem with your approach (although I don't have an opinion at this stage on its likelihood, or any potential workarounds).

My code says I need MooseX::Types v0.01 (among other things), so the following .travis.yml lines will be generated:

...
install:
    ...
    - cpanm --verbose PHAYLON/MooseX-Types-0.01.tar.gz
    - cpanm --verbose ...
    - cpanm --verbose ...
    - cpanm --verbose ...
    - cpanm --verbose --installdeps PHAYLON/MooseX-Types-0.01.tar.gz
    - cpanm --verbose ...
    - cpanm --verbose ...
    - cpanm --verbose ...
...

So there are two "passes," where the first installs the base versions of each needed module (downgrading if necessary), and the second ensures that all minimum dependencies for each of those modules are still met afterwards.

If one of the other modules later in the first pass require MooseX::Types v0.02, the newer version of that module will be installed. This is the correct, desired behaviour.

But in the second pass, it seems that the dependencies for MooseX::Types v0.01 (the old version) would still be checked and installed rather than those for the newer version.

I'm guessing that this would be a problem if v0.02 introduced dependencies not needed by v0.01, but which happened to be downgraded by something else in the first pass. The downgrades would not be corrected in the second pass because the correct required version is not being looked at. Does that make sense? Is this scenario even possible?

[Lx]

I suppose a really hacky workaround might be to replace the second pass with lines like these:

- cpanm --verbose --installdeps MooseX::Types~`pmvers MooseX::Types`
...

which, if I understand the cpanm docs correctly, would cause the dependencies of the currently installed version of MooseX::Types to be checked and installed again as necessary (assuming that backticks can even be used this way, and assuming the availability of the pmvers tool off CPAN).

[SineSwiper]

Seems like an edge case. You might have different deps installed that you don't actually need, but --installdeps shouldn't actually downgrade anything.

[SineSwiper]

This has now been released in v1.05. If there are still some outstanding issues, those can be covered in another ticket.