travis: Automatically invalidate caches if new package versions exist

[petertseng]

While working on #101 I discovered that because we cache .cabal, we'll get conflicts (and thus build failures) if there's a new version of a cabal package we use.

That's less than ideal (and we had to manually clear the cache to fix the build), but I believe there's a way out. Should examine the work done in https://github.com/hvr/multi-ghc-travis/blob/master/make_travis_yml.hs - it seems to check the install plan and compares it to the last known one. If there's a change, it clears the packages and asks cabal to reinstall. I think this will work, though it may need some testing... and depending on how often our used packages release new versions opportunities to test may be rare.

[rbasso]

Now that we decided to keep testing with cabal - to support ghc-7.6.3 - and everything else is solved, we need to address this problem.

I think that - because we cache the state of globally installed cabal packages - this will, by design, affect subsequent builds. Of course, we can try to identify when we need to invalidate the cache, but that will increase complexity and build times when triggered.

How do you think we should tackle this issue?

[petertseng]

Also a thing to note is that the symptom of the problem may change a little bit. Now that _test/bootstrap only asks to cabal install the not-yet-installed packages, if an already-installed package gets a new version, we'll just silently not install it. And if the new version happens to break us, well we won't find out until we clear our cache.

This is actually the opposite situation as the one I found myself in a while ago. In that situation, we were broken unless we clear the cache. In our current situation, we may find that we are broken only if we clear the cache.

Nevertheless, both situations are undesirable. But can I stop to ask: What is the chance that we will break? In the current situation, if we clear the cache and _test/bootstrap asks cabal install to install the full list of our exercises' dependencies, would cabal install really pick a set of versions that would cause breakage? What would cause this to happen?

You see that now I need some help thinking about this since the nature of the issue has changed since the time when I filed it.

[rbasso]

It believe it will not break on newer versions, but it will break on new dependencies.
A new dependency will not necessarily be compatible with all the versions of installed packages and will not install, so we live in cabal hell. We'll have to clear all the cache to allow it to build.

This is really undesirable, because we are joint-testing the repository and the history's cache. Failing will not tell us if there is really something wrong with the repository of if we just met an unfortunate install order.

From what I understand, this problem is intrinsic to installing packages globally.

[rbasso]

Maybe we could try to avoid this problem allowing cabal to reinstall packages, but I don't know if it would work.

With this issue, we are trying to solve problems already addressed by a newer version of cabal sandboxes and stack, which was designed specifically to allow reproducible builds.

If we really want to keep testing with global packages, maybe we should forget about caching it. It feels like we're trying really hard to use a tool in something for which it was not designed.

Edit: Or, we can just leave it the way it is. We can clear the cache manually when it breaks.

[petertseng]

I see, you are right. If we add a new dependency we could run into problems.

This issue has changed a lot from when I originally filed it. When I filed it I had a clear solution plan in mind (https://github.com/hvr/multi-ghc-travis/blob/master/make_travis_yml.hs), but now that we changed how we bootstrap, the problem and thus the solution have changed.

Then, I currently don't understand the problem well enough to be able to take any action on this issue. If, for example, we can find a potential dependency that we might add that would break our build given our cache, then we'd be able to discuss what to do. With clear reproduction steps, only then can we test whether a solution has solved the problem.

I may spend some time trying to find such a dependency, but if I cannot find, I can't really consider this issue actionable, so I may be tempted to close it. Or at least unsubcribe.

[petertseng]

Where might I start looking to look for potential candidates that could break though? Clearly I can't pick packages at random, but must be something principled

[rbasso]

I'm stunned that it appear that with a single cabal installs it worked but with individual cabal installs we apparently fail on old-locale

I guess you met this problem when you tried to install packages individually with cabal install.

I may spend some time trying to find such a dependency, but if I cannot find, I can't really consider this issue actionable, so I may be tempted to close it. Or at least unsubcribe.

This issue is related to a design choice we made. I don't think we can do anything about it without making major changes, so I would mark it as wontfix and close it.

If we decide in the future to drop support for ghc-7.6.3, we will not need cabal install anymore and the problem will be automatically solved. Until then, if someone adds a new dependency and it breaks, I'll clear the cache manually and rerun the job.

We already have an overly-complex Travis, and it's probably less work to manually solve those problems than trying to write and debug an even more complex solution. Of course, that will be annoying for contributors that will have failing tests without understanding why.

[petertseng]

I'm stunned that it appear that with a single cabal installs it worked but with individual cabal installs we apparently fail on old-locale

I guess you met this problem when you tried to install packages individually with cabal install.

Hmm, now that bootstrap will install all packages though, it's less clear how we will see the problem now, but now you give me an idea as to how to reproduce the problem, with old-locale as the hint - thanks for this reminder.

We are agreed that we'll close for now. If we run into problems in the future the experience gained can be described there and we'll be better able to come up with a solution when we have a clear problem