Test with Travis environments

[jayvdb]

This change is 

[joshfriend]

Wow this makes the time to run tests unbearably long 😉

[jayvdb]

We can remove some of the failures by fixing the failures.
i.e. some of the failures are ilustrative of problems and failed attempts to workaround the problem.
Elapsed time is quite short actually; some projects have several hours elapsed.
If time is a problem, I can set it up to do one successful test job, and do a random config in that job.

[jayvdb]

The need for LD_LIBRARY_PATH here seems like it could be fixed by pyenv, but also fixed upstream...?

[jayvdb]

Maybe pypa/virtualenv#899
And probably fixed by pypa/virtualenv#923

[jayvdb]

Something I havent tried yet is whether the latest version of virtualenv addresses any of these failures.

[jayvdb]

Ironpython install a binary python which needs to be run by mono, or use this trick on a sudo enable worker, which is slower than the precise containers. Otherwise virtualenv tries to run it, and fails.
Is it possible to detect this and install a script python which invokes mono?

[jayvdb]

Elapsed is now 11 mins, with more test environments included.

The previous problem with pypy-dev has been fixed, by installing 2.7.11 first. Now it dies during the build, and has been excluded.

Note that it is not possible to include virtualenv from apt source debian-sid, as pulls in many other Debian sid packages which conflict with the Ubuntu packages.

I tested using a the latest virtualenv, and some of the failures are fixed by it, namely 3.4.4 and stackless on language: generic.
I have a slight preference to keep them as failures until we know why they are failing on the older virtualenv, but I can integrate the virtualenv upgrade code if we want more successes before this is merged.

[yyuu]

@jayvdb just a silly question, but what do you want to test by this? How do you think what is insufficient with stub based testing?

[jayvdb]

Well, integration testing finds different types of problems, especially ones that havent been seen before.

Obviously the -dev series of versions will benefit the most from integration tests, as the contents of those branches are unpredictable. And as seen by #618, integration tests will find these problems quickly when upstream changes.

There is also the benefit that defining integration tests highlights minimum supported versions of other programs that are used together with pyenv, such as virtualenv. Currently it is obvious that the virtualenv with Ubuntu precise (v12.0.6) and trusty (v13.1.0) is not supported by all pyenv install-able versions. There is no hint of that in the README ;-)

Ideally we can install minimum versions of each dependency and run pyenv install x for every install-able version, and while it may take a while it should install them all, and basic operations should be possible with each of them. When they all can be installed and used, and there are unittests for the known problems found, this integration testing could done by a weekly build (not yet possible with Travis) instead of with each commit, as it is only looking for unexpected breakages.

Until then, I believe it is useful to show the known good environments and the known broken environments, so we can focus on fixing these problems, and/or raising them upstream. What is the point of shipping these install-able versions if we cant get them to work ourselves?

Whether or not you merge, I'm going to keep working on fixing these broken versions, and this set of integration tests will get quicker as more versions have been fixed and dependencies reduced.

[jayvdb]

Testing only on precise without any language: python.
Elapsed down to 7.5 mins.

[jayvdb]

this is also fixed with VIRTUAL_ENV_ADD_LIB_PATH=1
now just need to get ironpython working... ;-)

[yyuu]

This could help preventing the issues like #638.

[native-api]

Superceded by 8ed7912