Squeeze CI running times

[astrojuanlu]

We optimized CI running times (#395), and we went from > 45 minutes (with timeouts) to < 30 minutes:

Now, we have to squeeze them 😉 Ideas:

• Circle CI allows for more jobs at the same time than what we're using now, so we could split the slow tests apart in a separate job (for instance using Python 3.6, or even two Python versions). I estimate a reduction of factor 2x on the test jobs (so, from ~15 to ~7 minutes).
• Current Circle CI executors have 2 CPUs, so we could use parallelization at the tests level, running two tests at a time. This should bring down the times another factor of 2x (so, from ~7 to ~4 minutes).
• We should consider running the coverage job in parallel with the tests, because it's probably going to be the slowest one.
• An alternative would be to split the coverage measurement and then combine the results as described here https://labs.ebury.rocks/2017/02/23/unit-test-execution-part-2-coverage/
• When we make enough progress, we should also cache the installation, using some sort of lock files, as described here https://github.com/CircleCI-Public/circleci-demo-python-django/blob/master/.circleci/config.yml

So, we should be able to bring the current times from ~30 to ~10 minutes, or at least ~20.

[astrojuanlu]

#404 brought this to 18 minutes. Next step: 10!

[astrojuanlu]

Caching would help in situations where the docs (the last step) fail.

[astrojuanlu]

We should also mark some tests as slow, integration, or something. The conda-forge package currently runs everything, which is way too much.

[astrojuanlu]

Also, it seems that there's a way to produce proper coverage reports using path aliasing in one run:

pytest-dev/pytest-cov#176

However, we have to think about this carefully, since slow tests would be even slower.

[astrojuanlu]

Another thing I should do is to cache the docs dependencies or use another image. Installing texlive takes a ridiculous amount of time.

[astrojuanlu]

We completely changed the CI and now we are at around ~20-25 minutes. Definitely good enough, so closing this!