Make the CI great again!

[emilevauge]

I can remember the good old times when all our tests were passing in less than 10 minutes on Travis.
Now it's more like that:

gopkg.in is flaking again. Retriggered the build.
...
Grrr, and now the integration tests are flaking. :(
Retrying HARDER...

ALL THE TIME!

This global issue is here to list all the work we need to do in order to make the CI great again.

• replace fixed timeouts re: docker compose readiness with utils.Try Replace fixed readiness timeouts in docker-compose fixture setups by retry-based readiness checks #1267 by @timoreimann
• test Travis alternative like CircleCI or SemaphoreCI
• Improving Parallelism of tests.
• Reducing the setup time of a testing environment
• Simplify the process for a contributer to add a parallel test such as integration

[dtomcej]

We should also work on:

• Improving Parallelism of tests.
• Reducing the setup time of a testing environment
• Simplify the process for a contributer to add a parallel test such as integration

[vdemeester]

Simplify the process for a contributer to add a parallel test such as integration

It's more like : simplify the process for a contributor to add tests, mainly unit-test (we should have way more unit test, and very good coverage in unit test, than integration tests)

[atbore-phx]

Today I have performed a CI platform switch to circle-ci v.2 (cci2), forking the main traefik repo and the library too.
I've also translated the .travis.yml in the new .circleci/config.yml here https://github.com/atbore-phx/traefik.
The job created use the "machine" executor, the new job seems slower than the travis one, see the picture and the link attached. (vm/hw used during the test ?)
https://circleci.com/gh/atbore-phx/traefik/12

those features could help to reduce the execution:

• cache for the docker images
• test parallelization (we need to evaluate the impact on the actual test suite)
  but the slower part is the cross compilation.

Next days I'll test SemaphoreCI.

[vdemeester]

The main problem is cross compilation. It is taking more than all other steps.. We should run it parallel/separate.

[timoreimann]

One idea is to only test a single architecture per operating system on each PR (e.g., Linux, Windows, and Darwin each on amd64) and postpone the other architectures to a less frequently running job. the latter could be the master branch, a daily cron, or a pre-release step.

[atbore-phx]

today i've done some tests related to "make crossbinary".
i have deployed several droplets (ubuntu 14 x64 with build-essential) on digital ocean changing the CPU/RAM ratio and just a try with semaphoreCI.

here the results:

VM/task	make crossbinary
VM (digital ocean) 2 CPU 2GB RAM	27m10.272s
VM (digital ocean) 4 CPU 8GB RAM	18m3.101s
VM (digital ocean) 8 CPU 16G RAM	16m40.817s
VM (digital ocean) 12 CPU 32GB RAM	16m59.277s
VM (digital ocean) 16 CPU 48 GB RAM	18m6.143s
semaphoreCI	9m19 (~6m with docker cache and #1535)
Travis	19m
CircleCI	35m44s

semaphoreCI seems to have good performances, here its build :
https://semaphoreci.com/atbore/traefik/branches/master/builds/2

[atbore-phx]

i have introduced some parallelism at make level (make -j2) reducing the execution time of make crossbinary of ~28% respect latest semaphore test.
the exec time is now ~7min always on semaphore.
https://semaphoreci.com/atbore/traefik/branches/crossbinary-parallel/builds/2

[atbore-phx]

tested docker cache on semaphore https://semaphoreci.com/docs/docker.html, but i got "no space left on device" during the snapshoot.
https://semaphoreci.com/atbore/traefik/branches/master/builds/7
i have opened a ticket to semaphore support.

[atbore-phx]

docker cache is now working, i think we reach the limit.
make -j3 + docker-cache ~6 min
https://semaphoreci.com/atbore/traefik/branches/triple-parallel-crossbinary/builds/3

[atbore-phx]

SemaphoreCI has been added as integration to our repo.

the cache has been disabled:

• not a big gain for us: ~40/60 secs
• some sporadic failures during the snapshoot.

[atbore-phx]

I have started to reduce the docker images size with this PR: #1580

[emilevauge]

I think we can close this one now (tests are built in 15m instead of 50m) 🤗