Extend e2e testing

[mrIncompetent]

We should add the following test cases:

• Hetzner
  • Ubuntu + Docker 1.13
  • Ubuntu + Docker 17.03
  • Ubuntu + CRI-O 1.9
• Digitalocean
  • Ubuntu + Docker 1.13
  • Ubuntu + Docker 17.03
  • Ubuntu + CRI-O 1.9
  • CoreOS + Docker 1.13
  • CoreOS + Docker 17.03
• AWS
  • Ubuntu + Docker 1.13
  • Ubuntu + Docker 17.03
  • Ubuntu + CRI-O 1.9
  • CoreOS + Docker 1.13
  • CoreOS + Docker 17.03
• Openstack (We need a sponsor here)
  • Ubuntu + Docker 1.13
  • Ubuntu + Docker 17.03
  • Ubuntu + CRI-O 1.9
  • CoreOS + Docker 1.13
  • CoreOS + Docker 17.03

[alvaroaleman]

To keep tests still efficient and also not repeat ourselves too much I suggest putting this in the verify tool by:

• Adding a config file that allows to specify a matrix just like the one above
• Add a --create-only flag that only creates the machine objects so they will be created in parallel
• Add a --wait-for=<provider>-<distro>-<container-runtime> flag that blocks until the given machine was successfully created and then deletes it to see at a glance in CircleCI what failed if something failed

[mrIncompetent]

Why not implementing decent concurrency into the command?
Might be cleaner than just juggling with different command line calls

[alvaroaleman]

Because then you can't have a dedicated step per scenario in CircleCI which helps a lot if tests failed and you want to find out which one

[mrIncompetent]

We could also do the setup (terraform & cluster creation) in one step, and then run all tests in parallel.
Then we don't have to modify the command at all.

[p0lyn0mial]

@alvaroaleman @mrIncompetent Could we slightly change approach to writing test cases ?

I'd like to start writing test cases as standard golang tests.
But at the same I'd like to utilise what we have so far, that is: cluster creation (kubeadm) and the verify tool. The whole process would do the following steps:

1. create cluster (provision_master.sh)
2. run machine-controller (provision_master.sh)
3. run e2e test (using standard go test command)

Individual test cases would hold only data/parameters needed to feed the verify tool. We could also utilise go test's build-in concurrency to parallelise tests and various flags like --short to control how many test we would like to run on each build.

We would also need to change the verify tool to check ownerRef instead of counting the number of nodes.

[mrIncompetent]

Good idea!

[alvaroaleman]

Yes, sounds like a very good idea.