Tag elixir tests into meaningful groups

[iilyak]

Elixir has ability to tag test cases. Which means we can split tests into categories and run them in the sensible order. We would need to do some leg work to make it happen, but it is doable. We want to fail fast so we would run the group of tests in the following order:

• single_node
• degraded_cluster
• cluster
• performance
• property_based

[wohali]

poke @van-mronov @jjrodrig does this issue interest either of you?

[van-mronov]

@wohali @iilyak sure, I'd gladly deal with this task.

[van-mronov]

@wohali @iilyak so, do we already know which tests are single_node, etc?

[iilyak]

I don't know from the top of my head but it is easy to figure out.

1. one of the signs for tags is if we start tests differently https://github.com/apache/couchdb/blob/master/Makefile#L211
2. tests which require single node setup fail when dev/lib is present

• run dev cluster for few seconds (we just need it let to join) dev/run --admin=adm:pass
• stop the cluster CTRL+C
• modify Makefile so devclean is not called for elixir tests
  • add @echo on https://github.com/apache/couchdb/blob/master/Makefile#L473 line
• run elixir tests and check which ones fail
  • make elixir
  • make elixir-cluster-without-quorum
  • make elixir-cluster-with-quorum

[wohali]

I'm a little nervous about this proposal just pushing tests into a flaky category and then never being examined.

Who is volunteering to clean up these tests and ensure we don't ignore them forever?

[wohali]

@jjrodrig @dottorblaster Either of you still interested in taking this on?

[dottorblaster]

@wohali absolutely!

[nickva]

This was implemented in #3110

However we ended up using hard-coded suites instead it seems.