Support different levels of testing/initialization

[ebolyen]

It would be good to allow registrars of this plugin to choose how many test-cases are desired. This would allow quick sanity checking and iteration without spending time exhaustively running each permutation.

This could be accomplished using reservoir sampling to decide when to use a given iteration of the product.

[Oddant1]

That sounds reasonable. Will we be doing that during plugin registration so we only register a certain subset of the permutations? Because we just run all of the registered permutations by calling execute_examples.

[ebolyen]

Yep! So I imagine this plugin will generally be combined with this API which means it doesn't need to register itself, rather we can provide some factories which generate a plugin object of some degree of sophistication.

This also let's interfaces have q2-mystery-stew installed for test-runs, without having it clog up the standard interface (since q2-mystery-stew won't automatically register itself). The test case in this repo can be augmented to initialize a plugin manager and then add itself just like an interface might do.

[Oddant1]

We have this