Membership, storage and reminder providers as external extensions

[ashkan-saeedi-mazdeh]

I've realized the need to use a different storage provider for my application.
I'm writing a complete storage provider, IMembershipTable and IReminderTable for CouchBase KVS.
http://www.couchbase.com/

Should we in general consider moving these to Orleans like Azure, SQL and other cloud providers or keep them outside and just list them in contrib?

Also if in contributions, then should we include them in a visible part of documentation or not? We should really try the initiative of running Orleans everywhere for EVERYONE. As an unfortunate example in CouchBase's manual they mentioned Akka.NET in their async programming with .NET page but not Orleans and we should change this. I got offended :D

What should be the approach @sergeybykov @jdom @ReubenBond @galvesribeiro Add others as you see fit.

[ashkan-saeedi-mazdeh]

Understood from @jdom 's comment on gitter when talking to @galvesribeiro that the policy is to go to contributions.

[jdom]

Not necessarily. OrleansContrib can be an enabler to reach maturity, if it helps, but it is not the only option. Please read the thread at #2005 for clarification on @galvesribeiro's conclusion.

I do believe that regardless of whether it's in the main repo or Orleans contrib, we should be doing a better effort to decouple each provider from the main solution. We should have some sort of test kit (as @veikkoeeva suggested) that we can run our providers against, to guarantee at least a minimal set of behavior from them. Of course each provider can then have additional provider specific tests.
Providers in the main repo should not have any advantage (in terms of how in-the-box they are, or how easy it is to test them) compared to new experimental/external providers.
Also it should be good if they can be versioned and released independently of Orleans core nugets.

[veikkoeeva]

@jdom The test kit idea could be realizable today with little work, at least for storage providers. We could create public data sets such as example. Then create a core set of tests (copy-paste-rename ((this particular one inhertics for base ADO.NET one that isn't needed in general, I could fit ATS to that set too as for an example at some point if desired). The core tests intended to be stateless and test framework are these that the aforementioned MySQL one sugars over with test sets and some ADO.NET/MySQL specific things.

This probably needs rearrangement in that regard. Also things like this could generate parameters using just a string template.

<edit: To add the idea, it could be roughly described also so that:

1. Provide the substem to under test with simle new or a factory method and just call the public interface defined functions and observe the output and side-effects that the contract is upheld.
2. Set up environment in some fashion that can be parameterized from outside runner (here a JSON settings file in a well known path or pointed by a well known environment variable).
3. Looking at years ahead, not being dependent on a testing framework by building a testing framework that integrates tightly with some specific blend of a testing framework.
4. Make it run in parallel with arbitrary granularity.

As a corollary to 2., the tests could be run by using also XUnit.NET console runner or the MSBuild system. I would argue it should be possible for test kit and defining the tests shouldn't require special inheritance, but just feeding in data to the functions. Here the data is specifically arranged to generated data sets in a fashion that functions well with XUnit.NET, but it's not tied to it. One could also use IntelliTest, FsCheck or whatnot.

To some degree it has been established in that sketch (should be improved, not the only approach and this is naturally just my opinion).

[galvesribeiro]

@veikkoeeva if you can make providers (stream, storage, reminder and membership at least) fully testable outside Orleans code, maybe issues like #2005 could be implemented correctly without the need to depend on the core. Let me know if you need any help as I don't want that things to happen again.

[ashkan-saeedi-mazdeh]

@galvesribeiro exactly. as I pointed out in #2005 , the issues were mixed up and mostly this all happened because of lack of a technical capability (usability of tests for outside Orleans source), most of the times these issues happen due to management issues but this time was at least partially due to a technical one.

[shayhatsor]

The membership, reminders and statistics tests would also be easy moved into the test kit, it's basically copy pasting the test base classes (I do think that the liveness tests would need a bit of refactoring). How would you guys think this test kit would be packaged? as a nuget package, something like Microsoft.Orleans.Providers.Tests ?

[jdom]

That sounds great. My preference in naming would be Microsoft.Orleans.Providers.TestKit, but it's fine otherwise

[veikkoeeva]

@galvesribeiro

if you can make providers (stream, storage, reminder and membership at least) fully testable outside Orleans code, maybe issues like #2005 could be implemented correctly without the need to depend on the core. Let me know if you need any help as I don't want that things to happen again.

I added to my short note on how I envisoned it, the code supports it to some degree, I can point specifics in chat if someone asks or I'm around. I think the ADO.NET provider wen't largely to that direction (not fully, but closer) and I don't see why couldn't the other prociders too. It looks to me we should clean up the tests a bit. For instance, arrange more test arguments to test data generators and define a core set of tests that need to be satisfied and then concrete provider implementations could use these core sets and test data sets.

It's to some degree unrealised (looks like not much, I just go into that much specifics as it was inteded more as a spike) and perhaps a bit opinionated in that I would just like feed data in to functions and observe return values and side-effects. To arrange the "sandbox", the environment, could be realized too perhaps rather uniformly (how the ADO.NET provider gets the default arguments or gets them from outside and then sets up the environment, the factory methods perhaps could use inheritance, or then not).

My apologies for being sluggish here (happens in IRL and all that jazz).

[sergeybykov]

This has been addressed in 2.0.