Amazon AWS integration

[galvesribeiro]

This issue is part of #2005 by implementing a minimal set of providers to make Orleans to run well by integrating Amazon AWS services.

• DynamoDB Grain State persistence provider (AWS DynamoDB Grain State Provider #2007)
• DynamoDB Membership Provider (AWS DynamoDB Membership Provider #2008)
• DynamoDB Reminder Provider (AWS DynamoDB Reminders #2045)
• SQS Stream Provider (AWS SQS Stream Provider #2053)
• DynamoDB Metrics and Statistics
• Kinesis Stream Provider

[gabikliot]

Why do you think DynamoDB is the right storage AWS layer to implement Membership and Grain State provider? Why not SimpleDB? Or is it not popular any more?

[gabikliot]

A couple of other questions:

1. What about the hosting model? Would you run a silo in a vanilla VM, started as a startup service or by some cron job?
2. Windows on AWS initially and then maybe Linux with Core CLR? In both cases, the hosting model can be identical for AWS and GCP.
3. What about telemetry/logging/monitoring?

[galvesribeiro]

Why do you think DynamoDB is the right storage AWS layer to implement Membership and Grain State provider? Why not SimpleDB? Or is it not popular any more?

We can always add SimpleDB provider to the list as well and implement it. SimpleDB is Amazon offer that compete with Azure Table Storage yes however, I've see people using DynamoDB more for NoSQL/DocDB today. It was not a preference choice. I just made it to help someone else but yes, I can help add that as well if you have interest in it.

1. What about the hosting model? Would you run a silo in a vanilla VM, started as a startup service or by some cron job?

Initially I'm going to lay down the foundation of a very basic providers. Mostly port what is on Azure Table Store providers to use DynamoDB but yes, run on VMs. After that, will look at extensibility points on Elastic Beanstalk hosting model and make something like what we have for Azure Cloud Services.

2. Windows on AWS initially and then maybe Linux with Core CLR? In both cases, the hosting model can be identical for AWS and GCP.
3. What about telemetry/logging/monitoring?

Given that AWS/GCP provide some PaaS hosting model for either Windows or Linux, I would explore that to make easy deployment as we have with Azure Cloud Services later on. .Net Core is something that will be pending until we finish the port.

[galvesribeiro]

Correction... Look like AWS doesn't have a service like Azure Cloud Services like Web/Worker Role if I understood correct. Will see later on how to enhance the hosting model for it in case there is another possibility.

[gabikliot]

Yes, that was my understanding as well.
On the other hand, GCP does have PaaS offering, it is called App Engine Flexible Environment (previously called managed VMs) https://cloud.google.com/appengine/docs/flexible/ (this is in addition to Standard Environment, which is even more managed but less flexible).

It doesn't support C# out of the box (https://cloud.google.com/appengine/docs), but there is also custom runtime (https://cloud.google.com/appengine/docs/flexible/custom-runtimes/). I haven't tried that and don't know if .NET can be supported this way, but I can check.

[galvesribeiro]

@gabikliot perfect! When I get to google side after this AWS engagement I'll have a look to it and probably ping you for support. Thanks for the links :)

[claylaut]

Hi @galvesribeiro,
Are there any updates about the Kinesis Stream Provider?
Thanks in advance!

[galvesribeiro]

@claylaut I was waiting for an updated version that support .Net Core to be released. Last time I spoke with an AWS folk, they were about to release it in the coming months. Will check on that.

[claylaut]

@galvesribeiro, I've used AWSSDK.Kinesis and I've done a sample project with .NET CORE 1 and 2 and it worked fine.

How far along are you in your implementation? We were thinking to write one, but I wanted to void this since I don't have any experience with Stream Providers.

[galvesribeiro]

Not far. I stucked the first time when the native (Java) part of the Kinesis SDK didn't worked for me when running on .Net Core. Then I got in touch with some of the AWS folks for help and they told they was going to release a new version for it.

I pinged them back and I'm waiting for a reply.

In all cases, if you want to contribute the provider, we would really appreciate that. My backlog of stuff to do for Orleans in my very limited free time is on going :)

[claylaut]

Can you give me your branch name, please? So I will check it out and I will try to contribute.

[galvesribeiro]

I didn't pushed that @claylaut. In fact I discarded that because it was very early and the new SDK would change too much. Anyway, there wasn't much you could leverage.

[sergeybykov]

No need to track this as a separate top level issue anymore. Targeted issues can be opened for specific providers if there's interest.