Azure Tables Persistence #145

Closed
wants to merge 19 commits into
from

Projects

None yet

6 participants

@jvandertil

I have implemented the persistence engine for Windows Azure Tables. Could you please review the code?

@jvandertil

I'm having some issues getting ILMerge to work nicely for my Azure Tables implementation. The same issues appears to be in the Mongo implementation, but not in the RavenDB one. Namely that the Wireup assembly does not seem to be merged into the main assembly.

Any hints on how to tackle this?

@joliver
Contributor
joliver commented Dec 18, 2012

Does ILmerge give back any problems?

On Mon, Dec 17, 2012 at 3:01 PM, Jos van der Til
notifications@github.comwrote:

I'm having some issues getting ILMerge to work nicely for my Azure Tables
implementation. The same issues appears to be in the Mongo implementation,
but not in the RavenDB one. Namely that the Wireup assembly does not seem
to be merged into the main assembly.

Any hints on how to tackle this?


Reply to this email directly or view it on GitHubhttps://github.com/joliver/EventStore/pull/145#issuecomment-11463802.

@jvandertil

No, that's the strange thing. I've tried to manually merge the implementation and wireup assemblies, but ILMerge doesn't output anything and creates the target assembly. I assume that that means a successful execution.

Once I get behind my development machine I can provide more details.

Also, are there instructions on running the persistence acceptance tests? I have been unable to execute them.

@jvandertil

By the way, I already integrated the packaging of the Azure Tables persistence engine into the Build script. (commit 7210403)

The strange thing is that RavenDB does get packaged correctly, but MongoDB and Azure Tables are not. (The EventStore namespace is not in the merged DLL)

@jvandertil

Ok, after some more playing around with ILMerge I found that the issue lies with the '/internalize' switch.

Removing it resulted in a correct merge, adding it caused the EventStore namespace to not be merged.

@jvandertil

And fixed :)

@ptomasroos

Wouldnt this case an extremly slow read since your partition keys may be on 100 servers @jvandertil ?

@jvandertil

Well that depends on the use case. When loading multiple aggregates
potentially multiple servers would be queried, because I used the
aggregate id as the partition key.

However, loading a single aggregate shouldn't be extremely slow since it
belongs to a single partition.

I believed that retrieving or writing a single aggregate was the most used
use case, and optimized accordingly. Since Windows azure can now distribute
each aggregate to a server the implementation should be very scalable.
Since GUIDs are also randomly created this implementation should not suffer
from any append only bottlenecks.

Ofcourse, if a single aggregate would grow too big or too active (not sure
what exactly is considered big or active by azure), azure could decide to
do a range partition on the version (row key) column. But I'm not sure if a
single aggregate getting that big or heavily queried is good design.
Op 16 jan. 2013 13:23 schreef "Tomas Roos" notifications@github.com het
volgende:

Wouldnt this case an extremly slow read since your partition keys may be
on 100 servers @jvandertil https://github.com/jvandertil ?


Reply to this email directly or view it on GitHubhttps://github.com/joliver/EventStore/pull/145#issuecomment-12316240.

@ptomasroos

Right 1 aggregate will be very fast!

But the main responsibility at least according to me and the benefit with the event store is that it takes care of the dispatching.

And to be able to find those undispatched commits it may be a real mess with the support azure gives you.

public IEnumerable GetUndispatchedCommits()

So if we have 10.000.000 aggregates that will be a query on 10.000.000 partitionkeys when you do a.
public IEnumerable GetFromTo(DateTime start, DateTime end)
Or
public IEnumerable GetFromTo(DateTime start)
Or the method above.

I'm not saying that you have done anything wrong i'm just concluding that since there is no secondary index in ATS it is a pain. Either you get fast writes or slow reads when you need to look at a complete stream.

And the risk for range partition is probably very low. An aggregate is probably just around 1mb as largest size :)

@jvandertil

Yeah, those methods will not be very fast (relatively speaking). But I don't believe that those methods are called very often while running the Event Store.

Dispatching is done by passing a Commit through a pipeline, and then the Commit is marked as dispatched.
GetUndispatchedCommits() is run (I believe) when the EventStore is bootstrapped, and not for each commit that is dispatched. With a system of 10.000.000 aggregates that would still increase bootstrap time significantly, and I'm unsure if that is done in a seperate thread or that the entire application is blocked until this function returns.
While I'm not sure if this is true, I assume that the GetFromTo() functions are called when rebuilding 'views' or reports of your aggregates. Also not really a use case to optimize for.

To assist other developers in tracking down these issues, I included in the Logging output a message 'using a FULL TABLE scan' whenever such a query is done. This should raise a flag when people look in to possible performance issues.

Should I include a readme or wiki page that outlines these potential issues?

A bigger issue (I feel) is that I currently limit the header and payload size of a commit to 64 KBytes each, since that is the 'column' limit for a byte[]. I could expand this with a couple of multiples of 64 KB, but I'm not sure what a 'reasonable' limit would be, and how big a normal Commit Payload / Headers byte[] is in a production system.

edit:
It might be possible to parallelize the queries for those functions, but I think that would be better suited for a separate pull request when it is proven to be a huge problem for certain systems.

Still, it does allow you to store 52 GBytes of data as opposed to 'up to' 100 MB in Azure SQL for the same price per month ;).

@ptomasroos

Sure but SQL is still quite cheap comparing to say ravenhq. And everything is relative :)

And you don't pay for the transactions against SQL. But each read in a table will.

How many items / entities have you tried a full table scan on? Would be interesting with 1m. We can provide account and pay the research.

Would like to have some number. I do neither know how slow slow is :)

64KB is not an issue. Having larger messages than that is not recommended and not likely. My are usually around 80 in avg.

Haven't looked through all the code but I wonder is it using optimistic concurrency / etag if I were to use two trying to write to the same ARid?

Tomas

On 16 jan 2013, at 17:33, Jos van der Til notifications@github.com wrote:

Yeah, those methods will not be very fast (relatively speaking). But I don't believe that those methods are called very often while running the Event Store.

Dispatching is done by passing a Commit through a pipeline, and then the Commit is marked as dispatched.
GetUndispatchedCommits() is run (I believe) when the EventStore is bootstrapped, and not for each commit that is dispatched. With a system of 10.000.000 aggregates that would still increase bootstrap time significantly, and I'm unsure if that is done in a seperate thread or that the entire application is blocked until this function returns.
While I'm not sure if this is true, I assume that the GetFromTo() functions are called when rebuilding 'views' or reports of your aggregates. Also not really a use case to optimize for.

To assist other developers in tracking down these issues, I included in the Logging output a message 'using a FULL TABLE scan' whenever such a query is done. This should raise a flag when people look in to possible performance issues.

Should I include a readme or wiki page that outlines these potential issues?

A bigger issue (I feel) is that I currently limit the header and payload size of a commit to 64 KBytes each, since that is the 'column' limit for a byte[]. I could expand this with a couple of multiples of 64 KB, but I'm not sure what a 'reasonable' limit would be, and how big a normal Commit Payload / Headers byte[] is in a production system.


Reply to this email directly or view it on GitHub.

@jvandertil

Well I could implement support for 128 KB headers and 128 KB payload (it's just adding a field and some splitting and merging logic), which 'should be enough' to support well designed systems for now.

Optimistic Concurrency is used in the code. I'll try and write some supporting tests to confirm this for you, as well as a load test somewhere in the 1M AR's region.

edit:
I do not have a Azure account myself, so I haven't tested this against the 'real' Azure Tables. Only against the DevFabric.

@ptomasroos

80 bytes that is! Not kb! Sorry for not making this clearer!

On 16 jan 2013, at 18:42, Jos van der Til notifications@github.com wrote:

Well I could implement support for 128 KB headers and 128 KB payload (it's just adding a field and some splitting and merging logic), which 'should be enough' to support well designed systems for now.

Optimistic Concurrency is used in the code. I'll try and write some supporting tests to confirm this for you, as well as a load test somewhere in the 1M AR's region.


Reply to this email directly or view it on GitHub.

@jvandertil

Ah, that makes much more sense :)

@jvandertil

A small initial test on the DevFabric shows that inserting 10000 AR's (with 1 commit of 80 bytes) takes about 2 minutes, retrieving them takes about 3 seconds.

These numbers are of course meaningless since the real Azure will not behave like this.
I can put the code to test the implementation in a repository, so you can run some tests with it against the live Azure. Or if you want we can collaborate on this.

Optimistic Concurrency also functions as expected:
https://gist.github.com/4550276

@ptomasroos

If you got the code available I can try something out on production tomorrow.

With 1m streamids. Writing will be fast each PK has up to 20k trans/second.

And as long as the boot time isn't 50 minutes for the queries I think it sounds very nice.

When you do queries and over multiple partitions you can only get 1000 rows back and a continuation token and that as well worries me a bit.

But I wanna give it a try at least :)

Writes are slow in SQL but reads much more optimized for multiple indexes.

They have recently flattened their internal network so the throughput and latency should be better as well.

On 16 jan 2013, at 20:56, Jos van der Til notifications@github.com wrote:

A small initial test on the DevFabric shows that inserting 10000 AR's (with 1 commit of 80 bytes) takes about 2 minutes, retrieving them takes about 3 seconds.

Optimistic Concurrency also functions as expected:


Reply to this email directly or view it on GitHub.

@ptomasroos

Sure lets collaborate on it, i'll fix a Persistent vm you can remote to and then a storage account we can abuse.
And meassure the recovery to give some kind of numbers on real use-cases.

Are you available on gtalk/skype?

@jvandertil

Yes, I am available on Skype :). You can find my contact details on my profile page now.

@ptomasroos

Added, lets continue on there.

jvandertil added some commits Jan 17, 2013
@jvandertil jvandertil Now really fixed the internalized Wireup assembly issue c9de241
@jvandertil jvandertil Fixed StorageException being bubbled up to the user when updating str…
…eam heads (which is async) when a concurrency conflict happened when inserting the stream head (Thanks to pthomasroos!)
5c301ae
@gblmarquez

Hi guys,

Maybe you could use the project that Damian had done to test performance on EventStore.
https://github.com/damianh/EventStorePerf

@damianh
Contributor
damianh commented May 17, 2013

In v4 we're going to split out the provider / storage implementations from the core lib and put them into their own repos. We'd be delighted if you could maintain the AzureTables provider?

@ptomasroos

I can't really help out since i'm not using the event store / .net platform.
Or rather its not aligned with my interest :)

@damianh
Contributor
damianh commented May 21, 2013

@jvandertil Will you be interested in maintaining the AzureTables provider in v4?

@jvandertil

Yes I will be. It is however not yet production ready, so it should be marked as such.

Until I can improve the start up performance to an acceptable level it should remain in testing.
The start up performance starts to deteriorate when you increase the number of commits, due to the inability of Azure Tables to maintain secondary indexes. The query to retrieve the list of commits that have not been dispatched results in a full table scan.

I'll have to look into that, but I didn't have the time recently.

@damianh
Contributor
damianh commented May 21, 2013

@jvandertil Thx for the fast reply and that is absolutely fine. We've just pushed out 3.2.0-beta1 packages to nuget.org and going start working on V4 straight away where we'll be splitting out the providers into seperate repos so they will have their own release cycles. As such, I'm not going to pull this into this repo, but please hold on to it for it's own repo.

@damianh damianh closed this May 21, 2013
@danielHalan
Contributor

What is the current status on the Azure Table engine?

@damianh
Contributor
damianh commented Nov 27, 2013

Up for grabs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment