CosmosDB docs should contain a recommendation on how to save costs when using actor state management to handle large documents

[KrylixZA]

Feel free to throw this issue out the window if you believe it doesn't belong in the Dapr docs, but I thought it might be nice to have :)

What content needs to be created or modified?
The Dapr CosmosDB state management documentation should maybe mention that CosmosDB's billing profile is a consumption-based billing profile. This means that when using Dapr to read and write state, more specifically Actor state which is raw, formatted JSON, if the actor's state object grows in size, it will consume a fair chunk of money to run. This grows linearly with document size and concurrency. One way to combat this growth in cost is to compress the document using something like GZip before saving state and decompressing when getting state.

Describe the solution you'd like
Something simple like a new heading on savings costs. Could look as simple as:

Saving costs when using actors to store large documents

Cosmos DB uses a consumption-based billing model. This means that if you have an actor state object that is large in size, you will pay for every KB you write on every actor call and for every get each time an actor is hydrated. For example, if you have an actor state object that is 75KB in size when the actor is hydrated, you will use 75 RU/s to read that object out of state. If you then modify the state object and it grows to 100KB, you will use 100 RU/s to write that object to Cosmos DB. Totalling 175 RU/s for the I/O operation. Let's say your actors are concurrently handling 1000 requests per second, you will need at least 175,000 RU/s to meet that load.

This is largely because Dapr actor state is saved as raw, formatted JSON in the state store. Furthermore, Dapr interacts with state as key-value pairs and as such, there is no definite need to store the actor state as raw JSON. One way to save costs is to compress the state object before saving. This can be done with any compression algorithm but ideally pick one that favours speed over outright compression optimisation.

Done correctly, this can reduce costs by 95% or more.

It could also be worth linking to this header where it mentions:

If you are using the Dapr SDKs (for example the .NET SDK), the SDK automatically serializes your data to JSON.

Where should the new material be placed?
Under the Azure Cosmos DB (SQL API) page at the bottom below the Optimizing Cosmos DB for bulk operation write performance header.

The associated pull request from dapr/dapr, dapr/components-contrib, or other Dapr code repos

Additional context
This is inspired by some good work one of my colleagues has recently completed and published an article about: Maximizing Efficiency in Scalable Systems: Part I — Reducing costs of consumption-based state stores by up to 97% when using Dapr actors at scale

[olitomlinson]

@KrylixZA IMO this is good content to add!

Also, did you attempt to disable indexing on the container, as it's my understanding that this can be a huge RU saver for writes?

[KrylixZA]

Hey @olitomlinson

Yeah we've been running with all properties excluded from indexing for a while. That was driven more from a performance aspect in our case but happily helped with costing too. So we're running no indexing and compressed data at the moment and seeing huge ROI for that. Obviously the compression really only works when you're interacting with the state store in a key-value kind of way but that is how Dapr works through the state management API so it's relevant.

Truth be told, they've made our product viable in the longer term.

I'll try get a PR out for this soon 👍🏻

[olitomlinson]

@KrylixZA

Going back to your original point about compressing the state, I assume you're doing all this in your application layer?

To me it seems reasonable there should be an option to compression / decompression the state transparently by the cosmosDb state store component, so that end-users such as yourself don't have to take on this responsibility. What do you think?

[joshuadmatthews]

Is there something inherent to actors that cause extra queries to be done here? I am thinking, if you were to build the same "workflow" without the actors and query the state you need from a database each time your long running transaction needs to continue, would that be cheaper than using actors?

I guess what I'm getting at here is, does the actor framework abuse cosmos or is this just by nature of what you are asking it to do? If the latter, I don't really see how a warning is relevant.

[KrylixZA]

Hey @joshuadmatthews

From my experience, the actor framework is extremely efficient at managing state interactions.

Given it only reads from state when the actor is hydrated, you're saving a read every time the actor is called and it's sitting alive in memory.

The only thing about actor state is that it's raw, formatted JSON. By nature, raw JSON uses up a lot of KBs, and therefore RU/s in CosmosDB. This is because of all the white space and line endings to make it formatted.

Building the same code without the actor pattern would be very hard to get right. Specifically working around the race conditions reading and writing state. Actors are good at that, by nature of their implementation. You'd likely build it worse if you had to do it yourself. Or at least I would ;)

Alls I am suggesting, instead of a warning, is just a little note block that informs the user that if they are using actors and have no intention to query the state through bindings or the SDK directly, then dapr is only going to use the state in a key-value fashion. And by virtue of that, users stand to gain a lot in terms of financial benefits if they store their actor state in a compressed format.