CRDT-Batching support

[hsanjuan]

Describe the feature you are proposing

go-ds-crdt supports batching. Multiple updates can be included in the same DAG node. With batching, we do not need to issue one block per update.

Cluster with heavy pinning/unpinning intake can take advantage of batching to greatly reduce (I think around ~20000 pins can go in a batch) the DAG size.

Batching can be two ways:

• Time based: A batch will be published every X seconds, if any updates have been made. This allows to group all updates every 5 seconds in a single batch.
• Size based: A batch will be published as soon as X updates have been made.

Both approaches can be combined (whatever condition is hit first issues the batch). i.e. A batch should be issued every 100 updates, but in should be sent in any case if 10 minutes have passed from the previous batch.

Note that in-principle batch support is in-memory. Uncommitted batches will be lost.

Batching should be configurable in the crdt config section:

"batching" : {
  "max_batch_size": number_of_updates,
  "max_batch_age": "duration"
}

0 values disable batching.

[RubenKelevra]

Sounds nice, should reduce the amount of height I collect per day significantly ;)
INFO crdt: crdt Datastore created. Number of heads: 1. Current max-height: 10990 crdt.go:262

---

How about the ability to open a batch and either commit it or discard it?

This way we can commit all changes to an open batch, and when everything is done can commit it as an atomic operation to the cluster.

If it fails to get the cluster head, the commit operation should fail and the whole batch gets discarded by the cluster.

This helps if multiple servers should write to the same cluster since you can add a retry on the application level, just running the same operation again on the new head/height.

So we're basically talking about 3 different modes:

• set number_of_updates/duration -> just commit from time to time to the cluster (might result in data loss on concurrent writes). A commit/sync operation would be a nice addition, so a not yet expired duration can be considered expired and the batch will immediately pushed to the cluster.

• no batching at all -> commit every operation alone (one operation might fail on concurrent writes)

• open batch/close batch -> a logical write operation with many changes coalesces into one big batch. The close batch will wait until the majority of the cluster has returned a status, thus deciding that that's the new head/height.

Edit: Rewrote the comment

[hsanjuan]

How about the ability to open a batch and either commit it or discard it?
...

Sorry, but this issue is very well scoped already. Batches are local. There is no synchronization between batches started in different peers. Concurrent write operations in a cluster don't fail by design.

The close batch will wait until the majority of the cluster has returned a status

You underestimate the pitfalls and effort needed to set up coordination like this on a distributed system. It is way more complicated that it seems.

The ability of manually committing an open local batch, while useful, would need to be extended to Raft too which has no batching support at all. Therefore out of scope for the moment.

[RubenKelevra]

How about the ability to open a batch and either commit it or discard it?
...

Sorry, but this issue is very well scoped already. Batches are local. There is no synchronization between batches started in different peers. Concurrent write operations in a cluster don't fail by design.

I think you misunderstood what I meant.

Your approach sounds like auto-commit of mysql. Every now and then changes are flushed to the cluster.

To cut my idea down, can we get non-automatic batching?

• one command to open a new batch
• one command to discard the current batch (like IPFS-Cluster crashed)
• one command to close the batch and flush it to the cluster.

The close/flush command could get a similar --wait flag like the pin-operation already has. So it will return when the cluster has completely answered.

The idea with a reduced 'wait quorum' would need to get the current number of peers, like peers ls from the cluster and return as soon as half of the numbers of acknowledges came back instead of 'all', which might wait indefinitely when a cluster member goes offline.

If that's too much for one change I fully understand! :)

[hsanjuan]

can we get non-automatic batching

I'd say yes, after automatic batching support as scoped here exists (this is smaller in scope and does not require API changes).

I see it would be super useful to you, just trying to define well scoped work units. I will give some thought and open a follow up issue on non-automatic batching.

[RubenKelevra]

Sounds great!