Engine

[huachaohuang]

Applications can use different data APIs to manipulate data in Engula. We can provide some common APIs like map, hash map, queue ...

An example usage may be like this:

let db = Database::open();

// Uses the Map API to manipulate the "map" collection.
let co = db.collection("map");
let map = Map<Int, String>::new(co);
map.insert(1, "hello");

let txn1 = MapTxn::new();
txn1.delete(1, "hello");
txn1.insert(2, "world");
map.commit(txn1);

// Uses the Queue API to manipulate the "queue" collection.
let co = db.collection("queue");
let queue = Queue<Int>::new(co);
queue.push(1);

let txn2 = QueueTxn::new();
txn2.push(2);
txn2.push(3);
queue.commit(txn2);

We should also support cross-collection transactions. For example:

let txn1 = MapTxn<Int, String>::new();
txn1.insert(1, "hello");
let txn2 = QueueTxn<Int>::new();
txn2.push(2);

// Creates a database transaction to commit the above two sub transactions.
let txn = Txn::new();
txn.add(txn1);
txn.add(txn2);
db.commit(txn);

We may also have nested data types like HashMap<Bytes, List<Bytes>>, but the API needs some further design.

The benefits of such APIs are multi-fold:

• They are similar to common data structures that users already familiar with
• They give more context to the implementer to optimize for specific access patterns. For example, a hash map API can assume that it should optimize for point access.

The implementation of these APIs can be very similar:

• Pack transactions into a log record and write it to a Journal
• Insert transaction operations to a WriteBuffer
• Flush the WriteBuffer to a Storage when they see fit
• Implement some compaction strategy to reorganize data periodically

[skyzh]

// Cross collections transaction

This part looks weird to me. If txn1 and txn2 operate on the same collection of HashMap, txn1 deletes a, and txn2 also deletes a, what will be the outcome?

And if either of the txn in a collection fails, how can we know which txn failed?

[huachaohuang]

Hmm, maybe some misunderstanding here, let me improve the example first.

[huachaohuang]

So if two transactions operate on the same collection, the result will be the same as running the two transactions sequentially and atomically.
Cross-collection transactions are committed atomically, so they should fail together.

[wolfkdy]

So if two transactions operate on the same collection, the result will be the same as running the two transactions sequentially and atomically. Cross-collection transactions are committed atomically, so they should fail together.

So you are considering Serializable Isolation, the single-node Serializeable Isolation level is not easy to implement, it checks the relation between readset and writeset. A distributed implementation will be even more complex. Maybe the relaxed Snapshot Isolation is considerable. You may refer CockroachDB Txn and Papers for more informantion.

[huachaohuang]

Yes, those are very complicated. Maybe we only need to support some basic mechanisms so that users can implement various isolation levels on top of Engula.

[w41ter]

@huachaohuang Have you considered adding condition checks in Txn?

[huachaohuang]

Do you mean something like compare-and-swap? I think we should and we need a nice API for this. For example, to decrease a counter only if the counter is greater then zero:

let map = Map<String, Counter>::new();
let expr = Counter::dec(1).when(gt(0));
map.mutate("abc", expr);

[tisonkun]

On database domain, data migration, replication, or general stream processing require stream api or observable data structure. It seems the design above omits this part.

[tisonkun]

Even if we don't have to care about DDL synchronization and other complex concept on data replication in database domain, users of a storage engine may want to subscribe data changes (change data capture). I don't know whether a subscription on journal could be sufficient.

By the way, a bare data stream struct may be useful for storing event stream:

let pub = Pub<Int>::new()
let sub = Sub<Int>::new(onSubscribe, onNext, onError)
pub.subscribe(&sub);
pub.submit(items);

[wolfkdy]

WAL(journal) is naturally change data capturer, RocksDB's GetUpdatesSince API is what you need.

[huachaohuang]

This confuse me because Engula is aimed to be a storage engine, not a database. Are those data migration, replication part really needed?

You can regard it as a distributed storage engine or cloud storage engine. Engula takes care of the storage part of databases, which naturally includes things like data migration, replication.

[wolfkdy]

I mean replication, migration and sharding are server-layer components, which should not be included in the engine part.

[huachaohuang]

Well, let's treat "engine" as a more general concept here. Engula aims to help users build distributed databases as easily as they build standalone databases. So common functionalities for distributed databases are also considered in Engula.

[huachaohuang]

To make the ideas more clear, I add a simple example to demonstrate the possible usage here.

[zojw]

I have another but maybe not related question, it seems engula-api module is Rust:ish API, how does another language users call those API? (maybe c/c++ users should be the potential user for this to build the new database by themselves?)

[huachaohuang]

I'm very glad that you ask this question. My idea is to use Protobuf to represent those mutations. So the purpose of different language APIs is to build mutation expressions in Protobuf. Then they send the Protobuf messages (expressions) to a unified API through FFI to the underlying Rust code.

[huachaohuang]

Oh, and it is also possible for other languages to access more low-level components directly. So components like Journal, Storage, Background will also expose programming APIs through FFI.

[huachaohuang]

Another way is that users can write remote components in different languages as long as they implement the same set of APIs. For example, you can write a remote compaction service in Python to replace the Rust one.

[tisonkun]

From #99 (comment) we encountered a situation where the same API name can imply different semantic and probably require to be configured (hints).

I'd like to share the client API design from Apache Curator which allows fruitful configuration on mutation or query.

• Examples
• JavaDoc

cc @zojw @huachaohuang