The design of leader based Journal

[w41ter]

In Engula, Journal provides abstractions and implementations to store data streams. For example, transaction logs. The Journal is mainly used by Engine, but also used by Kernel who need to persist metadata. Unlike the previous discussion of journal: #70 , this discussion only focus on the design and implementation of leader based Journal service.

I want to discuss design from two aspects:

• reconfiguration
• storage medium

Reconfiguration

Here are two kinds of reconfiguration of Journal:

• In of band reconfiguration: raft
• Out of band reconfiguration: LogDevice, Delos

Those out-of-band reconfiguration journal system use sealing and bridge records to switch configurations, so their advantages are 1:

• provide higher availability in reconfiguration
• privide flexible replication policy
• allowing different requirements and design choices in data plane vs. metadata plane

Of course, the out-of-band reconfiguration journal system requires another store to save metedata, such as epoch number, copyset. The in-of-band reconfiguration journal system haven't such requirement, so the could be used independently.

In Engula, the Kernel might use the in-of-band reconfiguration journal system to persist metadata, and Engine use the out-of-band reconfiguration journal system to persist log. So I plan to design and implement a raft based journal system to supply Kernel, a journal system using sealing and bridge records, flexible paxos to supply Engine. The latter also uses the former to persist out-of-band metadata.

Storage medium

The simple way is store all data in local disk, like some raft impelementation. But the local capacity is limitted and once reconfiguration occurs, the entires data need to replicate to the new nodes.

Cloud object store could be used to persist logs, and the underlying storage could provide availblity. But not all large-scale storage
systems have an atomic put-if-absent operation, so a separate lightweight coordination service need be used to ensure that only one client can append data to log 2. At the same time, each user operation put a piece of data to the object storage, which is not efficient; batching will increasing the latency.

In order to achieve both low latency and zero-move. we could save recently data in local disk but move sealed (or read-only segments) to cloud object store.

If you have a better design, welcome to discuss.

[huachaohuang]

Thanks for starting this discussion. The in-of-band and out-of-band classifications are informative.

In Engula, the Kernel might use the in-of-band reconfiguration journal system to persist metadata, and Engine use the out-of-band reconfiguration journal system to persist log.

This direction sounds very interesting.

So I plan to design and implement a raft based journal system to supply Kernel, a journal system using sealing and bridge records, flexible paxos to supply Engine.

For the raft-based journal, do you think we can store all data in cloud object storage? If so, Kernel doesn't require a disk to run.

[huachaohuang]

For the raft-based journal, do you think we can store all data in cloud object storage? If so, Kernel doesn't require a disk to run.

BTW, the idea of storing logs in cloud object storage comes from DeltaLake.

[w41ter]

For the raft-based journal, do you think we can store all data in cloud object storage?

Yes, we can easily adapt raft's LogStorage to the cloud object store.

In design aspects, if we abstract properly, then the underlying storage medium will not effect the reconfiguration layer design.

[Little-Wallace]

For the raft-based journal, do you think we can store all data in cloud object storage? If so, Kernel doesn't require a disk to run.

Cloud object storage may cost much duration on persist the log entries. For Kernal, we can implement two kind of journal, one is raft-based on local nvme, Kernal will persist data in the journal by RPC, the other is MQ based on cloud object storage, it will keep a small data in memory and flush them into cloud object storage periodly.
So Kernel doesn't not require a disk, it just require a different service.

[tisonkun]

A useful reference is KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum, how Apache Kafka implements their quorum mechanism. Sorry I don't take a closer look into our use case yet, but election by an append only data structure can share many natures.

[w41ter]

I have read the references, and I found that they use the way we want to solve the problem we meet.

[Little-Wallace]

I’m happy to talk about the question "Do we really need the raft-protocol?".
In actually, what we real need is to make sure the data will be applied in order just as the order we proposed them.
For database like Aurora, it persists redo-log with quorum-write instead of raft. But it acquires that Aurora needs an independent service to decide which is primary node and then this primary node can collect all redo-log data which has not been written to buffer-pool. This will cause some more problem such as some log has been replicated to one node but the other node does not exist this log, the new lead need to decide whether to commit this log.
Kafka may be a good choice to store data of journal, but I'm not sure whether Kafka can meet performance needs.
For database scene, I do not think we shall create a complex quorum-write algorithm rather than use raft directly. Aurora and Foundation did not choose raft, I do not think it is a good choice. Maybe these two projects were established so early that the raft consensus was not created at that time. They did a lot of work to make sure the replica algorithm for log-file was correct and it is not a good choice for a new project developed in 2022.

[Little-Wallace]

What I mean is that because we do not contain the journal and Kernal in the same process, so we can easy to replace it with a new journal. I have no objection to someone doing such a pioneering work. I just recommend that you first implement a log service that is as simple as possible so that other developers can quickly focus on the modules they are interested in.

[w41ter]

@Little-Wallace

Thanks for your replies. We really should first implement a log service as simple as possible, so that we can focus on other modules.

For now, we have to implement at least a raft based journal, while others can wait until more time is available. So at this stage, the design and implementation of a raft based journal should be completed first. @huachaohuang what is your opinion?

[huachaohuang]

The primary goal of early design is to reduce uncertainty. In this topic, the uncertainty comes from the engine part (WAL), which can not be solved by a Raft-based journal. If you want a working journal, you only need to design the API and plugin an existing log service, like Kafka, LogDevice, or simply use Etcd for Kernel. But I don't think we need to bury ourselves in anything complicated before we have a more clear path towards our desired state.

[huachaohuang]

Don't get me wrong, I am not anti-Raft or something, it just doesn't seem to work in a disaggregated architecture. But if you think it does, I'm happy to see a design document about it.

[w41ter]

@Little-Wallace After offline communication, @huachaohuang and I reached consensus. At this stage, the main purpose is to solve the uncertainty problem on the architecture and ensure that the expected goal can be achieved (only one RTT is required from engine to journal). This couldn't be achieved by introduce raft algorithm simplely, so I am planing to continue the design of out-of-band journal, the aim is proving the ability of this architecture. After that we will implement it as simple as possible, it might reusuing the exists system, eg: Etcd. If you have any questions, fell free to let me known.

[w41ter]

I have submitted a RFC to illustrate the design ideas: #280.

[tisonkun]

@w41ter-l as the development moved to w41ter-l/shared-journal. I think this RFC as well as the leader based journal can be moved to discussions.

Now, we have 2 RFCs and 2 discussion threads:

• The design of leader based Journal #251
• The API of Leader Based Journal #260
• rfc: add shared journal #291
• rfc: add leader-based journal #280

It's quite fragmented our discussions. You can continue this thread, update the description and send your update on works in w41ter-l/shared-journal so that all developers can catch the progress.

WDYT?

[tisonkun]

For example, the first question can be: What's your plan about integrating engula with shared-journal? What's the estimate and steps?

[w41ter]

@tisonkun Thanks, I will update this thread once I'm done verifying the shared journal.