Architecture

[huachaohuang]

This discussion is about the overall architecture.

References:

• Demo 1: Towards an Elastic Storage Engine
• Top-level Design

[tisonkun]

I'm trying to get the vision of this project and drawing the picture below:

It seems that nowadays, we have multiple data processing solutions for different business scenario, see left upper diagram: ES, Redis, MongoDB, RDBMS, ...

Each business logic should choose their preferring solution(s) and write tons of code to glue up. Thus here comes several federated query engine to try to implement "One SQL to Rule Them All".

Alongside, there are other reimplements on the storage engine just for optimizing the data format or disk representation for each solutions. Most of them are built on RocksDB or other famous storage engine but with also tons of code (duplicate, suboptimal) to be optimized. And here comes Engula who provides a multi-model & cloud native storage engine, which eases solutions to use the same API and gain an optimized storage engine (efficient, elastic, adaptive, and extensible).

Even when the federate query engine grows stronger, and Engula becomes de facto optimized, every business logic can write their load and store logic between these two layers (the middleware is thin enough to be covered by any programmer, with fruitful toolkits and APIs to use), while the operators & analyst make query from the unified query engine. Then we arrive the universe goal - one platform, all data. (To be clear, business logic cannot be "unified" innately).

[dragonly]

But I also think that in most cases, performance is not the most important factor. Simplicity and reliability are

I very much agree with this point. The value of sustainable software is greater. Simple and reliable can prevent the software from becoming too complicated.

This is great, except that those system programmers keep squeezing performance out of the machine (maybe?) still have to "re-invent" their own version of low level layers.

[huachaohuang]

This is great, except that those system programmers keep squeezing performance out of the machine (maybe?) still have to "re-invent" their own version of low level layers.

Yep, I actually quite respect people who build everything from scratch. It shows the craftsman's spirit. Although it is not practical in most cases, especially at work.

[levy5307]

But I also think that in most cases, performance is not the most important factor.

I agree with your opinion. But cost is a very important factor, so I want to ask a question: Is Engula cost-friendly in all scenarios?

[huachaohuang]

Yes, being cost-effective is one of the major goals of Engula. Engula tries to achieve that goal by using resources reasonably, like, don't waste resources and scaling on-demand. But it is impossible for Engula to provide the best performance in every case. So when I said "performance is not the most important factor", I mean that "performance is not so important that people have to build everything from scratch for their specific use cases".

[levy5307]

Got it. Thanks for you reply.

[tisonkun]

@huachaohuang I noticed that you have published 0.1.0 & 0.1.1 version of engula on crate.io https://crates.io/crates/engula .

Can it be revoked or overwritten or we should start with a later version?

[huachaohuang]

Seems it can not be overwritten. We can start with 0.1.2 or something.

[tisonkun]

From #32 (comment), we may start from 0.2.0. If you think it's good to go, please update the title of this discussion.

[huachaohuang]

Hmm, maybe we can treat Demo 1 as 0.1 and the next release to be 0.2.

[Little-Wallace]

I'm interested in how do plan to build multi-model storage. Aurora and FoundationDB are not similar although they both separate log and storage. The storage server of FoundationDB supports the get and put interface for client while the storage service of Aurora only support the page interface and the SQL server works on the page API of storage service. So we can find that the storage service of Aurora is like an elastic-block-service which only allows a special write operation of redo-log format and the storage server of FundationDB is more like a kv-server.

[huachaohuang]

I think you are talking about the data API. Welcome to share you are thoughts here :)

[Little-Wallace]

Their difference is so important so that I do not think we can ignore it. If the goal of Engula is similar to FoundationDB's layers, it would be hard to combine transaction code and storage server, because the storage node only support simple method such as Put and Get.

[huachaohuang]

Engula provides a framework and some components to build databases. So users can choose to use Engula in the way they see fit.

[Little-Wallace]

So engula is more like a HBase?

[huachaohuang]

You can regard it as an extended storage engine (e.g. RocksDB) that takes care of the storage part when you need to build a standalone/distributed/cloud database.

[Little-Wallace]

The most import thing which I care for is whether engula will share a storage (Object Storage or a FileSystem) for every instance. Why users choose a cloud native database, rather than deploying a mysql instance in their IDC? I think it is because user can share some resource with other user so they can pay according to their needs and actual usage.

If the databases shares a Key-Value cluster rather a file-system, there will be a lot of difference.

• Every key-value request will be processed in KV server and the shared KV cluster will cost a lot of CPU, because most of databases cache data by page or block rather than key-value.
• The complex requests such as calculating the sum of a column of data in the range, need to read a lot of data. If you push down them into key-value cluster, it will cost a lot of CPU resource, which may be shared with other users. If you want to only load the file data to calculate node, it will cost less resource to scan a range data in a file-system than a key-value system.
• For key-value engine like LSM Tree, it will be easier to running remote-compaction in a shared-storage.

I think the shared cluster should cost as little CPU resources as possible, because it is difficult to isolate CPU resources according to tenants.

[huachaohuang]

I totally understand your opinions. I need to make it clear that a simple key-value API is just one feature that Engula can provide in the API layer. It's totally fine to add another set of APIs dedicated to relational databases or other scenarios. And users can also decide how to organize their databases. For example, you can store a lot of databases in the same set of journal/storage units. Or you can simply store one database in one isolated set of units. Engula provides the capabilities for developers to build databases, but it is not the database.

[Little-Wallace]

Even If the developers of engula is not going to implement a shared storage, I think it does not matter.
But If engula is going to work on a shared storage, it may add much more code than rocksdb. Such as :

• The manifest writer of rocksdb is based on a sequential writable file. But most of distributed file-system do not support files opened with APPEND mode. Engula need to store the meta of files in a better way, such as storing them in a third-party storage.
• The journal module needs to be decoupled from the memtable module. Maybe engula need to design a better architecture to flush memtable and when to notify the journal module to recycle WAL files.
• If engula is going to support remote compaction, I hope the developers can implement it in a better way than RocksDB. And it still has to do with how to update information of file meta.

[huachaohuang]

Some implementation details are still being designed, but I can provide some thoughts here:

• We will provide a built-in manifest service to store metadata. It may be some kind of data store running a consensus algorithm. Of course, different implementations can exist, such as one based on a third-party storage.
• Journal will be an abstraction to store log streams as described in docs: add journal design document #77. It is up to the compute node to decide when to release logs. A simple strategy is to use sequence numbers for ordering. The compute node can record the minimal sequence number that has been flushed and then notifies the journal to release the corresponding logs.
• Yes, engula is going to support remote compactions. Maybe you can elaborate a little bit about what is "a better way than RocksDB" :)

[Little-Wallace]

Thanks. I gradually understand everything, maybe.

[huachaohuang]

@Little-Wallace Thanks for sharing your insights. I may not able to answer your questions one by one. But I think the following materials can help to answer some of the questions:

• https://github.com/engula/engula/blob/main/docs/design.md
• https://github.com/engula/engula/blob/main/docs/microunit.md
• docs: add journal design document #77
• docs: add storage design document #80
• https://engula.com/posts/demo-1/

[tisonkun]

You may update the TODO above #41 (comment) with the published design 👀

[huachaohuang]

Hmm, maybe we should redirect discussions to #57. However, most discussions here seem to aim for v1.0 instead of v0.2 :)

[huachaohuang]

I drew an architecture to help thinking today. Let me add a brief description here.

The first idea is to support a unit group concept in microunit. For example, we can create a group of storage units. We can locate units in the same group with a unique group name. Then we can implement a higher-level storage on top of a storage group. The higher-level implementation distribute objects among the storage group, and pulls information from all storage units in the group to form a global map of objects. So given an object name, the higher-level is able to tell the location of the object without external metadata.
Other components like Journal, Manifest, and Backgroud share a similar design with Storage. All these components can serve one or more databases, which provide the flexibility to share resources in different ways.
Another idea is to make Manifest a general-purpose metadata storage so that it doesn't need to care about upper-level logic. Then we can add a VersionSet on top of Storage and Manifest to get a transactional object storage system. "Transactional" means that we can add/remove multiple files atomically, which is very important for compactions.

[tisonkun]

By "higher-level storage", do you mean that in a universe we have a unique facade to talk to all storage units? Or there are still multiple groups in one universe so that a group works as a label of k8s pods.

Are instances outside "Microunit" border still units managed or single processes?

[tisonkun]

And "VersionSet" looks like a component factored out from the previous compute unit (ref arch).

[huachaohuang]

By "higher-level storage", do you mean that in a universe we have a unique facade to talk to all storage units? Or there are still multiple groups in one universe so that a group works as a label of k8s pods.

We can have an arbitrary number of resource groups (journal, storage, ...) in a universe. For each group, we can create a wrapper implementation to utilize the resource group.

Are instances outside "Microunit" border still units managed or single processes?

I don't really understand this question. Do you mean whether we manage external processes outside of microunit?

And "VersionSet" looks like a component factored out from the previous compute unit.

Yes, more or less. My intention is to move common functionalities to the kernel as far as possible to simplify the API layer.

[tisonkun]

Do you mean whether we manage external processes outside of microunit?

Yep. I remember that in demo-1 and one of our discussions before we have a consensus that engula the storage engine builds all of its components independent from microunit, while microunit is a cluster/resource manager that can run binaries of those components. I'd like to confirm this point and check that in the arch graph above, although Kernel & API layer parallels with Microunit, they can also be units managed by microunit.

[huachaohuang]

Oh, I think I understand your questions now. Let me make things more clear here. While microunit can run external binaries, I think most users don't need to do that. We (or users) can build a single binary that includes all the components. Then we can deploy a microunit universe using that binary. The universe allows us to provision various kinds of resource units/groups with APIs. Each unit can serve one or more upper-level services (API, journal, storage, ...). So ideally, all components of Engula can be run on microunit. Of course, users can choose to rely on other external services, although I think it is unnecessary in most cases. In other words, microunit is responsible for the "physical" part and kernel components are just logical concepts on the "physical" part.

[huachaohuang]

The paper Log-structured Protocols in Delos is interesting. The stackable engine part is what I prospect on Engula modules.

[w41ter]

@huachaohuang After reading the latest design document (#132), I still have some question about the detail.

• How is engula organize the data, hashing or ordering?
• How to shard the data to achieve elastic, hash or range?
• Does engula support multiple writing ?
• Does the journal need to ensure that there can only be one writer at the same time?

[w41ter]

Thinks for you reply.

I still have some questions to ask, but my English is too poor, so I have to introduce a specific scenario to illustrate my problem to avoid inadequate expression.

My question is mainly about how the different modules of Engula cooperate to provide the ability to build a database. Suppose I am going to use Engula to implement a KV engine similar to rocksdb.

Firstly, I dicide to serialize write batch into events and append to a stream, which could keep the data durability. Once persistent events successfuly, I will apply the mutations into a Memory Table. So the first question is should I implement a memtable manually?

After a while, the amout of data in memtable exceeds the threshold, I am going to evict some immtable to a SST file. Those SST files will upload to storage with a specified object id. After that, I need to update the Manifest to keep an snapshot of current version. The second question is who response to flush mem-tables to SST file? Shall I submit a backgroud job or just finish it in the compute unit?

And the third question is, now a user submit a read-only request, how do I determine which level each objects which I upload before.

After a long time, there is a lot of garbage data in a certain layer of LSM tree. I plan to add a compaction function. The forth question is, does Background just a module that provide code execution? and I should write the strategy manually?

The last question is, I don’t use the API module, how can I use it?

[huachaohuang]

So the first question is should I implement a memtable manually?

You can, but I think we should provide some built-in memtable implementation, for example, a skip-list like RocksDB.

The second question is who response to flush mem-tables to SST file? Shall I submit a backgroud job or just finish it in the compute unit?

The compute unit that owns the memtable can flush it to the storage.

And the third question is, now a user submit a read-only request, how do I determine which level each objects which I upload before.

This depends on how the API organizes objects. If you implement your own compaction strategy, then it is your responsibility to decide the object layout. Or we can provide some built-in compaction strategies, like level comapction or universal comapction in RocksDB.

The forth question is, does Background just a module that provide code execution? and I should write the strategy manually?

I think this is covered above.

The last question is, I don’t use the API module, how can I use it?

You can use the kernel directly. I just have a simple description in Discord, so let me simply quote it here:

I think we can make Engula work similar to k8s. We can run a kernel service in k8s. Then we can submit an engine specification to the kernel service. The kernel service is responsible to provision the corresponding resource for the engine. API can connect to the engine to do whatever it wants. An engine specification contains options of sub-modules, for example, what kinds of journa or storage to use, and how much resource it needs.

I think most of your questions can be answered once I provide the kernel abstraction, which is exactly what I am working on here.

[w41ter]

This depends on how the API organizes objects. If you implement your own compaction strategy, then it is your responsibility to decide the object layout. Or we can provide some built-in compaction strategies, like level comapction or universal comapction in RocksDB.

Is it means that Manifest or kernel should save some meta data of objects?

[w41ter]

If you implement your own compaction strategy, then it is your responsibility to decide the object layout.

So how is binary organized? Do I need to embed engula into customer's code?

[huachaohuang]

Is it means that Manifest or kernel should save some meta data of objects?

Yes, manifest is a metadata storage and kernel maintains versioned metadata on it.

So how is binary organized? Do I need to embed engula into customer's code?

Users can use an engula engine like how they use RocksDB. Check #140 for an example. Or we can provide a database service on top of an engula engine too.

[tisonkun]

When advocating Engula to potential contributors, I found it would help a lot if we can divide the module into logical parts.

Concretely, I can see four independent parts in our project:

1. Storage, the kernel part of Engula, consists of Storage, Journal, Cache, upper-level APIs for the server-side. They're coupled designed.
2. Data API, the client-side API, defines how users can communicate with an Engula instance.
3. MicroUnit defines how to manage deployment. In addition, the command-line interface.
4. Background defines how to run general background jobs.

If we somehow converge logic parts, we can advocate them in parallel. And potential contributors can find targets they're interested in easily - it's a fine-grained division than the whole project but coarse-grained than modules some of which are actually coupled.

@huachaohuang I need your help to confirm that this taxonomy works so that I can push further work in advocating and dev docs.

Background: I found that for open-source projects that indicate implementing a SQL database, the knowledge is well-known in the parser, the planner, the execution, the expression, etc. so that contributors can easily migrate knowledge. We should do the same educating job for contributors to understand how to zoom in Engula project and find their comfort zone.

[huachaohuang]

Maybe three logical parts:

1. Engine, consists of core modules like Storage, Journal, Metadata, Background jobs.
2. Data API, built on top of Engine, this is the part users build their databases.
3. Resource management, or Microunit, provides an abstraction to the underlying resource pool. Engine talks to Microunit to get the resources it needs.

Background can be counted as the Engine part.

[tisonkun]

Thanks for your insights!

From advocating perspective, I'd like to separate "Background" because it looks like running general background tasks so that a contributor without knowledge of storage engines can work on it.

But with a closer look, if "Background" means engine-specific background tasks such as compaction, it's coupled with the storage engine.

[huachaohuang]

Yes, "Background" may be able to run arbitrary kinds of jobs, but I don't see why users will use it that way. I mean, they can simply use k8s to run whatever they want :)

[feitian124]

It is a good idea to combine different implementations into a more powerful one. For example, we can create a hybrid storage that persists data to a slow but highly-durable storage and then reads data from a fast and highly-available storage.

I like this idea.
Time and space are the most important resource, the journal and storage design seems match the domain well.
there are time series databases, sql databases, graph databases and kv etc, each focus on a specific situation and only good at solving problems under that situation.

engula provide the potential ability to combine them together, right?

[huachaohuang]

Exactly, I think you get the essentials of Engula.

[zaorangyang]

Hi guys. I have read the design documention. I am interested in Engula and want to make contributions to it. But I have some questions：
● Engula is a storage engine separating storage and computing. The engine is compution layer and the kernel is storage layer. The computing layer can scale horizontally. Am I right?
● Currently, Engula is like a distributed LSM tree storage engine according to the design documention. Would you introduce b-tree storage Engine in the future? Although LSM trees are used in many new databases, the read performance of B trees is better than LSM trees. In particular, most scenarios have more read than write load.
● Finally, with the advent of non-volatile memory, do we consider optimizing for AEP at the beginning of our design? Perhaps in the near future, AEP will be cheaper and perform better, we will no longer need to write data to SSD/HDD, and all disk-based storage engines in OLTP will be redesigned.

[huachaohuang]

• Engula is a storage engine separating storage and computing. The engine is compution layer and the kernel is storage layer. The computing layer can scale horizontally. Am I right?

Yes.

• Currently, Engula is like a distributed LSM tree storage engine according to the design documention. Would you introduce b-tree storage Engine in the future? Although LSM trees are used in many new databases, the read performance of B trees is better than LSM trees. In particular, most scenarios have more read than write load.

I may or may not implement a b-tree engine myself. But I am open to this if anyone has a solid proposal and implementation.

• Finally, with the advent of non-volatile memory, do we consider optimizing for AEP at the beginning of our design? Perhaps in the near future, AEP will be cheaper and perform better, we will no longer need to write data to SSD/HDD, and all disk-based storage engines in OLTP will be redesigned.

To be honest, I am not sure if NVM will be mainstream in the future or it is just hype. Anyway, I don't see the necessity to design for NVM so far until cloud vendors buy in.