Should the Engine talk to "internal" concepts (Storage/Journal) directly?

[tisonkun]

@huachaohuang I commented on the original Warehouse docs PR that we've reached a consensus about wrapping the Storage and Journal behind an upper-level interface (I think it's Kernel now). But currently, an engine still talks to all of Kernel, Storage, and Journal. What's your plan for this abstraction?

Reference

In this way, a client should talk to both Storage and Warehouse. So a user should keep in mind these usages. It seems a bit awkward. I'll imagine with the Warehouse facade, a user can talk to the Warehouse only.

Originally posted by @tisonkun in #114 (comment)

Make senses. We can hide Storage from the abstraction.

Originally posted by @huachaohuang in #114 (comment)

With #140 and #143, a client (engine) is still holding all of:

• Kernel (Warehouse here)
• Stream
• Journal

#[derive(Clone)]
pub struct Engine<K: Kernel> {
    kernel: K,
    stream: K::Stream,
    bucket: K::Bucket,
    // ...
}

Originally posted by @tisonkun in #114 (comment)

[huachaohuang]

Well, it depends on what you mean "talk to". From the perspective of an engine, it doesn't care about where the journal/storage is. It uses whatever the kernel provides to it. However, inside the kernel, it may simply connect to a journal service and talk to it directly. We can't imagine that every read/write request goes to the kernel first.

[w41ter]

So instead of simply getting a stream/bucket from the kernel, you need to create a stream/bucket from the kernel first. And when you need to use a stream/bucket, you need to ask the kernel for the location first.

So, what is the role of Journal?

[huachaohuang]

Journal provides the functionalities for Kernel to manage streams. Engine doesn't need to know about Journal.

[huachaohuang]

We should separate the Journal and Stream concepts here. Engine uses Stream doesn't mean that Engine needs to know what a Journal is. From the perspective of Engine, it is Kernel who provides the Stream.

[huachaohuang]

Hmm, 🤔 maybe we can make Stream an independent concept that can be used without Journal. Journal is a concept that manages multiple streams. But we can manage streams in different ways. For example, Kernel actually provides another way to manage streams, which doesn't share the same interface as Journal.

[tisonkun]

If so, the functionality of journal is thin and likely to be inline into the kernel.

The kernel is growing to a oneforall interface that composes metadata management and storage interfaces. In other distributed storage system, metadata is handled separateltly like placement driver.

I like the original design of engula that servers consist of multiple units and let the client compose its requirement, while we can provide some basic configs like all in memory. And treat the metadata management (placement driver, coordinator, etc.) separately.