Replies: 5 comments 7 replies
-
For example, current memory kernel must init with preconfigured one memory journal & one memory storage. I don't think it's the final state for a serverless storage engine. A kernel should be able to manage multiple journal & storage and support engine create stream over a remote journal or an external journal like Kafka. However, what we deliver to end user is engines. If there is no engine talking to the engine at all, I don't know whether or not my patch on kernel is valid or satisfy the requirement of the engine. And thus I suspend myself for waiting the whole storage engine model to be compiled. |
Beta Was this translation helpful? Give feedback.
-
|
LGTM. A evolving workable software is also valuable for identifying the actual demands of the underlying components. |
Beta Was this translation helpful? Give feedback.
-
|
But also to ask another question: is it also related to branch development pattern? If we want At last, I agree with the opinion that we should have an always test-passed & workable dev-branch(main) to let other developers can work on it too :) |
Beta Was this translation helpful? Give feedback.
-
|
@zojw Here are my two coins:
Addition to 3, it hurts that we opt-out hash engine and leave it as is, no plan to bring it back nor drop it (#272). We cannot say "leave it as is and see what will happen", because its broken is an obvious problem and you must have a plan for it. So do the gRPC implementation and file implementation Opt-out things doesn't solving the inherent problems. Those implementation should help sharping the design instead of block it. If it blocks, the design itself is suspicious. Prototyping that opt-out things can be made in personal branch, and later shared on issues, call for contributors to work for a series of clean patches if it's too much to be covered by one developer. I think you guys are willing to help :) All in all, temporary breaks are acceptable as long as there is a plan to fix (and should be delivered in time), not leave it as broken windows. Any contributor can ask for help from others to overcome problems together.
I guess you are included in "other developers" :), as a cache storage is designed to help the luna engine but if it's uncompleted, you don't know whether or not it really helps. Also, you may change the kernel code to coordinate storage instances, but if you don't know how the engine talk to the kernel, you're unsure whether your changes satisfy the usage. |
Beta Was this translation helpful? Give feedback.
-
|
@tisonkun Thanks for your review as well as others' opinions. It sounds right to me, let's make it work. PS: I feel I can back to work now :) |
Beta Was this translation helpful? Give feedback.
-
|
After a discussion with @huachaohuang @w41ter-l @zojw, we notice that the {Kernel|Storage|Journal}Server concepts are redundant for a storage engine. We can write all compute logics in the engine, let the kernel coordinate metadata including provides the handle of storage or journal, and composite the engine & kernel logics running on the user client side. In this case, step 2 can be totally skipped. And Engula provides a storage engine library which can be composed with different storage / journal / manifest implementations. |
Beta Was this translation helpful? Give feedback.
-
|
We may still need to provide a binary for deployment in the future, I haven't considered that clearly yet. But yes, overall, engula is more a storage engine library than a binary, similar to how you use RocksDB. |
Beta Was this translation helpful? Give feedback.
-
|
@huachaohuang but we don't deploy any {Kernel|Storage|Journal|Manifest}Server of engula itself, there's no kernel server, other servers should be provision externally. Even if we'll provide a one-for-all binary to launch external services in the future, it doesn't look like how |
Beta Was this translation helpful? Give feedback.
-
|
Also, I think a storage server for shared storage is its own, i.e., a "shared storage server", not a general "Engula storage server". |
Beta Was this translation helpful? Give feedback.
-
|
Yes, we can clean things up to avoid confusion. |
Beta Was this translation helpful? Give feedback.
-
|
Get it. If you're busy on the luna engine, I'll start do this in the next week :) |
Beta Was this translation helpful? Give feedback.
-
As mentioned in #292, we are working without a workable software a few weeks after #232 merged. It brings several issues here:
localhostin address #222).Thus, I propose that we evolve Engula by these steps:
Deliver a workable software by implementing luna engine with current abstractions (engine, kernel, journal, storage).
And thus we have a minimal completed storage engine model. With a workable software, any contributor familiar with storage engine can get involved in. Also, LeaderBasedJournal's and CacheStorage's design can talk concretely how they help the luna engine deployed on cloud as cluster.
2. Provide a binary to deploy components (kernel, journal, storage) as in v0.2.Sooner or later we have to deploy components in cluster. Proving such a binary helps polishing the communication logic and abstractions between components. At least the design of so-called "Serverless Kernel" should be benefit to have something to validate itself.Replace components with more robust implementations.
This is where LeaderBasedJournal & CacheStorage or other implementations come in. As components replaced, we can finally achieve the target Cloud Engine on AWS #270. We just cannot achieve it without a completed storage engine model.
These steps follow the original targets of #203 and should be what we can actually do in the near future.
I believe during the steps proposed the internal APIs and design can be changed. However, at that time we're moving from one workable software to another better, which will encourage the whole community to make Engula a great serverless storage engine, instead of lack of interest for uncompleted fragments.
What do you think @huachaohuang @w41ter-l @zojw?
Beta Was this translation helpful? Give feedback.
All reactions