Roadmap v0.2 - Microunit

[huachaohuang]

Non-goals:

• Data persistence
• Leader election and failover
• Resource isolation and management

Tasks:

• Design document
• Commands
  • Start a node
  • Bootstrap a universe
  • Join a node to a universe

[huachaohuang]

We need to decide the command line interface. The Command Line Interface Guidelines is awesome. I prefer the noun verb style, like docker container create.

So for example, we can have the following subcommands:

• engula node to operate on a specific node
• engula unit to operate on a specific unit

[tisonkun]

We need to decide the command line interface. The Command Line Interface Guidelines is awesome. I prefer the noun verb style, like docker container create.

So for example, we can have the following subcommands:

• engula node to operate on a specific node
• engula unit to operate on a specific unit

The guidelines look awesome! Shall we name it engula? I can see a engula crate (what does it for) so far and am afraid of the word gets override too much.

But it's OK we implement a ctl of microunit and disptach engula node to microunit node or muctl node as a facade.

[tisonkun]

#57 (comment) @huachaohuang I think we don't need a consistent store for v0.2. We can store all data in memory for now.

Does it mean we deliver a single-node version microunit in v0.2? Or we can use a preconfigured leader election service and thus all nodes know where to get the data / register themselves?

[tisonkun]

@huachaohuang Node should provide HTTP API ...

I'd like to know whether we will make use of a certain HTTP server library or write from what Rust stdlib provides?

[huachaohuang]

We need to decide the command line interface. The Command Line Interface Guidelines is awesome. I prefer the noun verb style, like docker container create.
So for example, we can have the following subcommands:

• engula node to operate on a specific node
• engula unit to operate on a specific unit

The guidelines look awesome! Shall we name it engula? I can see a engula crate (what does it for) so far and am afraid of the word gets override too much.

But it's OK we implement a ctl of microunit and disptach engula node to microunit node or muctl node as a facade.

Actually, the binary does not belong to microunit, it is a tool for the whole Engula project. Let's move the discussion to #63.

[huachaohuang]

#57 (comment) @huachaohuang I think we don't need a consistent store for v0.2. We can store all data in memory for now.

Does it mean we deliver a single-node version microunit in v0.2? Or we can use a preconfigured leader election service and thus all nodes know where to get the data / register themselves?

We can deliver a multi-node version in v0.2. But maybe we simply start one control unit and let it control everything. So we don't need to worry about leader election and consensus in v0.2.

[huachaohuang]

@huachaohuang Node should provide HTTP API ...

I'd like to know whether we will make use of a certain HTTP server library or write from what Rust stdlib provides?

Well, I am also considering this question. Things we need to take into considerations include:

• Node servers are not performance-critical.
• It is OK to use different web frameworks for different parts of the project.
• We may embed a web dashboard on nodes and control units, so we may need a more powerful web framework in the future.

But for v0.2, I think we can start with a simple library. axum is a lightweight framework and developed by the tokio team, I think it is worth a try.

[tisonkun]

We can deliver a multi-node version in v0.2. But maybe we simply start one control unit and let it control everything. So we don't need to worry about leader election and consensus in v0.2.

Got it. That's the way "preconfigured".

Still I have some concern on bootstraping. Will check #62 later.

[PsiACE]

axum is a lightweight framework and developed by the tokio team, I think it is worth a try.

As far as some practices are concerned, axum is not the best choice at this stage. But maybe, as you say, it's worth trying.

[PsiACE]

We can deliver a multi-node version in v0.2. But maybe we simply start one control unit and let it control everything. So we don't need to worry about leader election and consensus in v0.2.

This is certainly feasible, but perhaps a flexible model needs to be provided to facilitate future expansion to some consensus or non-consensus

[huachaohuang]

axum is a lightweight framework and developed by the tokio team, I think it is worth a try.

As far as some practices are concerned, axum is not the best choice at this stage. But maybe, as you say, it's worth trying.

Can you share more details?

[huachaohuang]

We can deliver a multi-node version in v0.2. But maybe we simply start one control unit and let it control everything. So we don't need to worry about leader election and consensus in v0.2.

This is certainly feasible, but perhaps a flexible model needs to be provided to facilitate future expansion to some consensus or non-consensus

You can review #62 for some simple descriptions.

[tisonkun]

@huachaohuang about the communication topic, I'd like to know what method you'd like to use for quorum members talk to each other in consensus group? Still HTTP?

I can see we don't care about data compression too much in networking, but gRPC also provides flow control, authentication, error handling, streaming communication, and async communication. Dropping the ecosystem means we should develop our own or adopt a new one. It's OK but please take in consideration. And yes I like the idea you can access API by a HTTP client or just curl.

EDIT: Notice these comments.

• It is OK to use different web frameworks for different parts of the project.
• We may embed a web dashboard on nodes and control units, so we may need a more powerful web framework in the future.

Fine. Hopefully we can find such a framework; otherwise it's a huge stuff to cover although we can start with little functionality.

[huachaohuang]

@tisonkun I haven't decided that part. I think we can try to learn from other projects like etcd and k8s.

[huachaohuang]

I think one solution is to use HTTP API and gRPC together. We can use gRPC for inner communication and expose some HTTP API when necessary. This is the way etcd did and k8s seems also tried to do, you can check this and this.

[huachaohuang]

I am also reading a series of posts explaining how to multiplex HTTP and gRPC services in the same port. The posts are very educational on the Rust ecosystem, you can check them here.

[zojw]

CRDB also use grpc-gateway to export grpc service to its admin web UI(ps: also multiplex 3 ports into 1), but it seems it doesn't use json mapping, they choose protobuf over http to call grpc gateway https://github.com/cockroachdb/cockroach/blob/9ba8499e80a3234da094e061827f1c23d9d33341/pkg/ui/workspaces/cluster-ui/src/api/fetchData.ts#L74 and result as all-in protobuf solution in production communicate(both http and grpc)..

but manual curl test also can use json + json content-type, because gateway support using different marshaler for differnt content-type https://github.com/cockroachdb/cockroach/blob/9b7fdf014b598bf004f24c7d0367d1c04859076e/pkg/server/server.go#L1937-L1939, maybe we can do the same thing in rust.

[huachaohuang]

CRDB also use grpc-gateway to export grpc service to its admin web UI(ps: also multiplex 3 ports into 1), but it seems it doesn't use json mapping, they choose protobuf over http to call grpc gateway https://github.com/cockroachdb/cockroach/blob/9ba8499e80a3234da094e061827f1c23d9d33341/pkg/ui/workspaces/cluster-ui/src/api/fetchData.ts#L74 and result as all-in protobuf solution in production communicate(both http and grpc)..

but manual curl test also can use json + json content-type, because gateway support using different marshaler for differnt content-type https://github.com/cockroachdb/cockroach/blob/9b7fdf014b598bf004f24c7d0367d1c04859076e/pkg/server/server.go#L1937-L1939, maybe we can do the same thing in rust.

Yes, I think mapping HTTP + JSON to gRPC + Protobuf makes sense since HTTP + JSON APIs are not performance-critical.

[huachaohuang]

Seems Rust doesn't provide a built-in way to "fork" processes like https://man7.org/linux/man-pages/man2/fork.2.html. I think we can use threads in v0.2 instead. But we need to figure out how to fork and move units to cgroups sooner or later. Maybe we can learn from firecracker.

[huachaohuang]

Share some ideas so far:

• engula node start <ADDR> --init to start a node and bootstrap a universe
• engula node start <ADDR> --join <URL> to start a node and join a universe

A control unit is created on the first node of a universe. A control unit runs an HTTP server with the following APIs:

• /nodes
  • GET: list all nodes in the universe
  • POST: join a node (with a NodeSpec) to the universe
• /members
  • GET: list all members of the control plane (all control units in the universe)
  • POST: join a control unit to the control plane

Because the address of control units may change from time to time. We let the control plane contacts all nodes periodically to keep them up to date and pull statuses from all nodes as well. Each node can redirect control requests to the control plane so that users don't need to distinguish the addresses of nodes and control units explicitly. For example, users can use the URL of any node in the universe to join the universe without knowing where are the control units.

[huachaohuang]

We need to distinguish two kinds of APIs: one towards a node and another towards the whole universe. For example, we can list units of a node or the whole universe. Requests towards a single node should be handled by the node. Requests towards the whole universe should be handled by the control plane. So if we want to let each node in a universe redirect requests towards the universe to the control plane, the node should be able to distinguish these two kinds of requests.

I propose the following routes:

• mount requests toward a single node or unit to /v1
• mount requests toward the whole universe to /v1/universe

So if a node receives requests against /v1/universe, it will redirect the client to the control plane. In this case, users can send universe requests to any node and any control unit.

[tisonkun]

@huachaohuang according to #54 (reply in thread) this issue is possibly reconsidered. Especially whether to have a node concept.

[huachaohuang]

@tisonkun Yes, let's leave it for now before we are clear about how we should proceed.

[huachaohuang]

We are not going to work on microunit in v0.2 as discussed. Let's close it and postpone related thigns to v0.3.

[tisonkun]

Related discussions on zulip:

@huachaohuang We have developed a lot of designs, concepts, and abstractions recently. We apply the principle of public designs and discussions (engula has no internal documents or discussion channels at all), which means that we talk about a lot of early ideas that are subject to change. While these ideas give more opportunities for the community to involve, they also confuse the community if they change rapidly. So in order to converge the work we have done so far, I suggest that we cut a simpler version 0.2 with the following features at the end of Dec 2021:

• A design document that explains existing concepts
• A hash engine that can use different kinds of kernels
• A kernel, journal, and storage abstraction
• A memory kernel that integrates the memory journal and memory storage
• A file kernel that integrates the file journal and file storage
• A grpc kernel that integrates the grpc journal and grpc storage
• A blog post that announces the v0.2 release, demonstrates the hash engine and explains some future plans.

Specifically, I suggest cutting microunit off at v0.2 since we don't have a clear design about it for now. Then we can focus on resource management and deployment in v0.3.

@tisonkun Today I consider this topic also. I agree with you that we don't have to deliver multiple concepts about deployment before we have a clear design and so does microunit. However, we should still deliver a basic usable executable in v0.2 so that early users can explore the software and inspire ideas.