Absorb rust-abci

[tac0turtle]

Discuss absorbing the rust-abci repo as its own crate within this repo.

[tomtau]

that could be good for unifying some of the amino/protobuf types

[ebuchman]

seems like a good idea - we should add something about this to the ADR-001: #28

[davebryson]

It feels like pulling ABCI into this repo would just complicate things a bit. Unless this repo is slowly moving from just a client to a full blown tendermint core implementation...which would be really cool btw...

[tomtau]

Yeah, it can be a bit complicated at the moment, as the protobuf generated definitions versus the nice hand-written RPC types don't quite match up

[tarcieri]

For what it's worth, I'd prefer generated RPC types for use in the KMS. I think the current handwritten ones are a bit overcomplicated, and using protos means they won't diverge.

[tomtau]

I think that could work -- one would need to modify https://github.com/tendermint/rust-abci/blob/develop/build.rs

so that the generated ABCI types derive serde deserialisers etc.
https://docs.rs/protobuf-codegen/2.8.1/protobuf_codegen/struct.Customize.html#structfield.serde_derive

One thing to consider would be separating out the ABCI types into a dedicated rust-abci-types crate (versus keeping it with the ABCI server implementation)

[ebuchman]

Ideally we can use proto as the canonical definition for all the types, and re-use them both for ABCI and the RPC.

This won't 100% work for everything yet as there are still some things that depend on amino encoding with prefixes in the RPC (namely pubkeys, but I think that's it). But we plan to transition even amino interfaces to proper proto3 wellknowntype Any soon, so this should be fully possible. Definitely something to work towards.

Unless this repo is slowly moving from just a client to a full blown tendermint core implementation...which would be really cool btw...

That's the plan!

[tac0turtle]

Would like to revisit this issue. Should a new ADR be written or an amendment be made to adr-001?

[liamsi]

@marbar3778 an adr or even a draft PR instead (or in parallel with that adr) would be awesome.

[ebuchman]

Can we clarify the relationship between rust-abci and abci-rs? Has the former absorbed the latter in some way?

There's also Nomic's abci2. My understanding is that it's not using async/await, but unlike the other impls, it does allow abci methods to return asynchronously, which would allow concurrent processing at the app layer, which is something we ultimately would want. This use case might also be taken care of by introducing a DeliverBlock instead, which would just ship all the txs at once, so the app could still process them concurrently (contemplated, and proposed in ABCIx, but not currently in tendermint) .

If we don't have DeliverBlock, the design space seems to be mostly around whether we use async/await Rust, and whether the ABCI responses can be received asynchronously. If I understand correct, we basically have the following matrix now:

	Synchronous ABCI	Async ABCI
Synchronous Rust	rust-abci	abci2
AsyncAwait Rust	abci-rs	N/A

Ideally we'd be able to have one definitive abci crate we could all use, built up in layers to potentially satisfy all quadrants of this matrix as necessary. That said, I'm not sure there's a real use case for Synchronous ABCI, since Tendermint is already sending DeliverTx messages asynchronously in order (it doesnt wait to receive the result of one before sending the next). So it actually seems more like getting the right half of the matrix. Then its seems there's two options:

1. wrapping the abci2 with an async/await interface so it can be a drop in replacement for abci-rs
2. updating abci-rs to support asynchronous abci responses

It's not clear to me if (2) would also require a non-async wrapper or if users should just be responsible for calling some block_on themselves if they want synchronous behaviour.

cc @mappum here as well. I basically asked the same thing in tendermint/rust-abci#61 (comment)

[ebuchman]

On a related note, if it may take some time to still figure this out, we could start by just merging the abci proto work in here, while we figure out how to unify abci-rs and abci2. Then at least we would maintain the abci protos here which are the primary thing that need updating and external crates could experiment with connection semantics without having to worry about the protos. Downside of this might be difficulty of testing without connection code, so we may have to depend on one of the connection implementations anyways

[tomtau]

@ebuchman

Can we clarify the relationship between rust-abci and abci-rs? Has the former absorbed the latter in some way?

rust-abci was using tokio 0.1. In an unreleased new version (it failed to release due to @zramsay's expired crates.io token #489 (comment) ) with dependency upgrades: tendermint/rust-abci@af2f34a -- it partially moved to tokio 0.2 and async/await internally.

If I understand correct, we basically have the following matrix now.

I don't think that understanding is correct (or it depends on what is meant by "Synchronous ABCI") -- in both "rust-abci" and "abci-rs" requests and replies are still read/written asynchronously ("Synchronous ABCI" was "rust-abci" prior to version 0.5.3 -- which is when the discussion started); "abci2" uses synchronous I/O (within a single connection, it'll block -- it can probably be fixed by setting set_nonblocking on the sockets), but allows asynchronous processing.

The high-level differences are:

• "rust-abci" has one application trait corresponding to all ABCI connections and one internal application mutex (which limits application concurrency)
• "abci-rs" has three "async" traits corresponding to each ABCI connection and requires applications implementing the traits to be Send + Sync (so they themselves can handle concurrency however they like)
• "abci2" has no "application trait" and expects applications to handle ABCI processing via pairs of standard library queue communication primitives (so they themselves can handle concurrency however they like)

I suggested "abci-rs" for absorption, because it's more feature-complete (e.g. handling of unix domain sockets or logical checks that can prevent runtime gotchas) and has more tests. Using traits can reduce some boilerplate, e.g. a default implementation for Echo.
Both "abci-rs" and "abci2" are "breaking" / require changes to existing applications of "rust-abci". In that sense, IMHO what's important is that if a breaking change is adopted that it won't break again:

In the case of tendermint-rs, the ideal situation will be that the ABCI crate user will be able to seamlessly switch from "ABCI server" (with Tendermint implemented in Go talking to it) to the fully fledged Tendermint implementation in Rust (once ready; compiled to a single binary, just like Go applications can do with Tendermint now) -- with that goal in mind and tendermint-rs architecture plans, you can comment on #510 what makes most sense.

[ebuchman]

Ok thanks. I'm off grid for the next few days but will follow up next week.

[ethanfrey]

This use case might also be taken care of by introducing a DeliverBlock instead, which would just ship all the txs at once, so the app could still process them concurrently (contemplated, and proposed in ABCIx, but not currently in tendermint) .

@ebuchman I think this is the way to go. Can't we expose the external ABCI interface as before, then combine them in an abci-rs3 crate and expose a DeliverBlock interface to the app, which can figure out how to process all that data. When the two items are in the same process (or at least the same machine), the worry about latency of moving say 200kB (a huge block from what I have seen) seems excessive. And inside DeliverBlock the code could call the tx sequentially, do pipelining, or even do some async/await (couldn't we just start one Executor for the block, use async/await internally, and return all results when it is finished)?

[mappum]

I don't think that understanding is correct (or it depends on what is meant by "Synchronous ABCI") -- in both "rust-abci" and "abci-rs" requests and replies are still read/written asynchronously ("Synchronous ABCI" was "rust-abci" prior to version 0.5.3 -- which is when the discussion started)

Here "synchronous ABCI" means not being able to read multiple incoming ABCI messages before responding. abci-rs does synchronously process messages from an ABCI connection, consider its main connection processing loop:

while let Ok(request) = decode(&mut stream).await {
    match request {
        Some(request) => {
            let response = inner.process(request).await;

            if let Err(err) = encode(response, &mut stream).await {
                error!(message = "Error while writing to stream", %err, ?peer_addr);
            }
        }
        None => debug!(message = "Received empty request", ?peer_addr),
    }
}

The while loop cannot move to the next iteration and call decode until a response has been computed and encode has been called, meaning we are only processing one message at a time per connection.

The async IO doesn't really improve anything here, except that we could choose to use a single thread rather than 3 (4 with state-sync) and save a small amount of memory (at the cost of giving up parallelism for the encoding/decoding work). Actually, in this case async is hurting because each response gets a Pin<Box<dyn Future<Response>>>, which means every message will have a heap allocation, not to mention a dynamic dispatch every time the future gets polled. This cost will be significant when shooting for 10's of thousands of txs per second.

This use case might also be taken care of by introducing a DeliverBlock instead, which would just ship all the txs at once, so the app could still process them concurrently (contemplated, and proposed in ABCIx, but not currently in tendermint) .

I want to point out that concurrency for CheckTx is also important (e.g. maintaining N parallel mempools and doing as much tx validation work as early as possible across all CPU cores), and this doesn't address that.

[tarcieri]

Actually, in this case async is hurting because each response gets a Pin<Box<dyn Future<Response>>>, which means every message will have a heap allocation, not to mention a dynamic dispatch every time the future gets polled. This cost will be significant when shooting for 10's of thousands of txs per second.

If I'm understanding correctly, these costs are both due to the use of async-trait, and will eventually go away when Rust eventually adds native support for async traits (but that might be awhile, since it seems to be dependent on GATs)

[tomtau]

Here "synchronous ABCI" means not being able to read multiple incoming ABCI messages before responding. abci-rs does synchronously process messages from an ABCI connection, consider its main connection processing loop:

ok, and the current pending abci-rs still has this: https://github.com/informalsystems/tendermint-rs/pull/489/files#diff-70126217abd1cb67e8ff8d0aef5825f4R146

One may be able to eliminate that, but as the responses need to be written in order, one will still want to have a communication queue (e.g. mpsc channel from the standard library) in there and separate out the response writing.
I think Decision in https://github.com/informalsystems/tendermint-rs/pull/510/files#diff-8560f0a125ba5a5011ace9b97f504ab5R33 could be expanded with more details

This cost will be significant when shooting for 10's of thousands of txs per second.

It'll be good to measure that and expand on that in decision consequences in: https://github.com/informalsystems/tendermint-rs/pull/510/files#diff-8560f0a125ba5a5011ace9b97f504ab5R50

There are a lot of moving pieces though, e.g. a particular runtime / its configuration and async-trait native support. IMHO the performance of the current abci server implementation is a bit of a secondary concern:

1. there are other overheads -- e.g. the state machine high level logic runtime checking -- they can possibly be feature-guarded / optional if the crate user requires a higher performance (and can handle the correctness, e.g. resetting consensus state on TM abci connection dropping, themselves)
2. a lot of parts will evolve and could potentially be optimized
3. ultimately, the abci crate user will want to switch to a fully fledge TM node implementation in Rust instead of using the abci server

As I mentioned above, it depends on the tendermint-rs architecture plans -- if the "async-trait"-based applications fit well with that, I'm not sure -- #510 will ideally reflect / comment on that

[yihuang]

Actually, in this case async is hurting because each response gets a Pin<Box<dyn Future<Response>>>, which means every message will have a heap allocation, not to mention a dynamic dispatch every time the future gets polled. This cost will be significant when shooting for 10's of thousands of txs per second.

If I'm understanding correctly, these costs are both due to the use of async-trait, and will eventually go away when Rust eventually adds native support for async traits (but that might be awhile, since it seems to be dependent on GATs)

Maybe good to try some zero-cost async trait solution, like this and this

[ebuchman]

So what do we think about using abci2 (non-async-await rust, asynchronous abci) as the core and layering an async-await interface and other application types (like from abci-rs) on top of it?

Mentioned in #510 (comment) and I think this is basically what Tony is describing as option 1 in tendermint/rust-abci#61 (comment).

I think this is also what tungstenite does (websockets lib) with the async-tungstenite wrapper

[tomtau]

It should be ok. coming back to

ultimately, the abci crate user will want to switch to a fully fledge TM node implementation in Rust instead of using the abci server

so it doesn't matter that much whether the abci server is based off std network stack like abci2 or tokio... what matters more is how well the end-user traits will fit with the rest of tendermint-rs

[ebuchman]

Right, though it will be some time before we have a full node in Rust (probably minimum 12-18 months).

I'm going to bump the milestone on this as we're currently focused on IBC and getting a release out that's compatible with tendermint v0.34.

[devashishdxt]

I don't think that understanding is correct (or it depends on what is meant by "Synchronous ABCI") -- in both "rust-abci" and "abci-rs" requests and replies are still read/written asynchronously ("Synchronous ABCI" was "rust-abci" prior to version 0.5.3 -- which is when the discussion started)

Here "synchronous ABCI" means not being able to read multiple incoming ABCI messages before responding. abci-rs does synchronously process messages from an ABCI connection, consider its main connection processing loop:

while let Ok(request) = decode(&mut stream).await {
    match request {
        Some(request) => {
            let response = inner.process(request).await;

            if let Err(err) = encode(response, &mut stream).await {
                error!(message = "Error while writing to stream", %err, ?peer_addr);
            }
        }
        None => debug!(message = "Received empty request", ?peer_addr),
    }
}

The while loop cannot move to the next iteration and call decode until a response has been computed and encode has been called, meaning we are only processing one message at a time per connection.

The async IO doesn't really improve anything here, except that we could choose to use a single thread rather than 3 (4 with state-sync) and save a small amount of memory (at the cost of giving up parallelism for the encoding/decoding work). Actually, in this case async is hurting because each response gets a Pin<Box<dyn Future<Response>>>, which means every message will have a heap allocation, not to mention a dynamic dispatch every time the future gets polled. This cost will be significant when shooting for 10's of thousands of txs per second.

This use case might also be taken care of by introducing a DeliverBlock instead, which would just ship all the txs at once, so the app could still process them concurrently (contemplated, and proposed in ABCIx, but not currently in tendermint) .

I want to point out that concurrency for CheckTx is also important (e.g. maintaining N parallel mempools and doing as much tx validation work as early as possible across all CPU cores), and this doesn't address that.

I've updated the code to concurrently execute CheckTx requests in abci-rs. It now spawns a new async task for each request. Here is the code for that. Also, here is the test to check if requests are actually executed concurrently and responses are received in order.

[tac0turtle]