Skip to content

Commit

Permalink
rpc: Upgrade WebSocketClient to support full client functionality (#646)
Browse files Browse the repository at this point in the history 
* Expand WebSocket functionality

This commit is multi-faceted. It:

1. Drastically simplifies much of the subscription-related
   functionality.
2. Expands the WebSocketClient's capabilities to be able to handle other
   types of requests other than just subscriptions (i.e. it implements the
   Client trait).

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* The http-client feature also needs the tracing lib

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Make client mutable in example

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Update CHANGELOG

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Update ADR-008 to reflect recent changes

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Fix minor bug in response handling

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Update tests to use new WebSocketClient functionality

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Fix/ignore clippy warnings

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Fix documentation for rpc crate

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Fix broken link in crate docs

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Update RPC repo docs

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Refactor to remove ChannelTx mutability

ChannelTx::send now no longer needs to be mutable because the underlying
channel has no mutability requirement. This leads to viral changes
throughout the interfaces and clients (positive ones).

ChannelTx::send also doesn't need to be async, because it relies on an
unbounded channel. This also has some viral implications for other
methods throughout the RPC client. If, in future, we want to support
bounded channels, then we can consider making it async again. But we
probably shouldn't make it async if we don't need it to be.

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Client no longer needs to be mutable

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Use explicit borrow_mut to avoid having to obtain subscriptions map twice

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Rename method for clarity

Signed-off-by: Thane Thomson <connect@thanethomson.com>
  • Loading branch information
@thanethomson
thanethomson committed Nov 26, 2020
1 parent b618169 commit 5105efe
Show file tree
Hide file tree
Showing 38 changed files with 938 additions and 1,202 deletions.
4 changes: 4 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,12 @@
### IMPROVEMENTS:

- `[light-client]` Only require Tokio when `rpc-client` feature is enabled ([#425])
- `[rpc]` The `WebSocketClient` now adds support for all remaining RPC requests
by way of implementing the `Client` trait ([#646])

[#425]: https://github.com/informalsystems/tendermint-rs/issues/425
[#646]: https://github.com/informalsystems/tendermint-rs/pull/646


## v0.17.0-rc3

Expand Down
101 changes: 73 additions & 28 deletions docs/architecture/adr-008-event-subscription.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -218,7 +218,7 @@ WebSocket connection to provide subscription functionality (the
#[async_trait]
pub trait SubscriptionClient {
/// `/subscribe`: subscribe to receive events produced by the given query.
async fn subscribe(&mut self, query: String) -> Result<Subscription>;
async fn subscribe(&mut self, query: Query) -> Result<Subscription>;
}
```

Expand Down Expand Up @@ -323,19 +323,23 @@ pub enum EventType {
ValidatorSetUpdates,
}

pub struct Condition {
key: String,
op: Operation,
}

pub enum Operation {
Eq(Operand),
Lt(Operand),
Lte(Operand),
Gt(Operand),
Gte(Operand),
Contains(Operand),
Exists,
// A condition specifies a key (first parameter) and, depending on the
// operation, an value which is an operand of some kind.
pub enum Condition {
// Equals
Eq(String, Operand),
// Less than
Lt(String, Operand),
// Less than or equal to
Lte(String, Operand),
// Greater than
Gt(String, Operand),
// Greater than or equal to
Gte(String, Operand),
// Contains (to check if a key contains a certain sub-string)
Contains(String, String),
// Exists (to check if a key exists)
Exists(String),
}

// According to https://docs.tendermint.com/master/rpc/#/Websocket/subscribe,
Expand All @@ -346,7 +350,8 @@ pub enum Operation {
// operand types to the `Operand` enum, as this would improve ergonomics.
pub enum Operand {
String(String),
Integer(i64),
Signed(i64),
Unsigned(u64),
Float(f64),
Date(chrono::Date),
DateTime(chrono::DateTime),
Expand All @@ -361,7 +366,7 @@ track of all of the queries relating to a particular client.
```rust
pub struct SubscriptionRouter {
// A map of queries -> (map of subscription IDs -> result event tx channels)
subscriptions: HashMap<Query, HashMap<SubscriptionId, ChannelTx<Result<Event>>>>,
subscriptions: HashMap<String, HashMap<String, SubscriptionTx>>,
}
```

Expand All @@ -372,21 +377,61 @@ server [drops subscription IDs from events][tendermint-2949], which is likely if
we want to conform more strictly to the [JSON-RPC standard for
notifications][jsonrpc-notifications].

#### Two-Phase Subscribe/Unsubscribe
### Handling Mixed Events and Responses

Since a full client needs to implement both the `Client` and
`SubscriptionClient` traits, for certain transports (like a WebSocket
connection) we could end up receiving a mixture of events from subscriptions
and responses to RPC requests. To disambiguate these different types of
incoming messages, a simple mechanism is proposed for the
`WebSocketClientDriver` that keeps track of pending requests and only matures
them once it receives its corresponding response.

Due to the fact that a WebSocket connection lacks request/response semantics,
when managing multiple subscriptions from a single client we need to implement a
**two-phase subscription creation/removal process**:
```rust
pub struct WebSocketClientDriver {
// ...

1. An outgoing, but unconfirmed, subscribe/unsubscribe request is tracked.
2. The subscribe/unsubscribe request is confirmed or cancelled by a response
from the remote WebSocket server.
// Commands we've received but have not yet completed, indexed by their ID.
// A Terminate command is executed immediately.
pending_commands: HashMap<String, DriverCommand>,
}

The need for this two-phase subscribe/unsubscribe process is more clearly
illustrated in the following sequence diagram:
// The different types of requests that the WebSocketClient can send to its
// driver.
//
// Each of SubscribeCommand, UnsubscribeCommand and SimpleRequestCommand keep
// a response channel that allows for the driver to send a response later on
// when it receives a relevant one.
enum DriverCommand {
// Initiate a subscription request.
Subscribe(SubscribeCommand),
// Initiate an unsubscribe request.
Unsubscribe(UnsubscribeCommand),
// For non-subscription-related requests.
SimpleRequest(SimpleRequestCommand),
Terminate,
}
```

![RPC client two-phase
subscribe/unsubscribe](./assets/rpc-client-two-phase-subscribe.png)
IDs of outgoing requests are randomly generated [UUIDv4] strings.

The logic here is as follows:

1. A call is made to `WebSocketClient::subscribe` or
`WebSocketClient::perform`.
2. The client sends the relevant `DriverCommand` to its driver via its internal
communication channel.
3. The driver receives the command, sends the relevant simple or subscription
request, and keeps track of the command in its `pending_commands` member
along with its ID. This allows the driver to continue handling outgoing
requests and incoming responses in the meantime.
4. If the driver receives a JSON-RPC message whose ID corresponds to an ID in
its `pending_commands` member, it assumes that response is relevant to that
command and sends back to the original caller by way of a channel stored in
one of the `SubscribeCommand`, `UnsubscribeCommand` or
`SimpleRequestCommand` structs. Failures are also communicated through this
same mechanism.
5. The pending command is evicted from the `pending_commands` member.

## Status

Expand Down Expand Up @@ -433,4 +478,4 @@ None
[futures-stream-mod]: https://docs.rs/futures/*/futures/stream/index.html
[tendermint-2949]: https://github.com/tendermint/tendermint/issues/2949
[jsonrpc-notifications]: https://www.jsonrpc.org/specification#notification

[UUIDv4]: https://en.wikipedia.org/wiki/Universally_unique_identifier#Version_4_(random)
49 changes: 12 additions & 37 deletions docs/architecture/assets/rpc-client-erd.graphml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -94,10 +94,10 @@
<node id="n7">
<data key="d6">
<y:ShapeNode>
<y:Geometry height="56.0" width="209.0" x="390.5" y="182.0"/>
<y:Geometry height="56.0" width="155.0" x="417.5" y="182.0"/>
<y:Fill color="#DFC0FF" transparent="false"/>
<y:BorderStyle color="#000000" raised="false" type="dashed" width="1.0"/>
<y:NodeLabel alignment="center" autoSizePolicy="content" fontFamily="Helvetica" fontSize="14" fontStyle="plain" hasBackgroundColor="false" hasLineColor="false" height="18.0" horizontalTextPosition="center" iconTextGap="4" modelName="custom" textColor="#000000" verticalTextPosition="bottom" visible="true" width="189.2060546875" x="9.89697265625" xml:space="preserve" y="19.0">TwoPhaseSubscriptionRouter<y:LabelModel><y:SmartNodeLabelModel distance="4.0"/></y:LabelModel><y:ModelParameter><y:SmartNodeLabelModelParameter labelRatioX="0.0" labelRatioY="0.0" nodeRatioX="0.0" nodeRatioY="0.0" offsetX="0.0" offsetY="0.0" upX="0.0" upY="-1.0"/></y:ModelParameter></y:NodeLabel>
<y:NodeLabel alignment="center" autoSizePolicy="content" fontFamily="Helvetica" fontSize="14" fontStyle="plain" hasBackgroundColor="false" hasLineColor="false" height="18.0" horizontalTextPosition="center" iconTextGap="4" modelName="custom" textColor="#000000" verticalTextPosition="bottom" visible="true" width="123.0615234375" x="15.96923828125" xml:space="preserve" y="19.0">SubscriptionRouter<y:LabelModel><y:SmartNodeLabelModel distance="4.0"/></y:LabelModel><y:ModelParameter><y:SmartNodeLabelModelParameter labelRatioX="0.0" labelRatioY="0.0" nodeRatioX="0.0" nodeRatioY="0.0" offsetX="0.0" offsetY="0.0" upX="0.0" upY="-1.0"/></y:ModelParameter></y:NodeLabel>
<y:Shape type="roundrectangle"/>
</y:ShapeNode>
</data>
Expand Down Expand Up @@ -216,26 +216,15 @@ Detail)<y:LabelModel><y:SmartNodeLabelModel distance="4.0"/></y:LabelModel><y:Mo
<node id="n13">
<data key="d6">
<y:ShapeNode>
<y:Geometry height="56.0" width="109.0" x="429.5" y="607.0"/>
<y:Fill color="#BBDCFC" transparent="false"/>
<y:BorderStyle color="#000000" raised="false" type="line" width="1.0"/>
<y:NodeLabel alignment="center" autoSizePolicy="content" fontFamily="Helvetica" fontSize="14" fontStyle="plain" hasBackgroundColor="false" hasLineColor="false" height="18.0" horizontalTextPosition="center" iconTextGap="4" modelName="custom" textColor="#000000" verticalTextPosition="bottom" visible="true" width="65.482421875" x="21.7587890625" xml:space="preserve" y="19.0">Operation<y:LabelModel><y:SmartNodeLabelModel distance="4.0"/></y:LabelModel><y:ModelParameter><y:SmartNodeLabelModelParameter labelRatioX="0.0" labelRatioY="0.0" nodeRatioX="0.0" nodeRatioY="0.0" offsetX="0.0" offsetY="0.0" upX="0.0" upY="-1.0"/></y:ModelParameter></y:NodeLabel>
<y:Shape type="roundrectangle"/>
</y:ShapeNode>
</data>
</node>
<node id="n14">
<data key="d6">
<y:ShapeNode>
<y:Geometry height="56.0" width="109.0" x="616.75" y="607.0"/>
<y:Geometry height="56.0" width="109.0" x="440.5" y="607.0"/>
<y:Fill color="#BBDCFC" transparent="false"/>
<y:BorderStyle color="#000000" raised="false" type="line" width="1.0"/>
<y:NodeLabel alignment="center" autoSizePolicy="content" fontFamily="Helvetica" fontSize="14" fontStyle="plain" hasBackgroundColor="false" hasLineColor="false" height="18.0" horizontalTextPosition="center" iconTextGap="4" modelName="custom" textColor="#000000" verticalTextPosition="bottom" visible="true" width="58.482421875" x="25.2587890625" xml:space="preserve" y="19.0">Operand<y:LabelModel><y:SmartNodeLabelModel distance="4.0"/></y:LabelModel><y:ModelParameter><y:SmartNodeLabelModelParameter labelRatioX="0.0" labelRatioY="0.0" nodeRatioX="0.0" nodeRatioY="0.0" offsetX="0.0" offsetY="0.0" upX="0.0" upY="-1.0"/></y:ModelParameter></y:NodeLabel>
<y:Shape type="roundrectangle"/>
</y:ShapeNode>
</data>
</node>
<node id="n15">
<node id="n14">
<data key="d6">
<y:ShapeNode>
<y:Geometry height="56.0" width="155.0" x="1198.0" y="182.0"/>
Expand Down Expand Up @@ -305,16 +294,14 @@ Detail)<y:LabelModel><y:SmartNodeLabelModel distance="4.0"/></y:LabelModel><y:Mo
</y:PolyLineEdge>
</data>
</edge>
<edge id="e5" source="n3" target="n7">
<edge id="e5" source="n8" target="n7">
<data key="d10">
<y:PolyLineEdge>
<y:Path sx="0.0" sy="0.0" tx="0.0" ty="0.0">
<y:Point x="613.0" y="210.0"/>
</y:Path>
<y:Path sx="-110.009765625" sy="0.0" tx="0.0" ty="0.0"/>
<y:LineStyle color="#000000" type="line" width="1.0"/>
<y:Arrows source="diamond" target="crows_foot_one"/>
<y:EdgeLabel alignment="center" configuration="AutoFlippingLabel" distance="2.0" fontFamily="Helvetica" fontSize="12" fontStyle="plain" hasBackgroundColor="false" hasLineColor="false" height="28.0" horizontalTextPosition="center" iconTextGap="4" modelName="custom" preferredPlacement="anywhere" ratio="0.5" textColor="#000000" verticalTextPosition="bottom" visible="true" width="31.33984375" x="-123.43225215534017" xml:space="preserve" y="-51.67025936280061">Has/
Uses<y:LabelModel><y:SmartEdgeLabelModel autoRotationEnabled="false" defaultAngle="0.0" defaultDistance="10.0"/></y:LabelModel><y:ModelParameter><y:SmartEdgeLabelModelParameter angle="6.283185307179586" distance="33.36407661646361" distanceToCenter="true" position="left" ratio="0.4515565075001525" segment="0"/></y:ModelParameter><y:PreferredPlacementDescriptor angle="0.0" angleOffsetOnRightSide="0" angleReference="absolute" angleRotationOnRightSide="co" distance="-1.0" frozen="true" placement="anywhere" side="anywhere" sideReference="relative_to_edge_flow"/></y:EdgeLabel>
<y:EdgeLabel alignment="center" configuration="AutoFlippingLabel" distance="2.0" fontFamily="Helvetica" fontSize="12" fontStyle="plain" hasBackgroundColor="false" hasLineColor="false" height="28.0" horizontalTextPosition="center" iconTextGap="4" modelName="custom" preferredPlacement="anywhere" ratio="0.5" textColor="#000000" verticalTextPosition="bottom" visible="true" width="31.33984375" x="-65.30288833566169" xml:space="preserve" y="8.364076616463592">Has/
Uses<y:LabelModel><y:SmartEdgeLabelModel autoRotationEnabled="false" defaultAngle="0.0" defaultDistance="10.0"/></y:LabelModel><y:ModelParameter><y:SmartEdgeLabelModelParameter angle="6.283185307179586" distance="8.36407661646358" distanceToCenter="false" position="left" ratio="0.5954113682210318" segment="-1"/></y:ModelParameter><y:PreferredPlacementDescriptor angle="0.0" angleOffsetOnRightSide="0" angleReference="absolute" angleRotationOnRightSide="co" distance="-1.0" frozen="true" placement="anywhere" side="anywhere" sideReference="relative_to_edge_flow"/></y:EdgeLabel>
<y:BendStyle smoothed="false"/>
</y:PolyLineEdge>
</data>
Expand Down Expand Up @@ -385,26 +372,15 @@ Uses<y:LabelModel><y:SmartEdgeLabelModel autoRotationEnabled="false" defaultAngl
<edge id="e11" source="n12" target="n13">
<data key="d10">
<y:PolyLineEdge>
<y:Path sx="0.0" sy="0.0" tx="0.0" ty="0.0"/>
<y:LineStyle color="#000000" type="line" width="1.0"/>
<y:Arrows source="diamond" target="crows_foot_one"/>
<y:EdgeLabel alignment="center" configuration="AutoFlippingLabel" distance="2.0" fontFamily="Helvetica" fontSize="12" fontStyle="plain" hasBackgroundColor="false" hasLineColor="false" height="16.0" horizontalTextPosition="center" iconTextGap="4" modelName="custom" preferredPlacement="anywhere" ratio="0.5" textColor="#000000" verticalTextPosition="bottom" visible="true" width="25.33984375" x="2.04168701171875" xml:space="preserve" y="-22.0">Has<y:LabelModel><y:SmartEdgeLabelModel autoRotationEnabled="false" defaultAngle="0.0" defaultDistance="10.0"/></y:LabelModel><y:ModelParameter><y:SmartEdgeLabelModelParameter angle="6.283185307179586" distance="14.0" distanceToCenter="true" position="left" ratio="-2.919921875" segment="-1"/></y:ModelParameter><y:PreferredPlacementDescriptor angle="0.0" angleOffsetOnRightSide="0" angleReference="absolute" angleRotationOnRightSide="co" distance="-1.0" frozen="true" placement="anywhere" side="anywhere" sideReference="relative_to_edge_flow"/></y:EdgeLabel>
<y:BendStyle smoothed="false"/>
</y:PolyLineEdge>
</data>
</edge>
<edge id="e12" source="n13" target="n14">
<data key="d10">
<y:PolyLineEdge>
<y:Path sx="0.0" sy="0.0" tx="0.0" ty="0.0"/>
<y:Path sx="24.75" sy="0.0" tx="0.0" ty="0.0"/>
<y:LineStyle color="#000000" type="line" width="1.0"/>
<y:Arrows source="diamond" target="crows_foot_one_optional"/>
<y:EdgeLabel alignment="center" configuration="AutoFlippingLabel" distance="2.0" fontFamily="Helvetica" fontSize="12" fontStyle="plain" hasBackgroundColor="false" hasLineColor="false" height="16.0" horizontalTextPosition="center" iconTextGap="4" modelName="custom" preferredPlacement="anywhere" ratio="0.5" textColor="#000000" verticalTextPosition="bottom" visible="true" width="25.33984375" x="1.04168701171875" xml:space="preserve" y="-22.0">Has<y:LabelModel><y:SmartEdgeLabelModel autoRotationEnabled="false" defaultAngle="0.0" defaultDistance="10.0"/></y:LabelModel><y:ModelParameter><y:SmartEdgeLabelModelParameter angle="6.283185307179586" distance="14.0" distanceToCenter="true" position="left" ratio="-3.919921875" segment="-1"/></y:ModelParameter><y:PreferredPlacementDescriptor angle="0.0" angleOffsetOnRightSide="0" angleReference="absolute" angleRotationOnRightSide="co" distance="-1.0" frozen="true" placement="anywhere" side="anywhere" sideReference="relative_to_edge_flow"/></y:EdgeLabel>
<y:EdgeLabel alignment="center" configuration="AutoFlippingLabel" distance="2.0" fontFamily="Helvetica" fontSize="12" fontStyle="plain" hasBackgroundColor="false" hasLineColor="false" height="16.0" horizontalTextPosition="center" iconTextGap="4" modelName="custom" preferredPlacement="anywhere" ratio="0.5" textColor="#000000" verticalTextPosition="bottom" visible="true" width="25.33984375" x="1.0521240234375" xml:space="preserve" y="-22.0">Has<y:LabelModel><y:SmartEdgeLabelModel autoRotationEnabled="false" defaultAngle="0.0" defaultDistance="10.0"/></y:LabelModel><y:ModelParameter><y:SmartEdgeLabelModelParameter angle="6.283185307179586" distance="14.0" distanceToCenter="true" position="left" ratio="-3.919921875" segment="-1"/></y:ModelParameter><y:PreferredPlacementDescriptor angle="0.0" angleOffsetOnRightSide="0" angleReference="absolute" angleRotationOnRightSide="co" distance="-1.0" frozen="true" placement="anywhere" side="anywhere" sideReference="relative_to_edge_flow"/></y:EdgeLabel>
<y:BendStyle smoothed="false"/>
</y:PolyLineEdge>
</data>
</edge>
<edge id="e13" source="n5" target="n15">
<edge id="e12" source="n5" target="n14">
<data key="d10">
<y:PolyLineEdge>
<y:Path sx="0.0" sy="-11.0" tx="0.0" ty="0.0">
Expand All @@ -417,8 +393,7 @@ Uses<y:LabelModel><y:SmartEdgeLabelModel autoRotationEnabled="false" defaultAngl
</y:PolyLineEdge>
</data>
</edge>
<edge id="e14" source="n3" target="n0">
<data key="d9"/>
<edge id="e13" source="n3" target="n0">
<data key="d10">
<y:PolyLineEdge>
<y:Path sx="0.0" sy="0.0" tx="0.0" ty="0.0">
Expand Down
Binary file modified docs/architecture/assets/rpc-client-erd.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
10 changes: 9 additions & 1 deletion rpc/Cargo.toml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,15 @@ all-features = true

[features]
default = []
http-client = [ "async-trait", "futures", "http", "hyper", "tokio/fs", "tokio/macros" ]
http-client = [
"async-trait",
"futures",
"http",
"hyper",
"tokio/fs",
"tokio/macros",
"tracing"
]
secp256k1 = [ "tendermint/secp256k1" ]
websocket-client = [
"async-trait",
Expand Down
18 changes: 8 additions & 10 deletions rpc/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,25 +21,23 @@ This crate optionally provides access to different types of RPC client
functionality and different client transports based on which features you
select when using it.

Two features are provided at present:
Two features are provided at present.

* `http-client` - Provides `HttpClient`, which is a basic RPC client that
interacts with remote Tendermint nodes via **JSON-RPC over HTTP**. This
client does not provide `Event` subscription functionality. See the
client does not provide `Event` subscription functionality. See the
[Tendermint RPC] for more details.
* `websocket-client` - Provides `WebSocketClient`, which currently only
provides `Event` subscription functionality over a WebSocket connection. See
the [`/subscribe` endpoint] in the Tendermint RPC for more details. This
client does not yet provide access to the RPC methods provided by the
`Client` trait (this is planned for a future release).
* `websocket-client` - Provides `WebSocketClient`, which provides full
client functionality, including general RPC functionality (such as that
provided by `HttpClient`) as well as `Event` subscription
functionality.

### Mock Clients

Mock clients are included when either of the `http-client` or
`websocket-client` features are enabled to aid in testing. This includes
`MockClient`, which only implements `Client` (no subscription
functionality), and `MockSubscriptionClient`, which helps you simulate
subscriptions to events being generated by a Tendermint node.
`MockClient`, which implements both `Client` and `SubscriptionClient`
traits.

### Related

Expand Down
Loading

0 comments on commit 5105efe

Please sign in to comment.