Skip to content

IPC Flatbuffers API

cryptocode edited this page Jan 21, 2020 · 39 revisions

Nano API: Flatbuffers over IPC

This is a design proposal for establishing new internal and external APIs in the Nano node software.

Flatbuffers is proposed instead of protobuf since the former uses less memory and is much faster. Like protobuf, Flatbuffers has pros and cons compared to using a schemaless format like JSON.

  • Statically typed schema. This has a strict definition, but allows for decent schema evolution. Because it's statically typed, the message type is not serialized. This means the IPC protocol must first serialize the message type. With JSON, the message type is known through the endpoints and discriminator fields (in our case, the action field)
  • Much faster and memory efficient than json (https://google.github.io/flatbuffers/flatbuffers_benchmarks.html)
  • Streaming is possible
  • Flatbuffers (like protobuf) must compile the schema to generate (de)serializers, whereas JSON doesn't need a compilation step.

Scope

  • RPC 2.0 backbone
    • REST gateway
    • Separate /api/v2 endpoint
    • Actions in endpoint names to avoid envelopes, e.g. /api/v2/BlockCount
  • Service API
  • Authorization

Out of scope:

  • Schema definitions for RPC 2.0

Definitions

An internal API is a set of messages communicated between node processes, such as the node and a wallet process. An external API is a set of messages meant for use by integrators. These should have extensive documentation and have a higher threshold for allowing breaking changes. Internal APIs can typically break between versions, since all executables are updated on each release. A process can use a mix of internal and public APIs.

Relation to RPC 2.0

The proposed IPC changes would serve as the backbone for the planned RPC 2.0 refactoring. Combined with the REST/JSON gateway explained later, these changes allow clients to use proper JSON (as opposed to the ptree variation currently used, which is non-standard, slow and memory hungry)

RPC 2.0 queries and responses will be specified as flatbuffer tables, and with the proposed REST/JSON gateway, messages can be translated automatically to and from JSON.

Required IPC enhancements

Encoding

This proposals adds two new encodings: 0x03 - flatbuffers for IPC clients and 0x04 - flatbuffers_json for REST clients.

Connection model

The node is the IPC server, whereas external processes are IPC clients. Both server and client initiated requests should be possible through full-duplex messaging and subscriptions.

REST/JSON gateway

Flatbuffer messages over IPC enable efficient messaging between node-related processes. However, integrators usually prefer dealing with REST calls exchanging JSON messages.

With this proposal, the node will provide a REST endpoint which will automatically translate between JSON requests and flatbuffer messages used internally. Integrators interested in better performance and less memory use can opt to use the Nano IPC protocol directly.

Future enhancement: Processes and the IPC service model

The node currently starts child processes in an ad-hoc manner in the node and wallet entry points. This leads to code duplication, and is quite fragile. For instance, if the node is killed, child processes linger. When the node restarts, it's unable to start child processes because they already have ports open. Using an IPC service model, it should be possible for child processes to automatically shut down if the host process is gone. It could also be configured to keep running and reconnect to the node when its restarted.

The node will contain a service manager configured using a TOML file that looks something like this:

config-services.toml

[[service]]
name=rpc
path=nano-rpc-node

[[service]]
name=powserver
path=/usr/bin/nano-pow-server

# Communicate over the rest gateway instead of IPC
type=rest

# If the node doesn't respond in a timely fashion, shut down the process
orphaned=shutdown

An "IPC Service" is a process adhering to the Nano IPC/Flatbuffers protocol. We can also evaluate if the REST gateway can be used, making it easier to add services. This would require a config flag, like what is done for the powserver in the example above.

Implementation details

Error handling

The action handlers use C++ exceptions to communicate errors, by throwing nano::error objects. The rationale for throwing nano::error is that it already knows how to convert between common error types used in the node (error_code, boost::error_code) and it's possible to set custom messages.

Any error thrown from action handlers will be handled centrally to produce a common error response.

For instance, throw nano::error(nano::error_rpc::invalid_balance); will produce something like this when using the JSON gateway:

{
    "message_type": "Error",
    "message": {
        "code": 24,
        "message": "Invalid balance number"
    }
}

A generic message can also be thrown, in which case the error code is 1, e.g. throw nano::error("Invalid block type");

Building flatbuffer responses

Flatbuffers has a very fast encoding scheme, but this comes at the cost of having to build object graphs in a specific way. For IPC handlers where performance isn't critical, Flatbuffers offers a more convenient Object API.

Object API alternative

The generated object API produces classes with a "T" suffix.

Sample response code:

    nano_api::KeepAliveT ka;
    ka.sent = uint64_t (nano::milliseconds_since_epoch ());
    ...

    auto thing = std::make_unique<nano_api::ThingT> ();
    thing->name = "name of the thing";
    ka.things.push_back (std::move (thing));

   create_response (nano_api::KeepAlive::Pack (*fbb, &ka));

The JSON mapping would be something like this:

{
    "message_type": "KeepAlive",
    "message": {
        "sent": 1574533664334,
        "things": [
            {
                "name": "name of the thing"
            }
        ]
    }
}

Builder alternative

Using builders is the most efficient approach in terms of speed and memory consumption.

The basic rule is that anything referenced by a builder must be created before the builder. In the example below, the TopRepresentativeBuilder references a nano address. This address string must be created before creating the builder.

auto addr = fbb->CreateString ("nano_1...");
nano_api::TopRepresentativeBuilder builder (*fbb);
builder.add_address (addr);

create_response (builder);

All action handlers must finalize the root builder (the top-level object we send as the response) by calling create_response. This will wrap the response in a "response envelope" which contains the message type and a timestamp.

Permissions

See https://github.com/cryptocode/notes/wiki/IPC-Authorization

Status

Done

  • Integrate the Flatbuffers library into the node
  • New IPC encoding type
  • NodeJS test client framework to validate interoperability (scope is for internal testing only)
  • Auto-generate source files from schema in CMake
    • A dependency between FBS and generated header files is established using a custom command. When an FBS file is changed, recompiling the node will automatically regenerate the header files.
  • Implement an IPC handler that understands the flatbuffer encoding
    • Use asio's scatter/gather API to avoid first moving the header+body to a vector
  • An IPC action handler with support for both Flatbuffers builders (most efficient) and Flatbuffers Object API (more convenient)
  • TOML config for Flatbuffers knobs, such as a message verification flag.
  • Submit PR to Flatbuffers to fix empty-vector JSON serialization: https://github.com/google/flatbuffers/pull/5653
  • RPC SSL support restored (broke some time during v19)
  • Make IPC server shutdown/close gracefully (like was done with sockets) instead of risk failing with exceptions.
  • Define overall message structure
    • Must be suitable for IPC, REST and possibly WebSocket clients
    • Basic proposal is to use an envelope/message approach where the envelope will minimally contain the message type. The WebSocket envelope would include a correlation ID as well. Request envelopes is a common extension point, allowing out-of-band info such as API keys (as HTTP headers are not available for IPC clients). Response envelopes may contain other out-of-band information, such as timing.
    • Error handling: The envelope should have a consistent layout; an error message type, then an error object with code and message
  • REST gateway
    • Ideally we want a robust embedded web server which supports endpoints. The first iteration will reuse the HTTP server we have for RPC 1.0. However, persistent connections should be implemented as the current design of disconnecting after every request is inefficient and consumes file descriptors (separate PR)
    • Using namespaces in Flatbuffers IDL means messages names such as "accounts_History", which is not ideal. On option is to support injection of message type based on endpoint path. For instance api/v2/accounts/History
  • Add multi-writer queue and strands to the IPC server
  • Full duplex IPC messaging. Enables a model where the node is always the server, but the node can still initiate requests.
    • Useful for the wallet split, which will need pending notifications (polling is too slow)
  • Multi-writer and strands on ipc client as well
  • CMake on Windows need some work regarding auto-generation of source files
  • Basic services

Not started

  • Documentation
    • All actions should have well-defined error responses. Can we generate from Flatbuffer source IDLs?
    • Some sort of programmer's guide for node coders (Flatbuffers docs are lacking in some areas)
You can’t perform that action at this time.