Create new, modern, backwards-incompatible RPC interface.

[nathan-at-least]

Is your feature request related to a problem? Please describe.

The problems with the current RPC interface are many:

• There's no toolchain support for ensuring online help docs are accurate, and I occasionally run into outright incorrect help strings.
• There's no static schema for messages, and in fact, zcashd dynamically uses different shapes or semantics based on runtime or configuration parameters.
• The API is a chimera of older Bitcoin-inherited methods and newer Zcash-specific methods.
• The API serves many distinct use cases (e.g. block explorers, zcashd wallet users, etc…) with functionality sectioning done "by hand" in the zcashd implementation, rather than via distinct well defined use-case-specific interfaces.
• The overall API has many style/behavior inconsistencies:
  • some boolean parameters are JSON true/false, others are non-zero ints.
  • some Zcash amounts are ZEC, some are ZAT.
  • some Zcash amounts are strings, some are JSON decimals, some are JSON integers. (Also there's no distinction in JSON between number types.)
  • naming conventions are mixed, such as in capitalization styles.
  • inconsistent with positional vs named parameters.
• There is no explicit versioning, so it's difficult to ensure a given client interoperates correctly with a specific zcashd version.
• There is no consistent forwards/backwards compatibility functionality in the RPC layer, so clients must often be modified directly as soon as zcashd is updated with no graceful transition.
• There's no API help in the client without connecting to a running zcashd. (Sometimes this is inconvenient when I just want the usage help for a call and need to wait ~2 min for zcashd to boot up.)
• JSON-RPC does not support server->client notifications or streaming, so API calls like z_getoperationstatus are polling kludges.
• JSON does not specify representations, so implementations may be inconsistent in representation (e.g. f32 for JSON numbers, or fixed size ints of the wrong size, etc…)
• JSON cannot encode binary data such as serialized transactions, so server & client must both do an extra serialization layer (e.g. hex)

Describe the solution you'd like

Rather than iteratively extend the existing interface, which leaves a long tail of "straddling-the-fence" flaws of older-vs-newer APIs, what if we simply start a new RPC interface ex-nihilo that uses good modern clean design and mark the existing RPC as "maintenance-only mode".

There could be a variety of good options, but I'm most familiar with gRPC:

• good tooling support across languages and platforms.
• explicit schema
• built-in standardized forward/backward compatibility (e.g. tolerance of unrecognized fields)
• explicit runtime representation (e.g. uint32 vs sint64 vs double)
• support for server->client notifications via streaming responses, e.g. rpc GetWalletEvents () returns (stream WalletEvent) which streams events for things like new transaction request, transaction constructed, transaction confirmed at depth D, transaction expired, etc… This would replace the polling-style design of z_getoperationstatus.
• distinct use cases could be distinct gRPC services, eg ZcashdBlockExplorerBackend vs ZcashdWallet. (I'm not familiar with multiplexing multiple gRPC services on a single connection; worst case, just have separate gRPC service ports for different interfaces.) These services could then be configured by their service name, ie zcash.conf might have grpc_services_wallet = true.

For zcashd implementation, we could add a rust gRPC server, perhaps async within its own thread(s), then communicate between rust<->C++ with a message-passing design so that the rust code avoids the complexity of touching other C++ synchronization primitives.

One advantage of doing this is that the direct gRPC server could be a reusable crate with the "zcashd provider" as only one backend, thus enabling us to create standalone testmode services, zebra backends, etc…

I'm not too clear on the rust<->C++ interface or how convenient message passing could be or what tech would work well there.

Alternatives you've considered

There may be alternatives to gRPC that have similar advantages: schema, precise scalar types, tooling, language SDKs, …

Just deprecate all of zcashd in favor of zebra. We'd still need to potentially define RPC interfaces for the components we want.

Another option: for a new headless wallet, rather than reifying the current zcashd wallet design, we could rely on a librustzcash -> lightwalletd -> zcashd stack and just bundle those together in a single convenient package (e.g. zcashd-wallet vs zcashd-no-wallet) for different use cases.

Additional context

This is just a brainstorm proposal, and we'll be considering a wide variety of ways to reduce zcashd technical debt, foremost of which is deciding if we should simply wind-down support for some entire use-case classes.