Five Bells Ledger API specification

[mDuo13]

(Formerly known as the "Common Ledger API")

This PR has been updated to a third iteration, following the work earlier in this PR which continued from PR #90.. In keeping with our current thinking, a common ledger API may not be needed in the long run, but a formalization of the Five Bells Ledger API currently in use is helpful.

Thus, this PR now more closely reflects the existing implementation of the five-bells-ledger, including some of the deprecated and odd functionality such as using "actual" crypto-conditions instead of just hashlocks, supporting cancellation conditions, having multi-credit, multi-debit with a proposed stage, not separating the ID and client ID, lacking URL fields for some operations, etc.

[emschwartz]

Overall I think this is really good. Major simplification of the five-bells-ledger API. I'm excited to see this rolled out!

[emschwartz]

Remove "cancel" mentions (there are a couple others spread throughout the doc that should be taken out since there are no cancellation conditions here anymore)

[emschwartz]

The ledger MUST...

[emschwartz]

Typo: "can server"

[emschwartz]

I liked when this mentioned some of the other things you might want to implement if you're building a ledger from this (e.g. creating and updating accounts)

[emschwartz]

Minor: the sentence that starts "The Common..." should be separated from the line before because it's explaining a pretty different point

[emschwartz]

I think the ledger MUST authenticate WebSocket connections

[mDuo13]

As agreed below, there's a valid use case for unauthenticated WebSocket connections.

[emschwartz]

I don't understand why you would allow unauthenticated users to subscribe to anything

[mDuo13]

I can imagine a ledger whose purpose is to share transfers publicly, in which case you wouldn't need to authenticate anyone for the WebSocket API.

[emschwartz]

I think we should change the event names to match the transfer statuses

[mDuo13]

I agree.

[emschwartz]

Looks like params is sometimes an array and sometimes an object -- is that allowed?

[mDuo13]

In JSONRPC 2.0, params is a "structured" value -- in other words, yes. The examples in their spec contain both.

That said, it's probably wise to make this an object so it's forward compatible and similar to the Subscribe to Account call.

[emschwartz]

Should be message.send to be consistent with the tense of the others

[dncohen]

Why restrict the prefix to just a currency code? Why not allow https://ledger.example.com/USD.bitstamp/transfers and https://ledger.example.com/USD.gatehub/transfers, for example?

[justmoon]

Good catch, the docs should be clear that the prefix can be anything. (As long as it's a valid URI/URI.)

Examples:

/USD.Bitstamp/
/USD/Bitstamp/
/Bond/Samsung/USD/10yr/
/funnymoney/
/€/
/©a$#/

These are all perfectly valid.

[mDuo13]

Yeah, that's what's intended, it's just not clear from how it's written now. (I'll fix later.)

[clark800]

We are currently discussing a proposal to replace fingerprints with hostnames, which makes certificate changes easier.

[clark800]

In our implementation we changed "prepared_at" to "created_at" so that the same name makes sense for transfers with and without execution conditions.

[adrianhopebailie]

By this logic we should change the API method to "Create Transfer" for consistency

[sappenin]

Should the API maintain both values, allowing for a clear distinction between created_at and a prepared_at?

Per RFC1, the term "Prepared" demands an escrowed payment, which is not accurate in an optimistic-mode transfer. However, if we use the term created_at, but in some instances (e.g., Universal Transfer) we really mean "when this was prepared", it it could lead to confusion. Likewise, choosing the value prepared_at when we really mean "when this was created" (e.g., Optimistic Transfer), seems equally imprecise.

Along those lines, RFC1 talks about a Proposed state. Would it make sense to allow transfers to be created in the Proposed state initially, and then (potentially at some point later) moved into the Prepared state, but only if its a conditional transfer?

That would allow optimistic transfers to have a timeline like the following:

"timeline": {
  "created_at": "2015-06-16T00:00:01.000Z",
  "executed_at": "2015-06-16T00:00:02.000Z"
}

whereas conditional transfers could have a timeline like this:

"timeline": {
  "created_at": "2015-06-16T00:00:01.000Z",
  "prepared_at": "2015-06-16T00:00:01.000Z",
  "executed_at": "2015-06-16T00:00:02.000Z"
}

[mDuo13]

The current five-bells-ledger implementation has a proposed state that matches this description. However, one of the goals of the Common Ledger API (when we started on this project) was to simplify the proposed state out of existence.

I think I'm going to go ahead with changing the remaining state to "created" though.

[clark800]

Not sure if this will cover all the desired use cases, but in our implementation we just have an optional notification_url stored in each account and a notification is POSTed to this URL when there is activity on that account. This makes it really simple because there are no subscriptions to manage. The downside vs websockets is that the recipient has to have a server, but you could also make a separate microservice that provides the websockets if you want to keep the ledger minimal.

[mDuo13]

Personally I think it would be awesome if the ledger API didn't have to include a WebSocket component at all.

It's possible to do transfer notifications by serving Get Transfer through an http response that doesn't finish until the transfer is in a final state, but I'm not sure how you implement that. (I've seen it done in web-based games before -- the GET just sits there pending until the server has something to respond with.)

But that still doesn't resolve the question of how clients and connectors would communicate and authenticate each other for quoting.

[clark800]

My comment above answers that, no?

[emschwartz]

In an earlier iteration of the five-bells-ledger we were using RESThooks instead of Websockets. Switching to websockets was the single biggest speed boost we saw in end-to-end interledger payments. I wouldn't object to having the notification_url as an option too (only downside being another feature for implementers to include), but I would strongly oppose replacing websockets with a slower notification method (not counting WebRTC, MQTT or another real-time messaging protocol).

[clark800]

The performance improvement was due to removing the database accesses that were required for robust notifications. What I'm suggesting is non-robust HTTP POST notifications, which is what our implementation uses and it is faster than the five-bells-ledger with websockets.

[earizon]

I was playing with an alternative solution to websocket based on gRPC. First thing I needed was to abstract all the "websocket subscription and message stuff" as a bidirectional asynchronous stream.

An early *proto defining the service is available at:
https://github.com/everis-innolab/ILP_gRPC_API/blob/master/ilp_ledger_subscriptions.proto

A comparative about a different alternative solutions for bidirectional async channels:

\	websocket+JSON/RPC 2.0	Client request + Client pooling	protobuf + gRPC	web-hooks
Notes	- Current five-bells implementation - Once the ws connection is stablished client and ledger become peers.	- Simplest solution. - Messages from client to ledger are sent in HTTP requests. Messages from ledger to client are pulled during periodic pooling requests.	- bidirectional async stream are created with: streamN(stream msg) returns (stream msg)	both ledger and client must provide a callback URL for peers messages/events
web-browser support	YES	YES	"WORK IN PROGRESS" github.com/grpc/grpc-experiments/issues/159 github.com/grpc-ecosystem/grpc-gateway	NO Android support
iOS support	YES	YES	YES	YES
Standard	YES (IETF RFC 6455)	NO	"Open-Source / IETF Draft"	NO
Works on 3G, behind proxies/firewalls,...	"SOMETIMES"	YES	YES/NO/SOMETIMES?	NO (both peers require IP visibility to expose REST web-hooks)
automatic mapping object to/from network data"	PARTIAL and IN SOME LANGUAJES	NO	YES	NO
auto-documented	NO	NO	YES. Using *proto files ( *proto can also be used as ref doc. for non-gRPC implementations )	NO
cross-language support	NO	NO	YES	NO
speed	GOOD	SLOW	VERY GOOD	UNDEFINED
"deprecated" support (backward compatibility)	MANUAL	MANUAL	YES	MANUAL
PREFERED FOR	CONNECTORS & SERVER APPS	SORT-LIVE CONNECTIONS WHERE MOBILE CLIENTS AND RANDOM NETWORK SETUPS ARE EXPECTED. (""WALLETS"")	CONNECTORS & SERVER APPS	CONNECTORS & SERVER APPS IF WEBSOCKETS AND gRPC ARE NOT AVAILABLE.

I'm also thinking whether it could be convenient to split this API in services for light clients ("wallets" that just connect time to time to the ledger posting new prepared Transfers and checking payments), server applications ("e-comerce" with permanent tcp/ip connections waiting for payments and fulfilling transfer) and connectors. Also many simultaneous "wallet" connections to ledger could be expected in some setups while only a very limited set of ilp-connectors and server apps ("e-commerce"). If I'm not wrong there is no distinction amongst them right now. The API treats them all equally.

[sappenin]

Great write-up of different channels @earizon! I've been doing similar research as I implement a Java ledger conforming to this Common Ledger API, and have been wanting to experiment with gRPC as an alternative to Websockets. I think once browsers fully support everything they need to (here's one such proposal) then Websockets will probably make less sense in the browser (IMHO).

Your proposal for API separation is an interesting one. In the short-term, it seems like the Websocket channel would be the minimum thing to support so that Browsers can connect, with gRPC maybe being an optional choice? Or do you have a different idea?

[earizon]

@sappenin Yes I think Websocket can be considered required all (web/non-web) clients. But I'm not 100% sure that it will work in all network environments due to HTTP proxies. See for example problems with whatsapp web using wss behind proxies @ google. A fall-back to client pooling could be useful too. It's clearly a suboptimal solution in terms of performance, but for clients that just want to connect "time to time" to make some random payment it could be useful. There are also some other problems when using permanent permanent tcp/ip connections and the number of clients start to grow.

[adrianhopebailie]

@sappenin and @earizon I think the core Java stuff is stabalizing to the point that we should definitely try a second adaptor implementation.

Could we try implementing a gRPC based interface between the client and your ledger?

You'd implement the org.interledger.ilp.ledger.LedgerAdaptor interface and then should be able to simply instantiate a Client with your adaptor instead of the REST adaptor.

[earizon]

@adrianhopebailie great news!

Can anyone confirm next claims (since it's not 100% clear to me from the spec). Using gRPC syntax I was thinking on:

Ledger acting as gRPC server, ledger-client initiating the gRPC connection:

    rpc subscribeToAccountNotifications(spec.ILPAccount[] account_list ) returns ( stream spec.ILPLocalTransfer ); // call started by ledger-client
    rpc subscribeToTransferNotifications(spec.ILPTransfer transfer ) returns ( stream spec.ILPLocalTransfer );

    rpc setupBidirectionalAsyncMessagesChannel              (stream spec.ILPMessage)       returns ( stream spec.ILPMessage );

ledger-client acting as gRPC server, ledger connecting as gRPC client:

    rpc listenForAccountSubcriptions(stream spec.ILPAccount ) returns ( stream spec.ILPLocalTransfer ); // call started by ledger. 
    rpc listenForTransferSubcriptions(stream spec.ILPtransfer ) returns ( stream spec.ILPLocalTransfer );

    rpc setupBidirectionalAsyncMessagesChannel  (stream spec.ILPMessage) returns ( stream spec.ILPMessage );

In common language:

• It's expected that the ledger will send transfer notifications to any ledger-client directly subscribed to a transfer or indirectly subscribed to any transfer affecting the account (as creditor or debitor).

• It's expected that any ledger or ilp-connector-ledger-client will send messages to its peer, to emit or receive events (those events are used to propagate routing information). ( Is there any use-case when an non-ilp-connector will need to listen for message events? )

Edit:
There were some obvious mistakes in the previous gRPC draft/interface. ("Fixed"). I'm considering two possibilities for two different scenarios: It's possible to place the gRPC server in the ledger or in the ledger-client. Placing it on the ledger is the "more intuitive" option, but it could be the case that for some reason that's not possible. gRPC makes it easy to "mirrow" the API inverting the role of the gRPC client & server.

[clark800]

Why is this needed?

[mDuo13]

answered above.

[clark800]

I think this should just be "alice". This is coupling to a lot of unnecessary things. Like if the server switches from http to https it looks like this will cause problems. Or if a new version of the code switches the endpoint name from "accounts" to "account".

[adrianhopebailie]

The type of this member should be URI and a value relative to the API endpoint should be allowed.

[clark800]

No reason?

[MatthewPhinney]

I'd also be interested to know why this member should be a URI.

[adrianhopebailie]

The ledger is a resource identified by a URI.
The account is a resource that is relative to the ledger.

This is exactly the use case that URIs are designed for.

There is very well defined and deterministic logic for resolving absolute URIs from relative URIs given a base (the ledger) to solve for exactly the issues @clark800 raises such as changing the protocol.

If we are saying that a ledger MUST have an account endpoint URL or an account URL Template (RFC6570) then it makes sense for the accounts to be identified either by the variable that is substituted into the template or by a relative URL that is resolved against the account endpoint URL.

On the other hand if the only normative requirement is for a ledger to have a base URL then the account MUST be identified by a URL relative to the ledger URL /accounts/alice or an absolute URL.

[emschwartz]

I think having accounts identified by a relative URL is better than expecting all ledgers to have an account endpoint. Some ledgers might want to have an account aliasing or internal sub-accounting feature, in which case I might want you to send money to /aliases/ceb8a630-2d9d-4ce2-ad81-72a862659ae8 instead of my real account /accounts/1234567890.

[emschwartz]

How large a message should the ledger support sending?

[mDuo13]

This question is tricky. The size of the message data depends on the encoding. Any limit I could think of, upper or lower, would be totally arbitrary. Maybe we should decide on a minimum limit in bytes and/or characters of UTF-8, chosen based on being able to convey any reasonable quote / ILP header data we expect to need?

[adrianhopebailie]

Suggest using the following URL as reference: http://www.iso.org/iso/home/standards/currency_codes.htm

[sappenin]

Can you clarify the intent of client_id?

For example, for a single ILP Payment with transfers on Ledger1 and Ledger2, is it intended that the client_id will be the same in both ledgers?

I'm trying to figure out if this value is meant to correlate ILP payments across modules (i.e., like an ILP transaction id), or if this field has a different purpose.

[emschwartz]

Good question. This field is only to ensure uniqueness within a single ledger. It is not meant as a way to correlate transfers across ledgers and actually should not be used that way.

The reason to use an id like this within one ledger is to prevent people from accidentally submitting the same transfer twice. If the transfer with that ID has already been submitted, POSTing it again will cause an error.

The client_id should not be kept the same across ledgers because of the potential for id squatting. If you are a connector that always uses the same id for the outgoing transfer as the incoming one, I can prevent you from executing the destination transfer if I know the source transfer's id (for example if I am the sender).

[adrianhopebailie]

The reason to use an id like this within one ledger is to prevent people from accidentally submitting the same transfer twice. If the transfer with that ID has already been submitted, POSTing it again will cause an error.

The client_id should not be kept the same across ledgers because of the potential for id squatting. If you are a connector that always uses the same id for the outgoing transfer as the incoming one, I can prevent you from executing the destination transfer if I know the source transfer's id (for example if I am the sender).

There are some normative requirements there that we should document in the spec.

[sappenin]

What do you guys think about having the ledger generate this identifier value instead of the client?

There are quite a few requirements for this identifier that should probably be the burden of the ledger since the ledger is the one mandating them -- e.g., unique across the ledger; conforming to a particular pattern that the ledger can use as a PK in the ledger's database, etc. Plus, the point about preventing id-squatting seems like something that should be the responsibility of the ledger as opposed to the client (who might mess it up or be unable to enforce the requirement).

For my proposal to work, though, the API would need to do something like the following to facilitate the creation of a transfer, which makes it a bit more chatty, but IMHO makes the API more robust:

1. POST the transfer payload (minus client_id) to /transfers
2. Receive the created transfer as a response, with a server-supplied transfer id.
3. When ready, execute a PUT to /transfers/{transfer_id} (or possibly a PATCH) to update the state of the transfer to indicate to the ledger that the transfer is ready to be acted upon (e.g., update the state from "proposed" to "prepared", or something like that).

While the second call requiring clients to overtly trigger the processing of a transfer is somewhat of a negative, this two-phase creation has the added benefit of allowing clients to operate upon the server-side state of a transfer using possibly multiple remote calls before actually having the ledger do anything with it.

[emschwartz]

This seems unnecessarily complicated to me. A trivial way for clients to avoid id squatting is to either use a completely random UUID or one that is deterministically generated from some detail about the interledger payment (as is supposed to happen in the connector here). Full HTTP round trips are relatively slow and we should avoid adding them wherever possible.

Whether or not the ledger uses the client_id as the database primary key, it shouldn't be difficult for ledgers using any standard database to enforce uniqueness on that value. A previous version of this spec said ledgers could optionally enforce client_id uniqueness on a per-account basis, but it seemed like having more options would just make implementing this spec more complicated.

[sappenin]

What are some scenarios where the decimal_digits attribute of an Asset would diverge from the metadata's precision and scale attributes?

[justmoon]

The description for this field could be improved. decimal_digits refers to the conventionally displayed number of decimal places for a given asset. It's purely a hint for display purposes and is ignored by everyone but human-facing amount formatters. (By contrast, precision and scale affect that actual available precision on a ledger.)

For example, a USD ledger might use a precision of 15 and a scale of 6. So, it can represent amounts like USD 10.123456. However, US dollars are typically shown with a precision of cents, so decimal_digits would be 2. How a client exactly renders this is up to the client, but it might show any precision beyond the decimal_digits in a deemphasized way, e.g. 10.12(3456)

[diminator]

Great job, thanks @mDuo13

My main question would be: how to enable this API spec for blockchain type or append-only ledgers (BTC, ETH, BDB, ...)

They have different design paradigms:

• no account information, merely public keys
• transaction IDs are hashes, no UUIDs (agreed that squatting could be a potential problem)
• When append-only there is only a notion of POST, not PUT. eg. PUT /transfer_*

[diminator]

From a blockchain perspective, the account resource belongs to the wallet, not the ledger itself.
Blockchain or append-only ledgers only rely on pubkeys.
I'm not saying we should throw this out of the door, but it seems to me to get this API working for both legacy payment ledgers and blockchain ledgers such as BTC, ETH, BDB, etc there would be the need of a proxy API relaying between the chain and the wallet.

Any thoughts?

[mDuo13]

I agree that blockchains would need a proxy server to host a Common Ledger API, which would sit between the client application (wallet) and the blockchain server / p2p network. I'm not sure there's a way around that.

The "account information" returned can be quite minimal. In a blockchain case, "Get Account" could use the pubkey as the name and then just provide the known balance that can be spent by that key pair.

Maybe we need a special value for balances that are unknown in order to support this. I'm not really sure.

[diminator]

Ok, we can split up the proxy requirements to somewhere else?
Maybe those will be more implementation specific - hence not RFC-ed?

[mDuo13]

I don't see how you'd split it off. The whole REST API would probably be served by some "proxy" server (not as in HTTP proxy).

[earizon]

I interpret that "client" means in this context the software connecting to the ledger while "server" is the ILP-compliant ledger. I think Using "client" and "server" could cause trouble. Explicitly speaking about "wallets" (the local GUI connecting to the ledger), "ledger" and "connector" will be safer. For example in a websocket stream the difference between the client and server vanishes once the bidirectional message channel is created.

[mDuo13]

I agree on using "ledger" instead of "server."

However, I think the term "wallet" tends to be rather confusing and misleading. People use the term "wallet" to mean "client application", "key pair", or "web service" somewhat interchangeably, which is a problem. I'm going to stick with "client application" for now.

[earizon]

Wouldn't rejection_webhook be more 'standard' (https://en.wikipedia.org/wiki/Webhook) ?

I don't have a global picture about how transfers can be rejected using such URL. Is there any RFCs explaining it?

As I understand "cancelling" and "rejecting" are two different terms in ILP:

• "cancelling" means creating a cancellation fulfilment matching a cancellation crypto-condition.

• rejecting means that at some point the execution fulfilment didn't match the execution condition or there was a timeout.

DOUBT: How this 'transfer_rejection' match in the ILP protocol. Does it means the (sender originator?) ledger must fulfil the cancellation fulfilment ? In such case transfer_cancellation (or cancellation_webhook) will be more appropiate.
(Actually I didn't find anything related in the RFCs, only the code in the five-bells implementation)

[mDuo13]

This API doesn't have any concept of "cancelling" in the ILP sense.

Rejecting transfers uses the "Reject Transfer" method also defined in this document. The receiver of a transfer can reject a conditional transfer if the execution condition hasn't been fulfilled yet.

[earizon]

In the five-bells-ledger and ilp-plugin-bells there is support for cancellation condition and cancellation fulfillments (https://github.com/interledgerjs/ilp-plugin-bells/blob/master/src/lib/plugin.js, https://github.com/interledgerjs/five-bells-ledger/blob/master/src/models/transfers.js). I don't see any mention about them in this rfc or the 0004 either.

Has cancellation been removed, or is it missing on the rfcs ?

If it hasn't, the final state of a cancelled transfer is also "rejected" or must it be "cancelled"?

[justmoon]

Has cancellation been removed, or is it missing on the rfcs ?

Cancellation and a few other five-bells-ledger features, such as multi-debit/credit turned out not to be needed for ILP, so they aren't part of the ILP ledger adapter API (IL-RFC-4).

We expect that many ledgers will support more than the minimum feature set, but the Common Ledger API is intended to be pretty close to the minimum required functionality, so those optional features are also excluded here.

[mDuo13]

Rebased this onto the latest master and updated the intro to reflect the new intended goal (formalize the existing 5BL without making all the cosmetic / nice-to-have changes).

[mDuo13]

Closed this in favor of #237. (Same contents, new thread.)