The Rosetta Construction API is the write-half of the Rosetta API. This RFC discusses the different chunks of work that need to get done to make this half a reality. Note that discussion of the Data API is out-of-scope for this RFC (and is already fully implemented and partially tested).
We wish to support Rosetta as it enables clients to build once and support multiple chains. Many vendors that wish to build on top of our protocol are asking for full Rosetta support. Not just the read-half (the Data API) but also the write-half.
The desired outcome is a full to-spec Construction API implementation.
The following flow chart is pulled from the Rosetta documentation, but useful for understanding the different pieces necessary for the implementation:
Caller (i.e. Coinbase) + Construction API Implementation
+-------------------------------------------------------------------------------------------+
|
Derive Address +----------------------------> /construction/derive
from Public Key |
|
X |
X Create Metadata Request +---------------------> /construction/preprocess
X (array of operations) | +
Get metadata needed X | |
to construct transaction X +-----------------------------------------------+
X v |
X Fetch Online Metadata +-----------------------> /construction/metadata (online)
X |
|
X |
X Construct Payloads to Sign +------------------> /construction/payloads
X (array of operations) | +
X | |
Create unsigned transaction X +------------------------------------------------+
X v |
X Parse Unsigned Transaction +------------------> /construction/parse
X to Confirm Correctness |
X |
|
X |
X Sign Payload(s) +-----------------------------> /construction/combine
X (using caller's own detached signer) | +
X | |
Create signed transaction X +-----------------------------------------------+
X v |
X Parse Signed Transaction +--------------------> /construction/parse
X to Confirm Correctness |
X |
|
X |
X Get hash of signed transaction +--------------> /construction/hash
Broadcast Signed Transaction X to monitor status |
X |
X Submit Transaction +--------------------------> /construction/submit (online)
X |
+
This flow chart will guide our explanation of what is needed to build a full to-spec implementation of the API. Afterwards, we'll list out each proposed piece of work with details of how it should be implemented. Upon mergin of this RFC, each work item will become an issue on GitHub.
The initial version of the construction API will only support payments. We can quickly follow with a version that will support delgation as well. This RFC will talk about all the tasks assuming both payments and delegation are supported. All token-specific transactions (and any future transactions) are not yet supported and out-of-scope for this RFC.
Before the derivation step, we need to generate a keypair. We'll use the private key to sign the payment and the public key to tell others who the sender is.
Derivation demands that the public key expected as input be a hex-encoded byte-array value. So we'll add functionality to the client-sdk, the generate-keypair binary, and the offcial Mina CLI to marshall the Fq.t * Fq.t pair (the native representation of an uncompressed public key).
The derivation endpoint would be responsible for reading in the uncompressed public key bytes which requires adjusting the Rosetta spec, compressing the public key, and base58-encoding it inline with how we currently represent public keys in serialized form.
The preprocess endpoint takes a proposed set of operations for which we'll need to clearly specify examples for different transactions. It assures they can be converted into transactions and it returns an input that is needed for the metadata phase during which we can gather info on-chain. In our case, this is just the sender's public key+token_id.
The metadata endpoint takes the senders public key+token_id and returns which nonce to use for transaction construction.
The payloads endpoint takes the metadata and the operations and returns an encoded unsigned transaction.
After the payloads endpoint, folks must sign the transaction. In the future, we should build support for this natively, but for now our client-sdk's signing mechanism suffices. As such, we don't need to do much here other than encode the signed transaction properly.
The parse endpoint takes a possibly signed transaction and parses it into operations.
The combine endpoint takes an unsigned transaction and the signature and returns an encoded signed transaction.
The hash endpoint takes the signed transaction and returns the hash.
The submit endpoint takes a signed transaction and broadcasts it over the network. We also should audit broadcast behavior to ensure errors are returned when mempool add fails.
We should integrate the construction calls into the existing test-agent. By doing this, we don't need to worry about getting this into CI since it is already there (or will be by the time this RFC lands, thanks @lk86 !).
We also will want to integrate the official rosetta-cli to verify our implementation.
In addition, we'll manually test on subsequent QA and Testnets.
Think of these as the tasks necessary to complete this project. Each item here will turn into a GitHub issue when this RFC lands.
Add support for creating/marshalling public keys (via Derivation)
Format
Compressed public keys are accepted of the following form:
Field elements are expected to be backed by a 32-byte array where the highest bits of the field are stored in arr[31].
Presented is a hex encoded 32-byte array where the highest bit of arr[31] is the is_odd parity bit.
|----- pk : Fq.t (32 bytes) ------{is_odd}--|
Example:
The encoding fad1d3e31aede102793fb2cce62b4f1e71a214c94ce18ad5756eba67ef398390
Decodes to the field represented by the number fad1d3e31aede102793fb2cce62b4f1e71a214c94ce18ad5756eba67ef398310. That's the same as the encoding, except that the 9 representing the high nybble of the final byte is replaced by 1, by zeroing the high bit. Because the high bit was set, is_odd is true.
Name
We'll call this the "raw" format for our public keys. In most places, we can get away with just adding a -raw flag in some form to support this new kind of representation.
a. Change the Client-SDK
i. Add a rawPublicKeyOfPrivateKey method to the exposed client_sdk.ml module that returns of_private_key_exn s which is then marshalled to a string according to the above specification.
ii. Add a new rawPublicKey : publickey -> string function to CodaSDK.
iii. Add new documentation for this change.
b. Change the generate-keypair binary
i. Also print out the raw representation after generating the keypair on a new line:
Raw public key: ...01E0F3...0392FA
ii. Add a new subcommand show-public-key which takes the private key file as input and prints the same output as running the generate command.
c. Change coda cli
i. Add a new subcommand show-public-key as a subcommand to mina accounts (reuse the implementation in (b.ii)
Read in the bytes, compress the public key, and base58-encoding it inline with how we currently represent public keys in serialized form. Adding errors appropriately for malformed keys.
Add support for our curves and signature to Rosetta (via Derivation)
Follow the instructions on this forum post to add support for the tweedle curves and schnorr signatures. This entails updating the rosetta specification with documentation about this curve, and changing the rosetta-sdk-go implementation to recognize the new curve and signature types. Do not worry about adding the implementation to the keys package of rosetta-cli for now.
Add examples of each kind of transaction that one may want to construct as JSON files. Eventually we'd want one for each type of transaction, but for now it suffices to just include a payment.
For example: The following expressrion would be saved in payment.json
[{
"operation_identifier": ...,
"amount": ...,
"type": "Payment_source_dec"
},
{
"operation_identifier": ...,
"amount": ...,
"type": "Payment_receiver_inc"
},
...
]
This is useful for manual testing purposes and sets us up for integration with the construction-portion of rosetta-cli integration.
Write a function that recovers the transactions that are associated with some set of operations. We should create a test forall (t : Transaction). op^-1(op(t)) ~= t which enumerates all the kinds of transactions. For an initial release it suffices to test this for payments.
First invert the operations into a transaction, we find the sender and include the address in the response (note that on our network an address is made up of a token id and a sender). The options type will be defined as follows:
module Options = struct
type t =
{ sender : string (* base58-ecoded compressed public key *)
; token_id : string (* uint64 encoded as string *)
}
[@@deriving yojson]
endThis is a simple GraphQL query. This endpoint should be easy to implement.
The Rosetta spec leaves the encoding of unsigned transactions implementation-defined. Since we want to make it easy for alternate signers to be created (eg. the ledger), we'll want this encoding to be some faithful representation of the bytes upon which the signature operation acts.
Specifically this is the user command having been transformed into a Transaction_union_payload.t and then hashed into a (field, bool) Random_oracle_input.t. We will serialize the Random_oracle_input in two ways as defined below and send that byte-buffer as hex-encoded ascii.
// Serialization schema for Random oracle input (1)
00 00 00 05 # 4-byte prefix for length of array (little endian)
#
xx xx ... # each field encoded as a 32-bytes each one for each of the length
yy yy ... #
# Field elements are represented by laying out their bits from high
# to low (adding a padding zero at the highest bit in the front)
# and then grouping by 8 and converting to bytes:
#
# (always zero) Bit254 Bit253 Bit252 ... Bit2 Bit1 Bit0
# |----groups of 8---|--groups of 8---|
#
#
00 00 34 D4 # 4-byte prefix for length of bits in the bitstring (little endian)
#
A4 43 D4 ... # the bool list compacted into a bitstring, pad the last 1 byte with
# extra zeros on the right if necessary
// Note: Edited on 8/18 to include 4-byte length of bits in the bitstring to remove any ambiguity between the zero-padding and true zeros in the bitstring
// Serialization schema for Random oracle input (2)
// This is denoted as "signerInput" in the output
//
// The prefix and suffix can be used by a signer more easily
SignerInput (JSON):
{
prefix: [field],
suffix: [field]
}
// where the fields are encoded as strings like above
// example:
{
prefix: [ "000000000000000000000000000000000000000000000000000000000001E0F3", ... ],
suffix: [ "000000000000000000000000000000000000000000000000000000000001E0F3", ... ]
}
A signer would take the prefix and suffix and use it during `derive` (which doesn't necessarily need to be exactly the same as the implementation Mina (it just needs to be "random"). And use `px`, `py`, and `r` in between prefix and suffix for hash.
Another important property of the unsigned-transaction and signed-transaction representations is that they are invertible. The unsigned_transaction_string is then a JSON input (stringified) conforming to the following schema:
{ randomOracleInput : string (* Random_oracle_input.t |> to_bytes |> to_hex *)
, signerInput : SignerInput
, payment: Payment?
, stakeDelegation: StakeDelegation?
}
// where stakeDelegation and payemnt are currently defined in the client-sdk shown below
// it is an error to treat stakeDelegation / payment in any way other than a variant, but it is encoded unsafely like this becuase JSON is garbage-fire and can't represent sum types ergonomically
// Taken from Client-SDK code
type stakeDelegation = {
[@bs.as "to"]
to_: publicKey,
from: publicKey,
fee: uint64,
nonce: uint32,
memo: option(string),
validUntil: option(uint32),
};
type payment = {
[@bs.as "to"]
to_: publicKey,
from: publicKey,
fee: uint64,
amount: uint64,
nonce: uint32,
memo: option(string),
validUntil: option(uint32),
};
Note that our client-sdk only has support for signing payments and delegations but this version of the construction API only supports those transactions types as well. We'll default to the client-sdk for now.
Additionally, we should expose a new method in the client-sdk to feed the raw Random_oracle.Input.t to signer logic in addition to going through the existing signPayment and signDelegation ones. The Client-sdk implementation should enforce that these two implementations agree with an adjustment to our CI unit test.
First convert the operations embedding the correct sender nonce from the metadata. Return an encoded unsigned transaction as described above.
This endpoint will also accept a query parameter ?plain_random_oracle
Since we'll later be broadcasting the signed transaction via GraphQL, our signed transaction encoding is precicesly the union of the format required for the sendPayment mutation and the sendDelegation mutation (stringified):
{
signature: string (* Signature hex bytes as described below *),
payment: payment?,
stakeDelegation: stakeDelegation?
}
Format
// Signature encoding
a signature is a field and a scalar
|---- field 32bytes (Fp) ---|----- scalar 32bytes (Fq) ----|
Use the same hex-encoded representation as described above for the public keys for each of the 32byte chunks.
The parse endpoint takes the transaction and needs to return the operations. The implementation will use the same logic as the Data API transaction -> operations logic and so we do not need an extra task to make this happen.
Importantly, we've ensured that our unsigned and signed transaction serialized representations have enough information in them for us to recreate the transaction full values at parse time (ie. we don't only store the hash needed for signatures).
The combine endpoint encodes the signed transaction according to the schema defined above.
The hash endpoint takes the signed transaction and returns the hash. This can be done by pulling in Mina_base into Rosetta and calling hash on the transaction.
Upon skimming our GraphQL implementation, it seems like it is already succeeding only if the transaction is successfully added to the mempool, but it important we more carefully audit the implementation to ensure this is the case as it's an explicit requirement in the spec.
The submit endpoint takes a signed transaction and broadcasts it over the network. We can do this by calling the sendPayment or sendDelegation mutation depending on the state of the input after parsing the given transaction.
The existing Rosetta test-agent tests our Data API implementation by running a demo instance of Coda and mutating its state with GraphQL mutations and then querying with the Data API to see if the data that comes out is equivalent to what we put in.
We can extend the test-agent to also send construction API requests. We should at least add behavior to send a payment and delegation constructed using this API. We can shell out to a subprocess to handle the "off-api" pieces of keypair generation and signing.
We also should include logic that verifies the following:
- The unsigned transaction output by payloads parses into the same operations provided
- The signed transaction output by combine parses into the same operations provided
- After the signed transaction is in the mempool, the result from the data api is a superset of the operations provided originally
- After the signed transaction is in a block, the result from the data api is a superset of the operations provided orginally
The rosetta-cli is used to verify correctness of implementations of the rosetta spec. This should be run in CI against our demo node and against live qa and testnets. We can release a version on a testnet before we've fully verified the implementation against rosetta-cli, but the project is not considered "done" until we've done this properly.
It's worth noting that the rosetta-cli is about to get new features to be more flexible at testing other construction API scenarios. These changes will certainly support payments and delegations. We don't need to wait for the implementation of this new system to support payments today.
It's extra work, but we really wish to enable folks to build on our protocol in this way.
Decisions were made here to limit scope where possible to enable shipping an MVP as soon as possible. This is why we are explicitly not supporting extra transactions on top of payments (initially) and payments+delegations (closely afterwards).
Luckily Rosetta has a very clear specification, so our designs are mostly constrained by the decisions made in that API.
In marshal keys (c), we could also change commands that accept public keys to also accept this new format. Additionally, we could change the GraphQL API to support this new format too. I think both of these changes are unnecessary to prioritize as the normal flows will still be fine and we'll still encourage folks to pass around the standard base58-encoded compressed public keys as they are shorter.
In the sections about encoding unsigned transactions and encoding signed transactions, we make an explicit decision to pick a format that supports arbitrary signers. There is minimal change involved with the client-sdk to make that supported; additionally, this was done to improve implementation velocity and because we did conciously choose that interface with usability in mind. JSON is chosen to pack products of data as using a readable JSON string makes it easy to audit, debug, and understand our implementation.
The spec
There are no unresolved questions at this time that I'd like to answer before merging this RFC.
As stated above, explicitly out-of-scope is any future changes to the Data API portion of Rosetta and Construction API support for transactions other than payments and delegations.