API Functions for Exchange Implementation

[poloniex]

Some additional features are needed for exchange implementation to be possible:

1. An API method that lists all transactions the wallet is involved with, including both sending and receiving. Each transaction listed should include at minimum the amount, the number of confirmations, the destination address, the transaction ID/hash, and the timestamp.
2. Methods for wallet encryption. This could be like bitcoin, where there is an unlock call and a lock call, or the encryption key/passphrase could be supplied as a param of the /wallet/send method.

[DavidVorick]

@poloniex

Sorry it took so long to get here, but I'm working on the wallet right now. The new wallet should be coded up by Aug. 1st, and the first release with the changes should be on Aug. 15th.

Wanted to run the api calls by you though.

/wallet (GET) :: display the balance of the wallet, both siacoins and siafunds, potentially with additional information

I wasn't planning on having this call list all transactions related to the wallet, it seemed unwieldy.

/wallet/address (GET) :: list all active addresses (and maybe their respective balances)

/wallet/address (POST) :: returns a new address that can receive coins

/wallet/$(ADDR) :: display the balance of an address, and return every txn related to the address. Each txn will also have a number of confirmations. The receiver will need to parse the transactions to figure out how many confirmations the balance as a whole has (or some subset of the balance). In the simplest case, this just means looking at the # of confirmations on each transaction and then taking the lowest number - that will give you the number of confirmations on the balance.

From the transaction you will be able to parse the amount, the confirmations, the source(s), the destination, the ids (both of the outputs and of the transaction), and the timestamp, as well as the actual raw transaction.

/wallet/siacoins/send (POST) :: requires a password, which will be established by the user when the wallet is created. The wallet will temporarily decrypt the private key, send the coins, and then wipe the decrypted key from memory.

/wallet/siafunds/send (POST) :: same as /wallet/siacoins/send, except it sends siafunds.

There will be a few other calls related to backups, setting passwords, and similar functions, but the above are the ones related to sending and receiving coins.

[poloniex]

If /wallet/siacoins/send automatically selects inputs from all addresses in the wallet (rather than requiring a single source address to be specified), then an exchange will need a method to list all transactions in the wallet. If you provide only the methods proposed above, I would need to construct this list using /wallet/$(ADDR) by querying every deposit address, collecting the txns, and (if /wallet/$(ADDR) provides only txns and no further transaction information) fetching the amount, destination, and confirmations of each transaction. As more deposit addresses are generated, this could end up becoming very slow; it is preferable to minimize the number of API queries needed.

Unless sending funds must be done by specifying both a source and a destination address, knowing the balances of each address in the wallet is actually of little use to an exchange; because trades happen off the blockchain, the balance of a user's deposit address rarely matches that user's balance on the exchange or even the sum of that user's deposits.

[DavidVorick]

I am concerned about the size of a /wallet/transactions call, it could potentially exceed the amount that you could safely send over an API.

I guess we could add pagination, so that if the size exceeds 1mb or something, it'd be broken up into several pages, and you could specify a page with each call. I think that's reasonable.

[poloniex]

Most wallet APIs provide a way to limit the number of transactions returned. Bitcoin's allows you to specify such a number, and will return the latest n transactions. Others allow you to specify a starting block, sometimes an ending block as well. The latter is ideal -- specifying a starting block would return all transactions between that block and the latest or a specified ending block.

[DavidVorick]

Working on it now, I think I can have everything you've request ready in the next release.

Timestamps on the transactions are vague. Do you want the time that the transaction was first seen, or the timestamp on the block it appeared in? Transactions in Sia don't have a timestamp on their own. First block appearance would be pretty easy, but timestamp first seen would be harder.

[poloniex]

First block appearance is fine.

[DavidVorick]

For the most part, the implementation is complete. I still need to update the actually http api, but the backend is ready.

All wallet keys and seeds are encrypted on disk. After loading, the wallet needs to be unlocked. The wallet won't start until it has been unlocked. The wallet is unlocked using a password/key. The wallet can be relocked by calling 'Close', which will wipe all of the keys from memory (to the extent possible. Because of context switching, swap space, and other confounding variables, it's possible that keys will be in memory unreachable to siad) You can call unlock and close as many times as you like.

When requesting transaction history, you can either ask for all transactions, or you can ask for transactions between two block heights [start, end]. Instead of actual transactions being returned, a 'Wallet Transaction' will be returned, which contains a single siacoin or siafund input or output. It also contains the full transaction (which means there's high redundancy for large transactions - this could perhaps be replaced by using transaction ids).

As an example, transaction A might have 3 inputs, 1 output, and 1 refund. 5 wallet transactions will be created. One for each input, and one for each output. Each will specify the type of movement (input vs. output, and siacoin vs. siafund), the address involved (generally only useful for incoming coins (appearing as outputs in a transaction), as the wallet arbitrarily selects addresses when creating inputs to a transaction), and the value. Each wallet transaction contains a timestamp and a confirmation height (to get the number of confirmations, you'll need to compare to the current height).

All 5 are guaranteed to appear consecutively in the transaction history.

When requesting a balance, you can request 'ConfirmedBalance' and 'UnconfirmedBalance'. Confirmed balance currently will only look at 1 confirmation, other than parsing the transaction history yourself you can't get 6 confirmation balance right now. Unconfirmed balance will return the number of incoming and outgoing siacoins, which will include refunds.

For example, if you create an unconfirmed transaction that sends 5 coins, and a 500 coin input is used, the result reported will be 500 outgoing ,and 495 incoming.

Hope this is all okay. Let me know if there are changes you think should be made.

[DavidVorick]

Queries:

• /wallet [GET]
• /wallet/close [PUT]
• /wallet/history [GET]
• /wallet/history/$(addr) [GET]
• /wallet/seed [GET]
• /wallet/seed [PUT]
• /wallet/seed [POST]
• /wallet/siacoins [GET]
• /wallet/siacoins [PUT]
• /wallet/siafunds [GET]
• /wallet/siafunds [PUT]
• /wallet/transaction/$(id) [GET]
• /wallet/unlock [PUT]

/wallet [GET]

Function: Returns basic information about the wallet, such as whether the
wallet is locked or unlocked.

Parameters: none

Response:

struct {
    encrypted bool
    unlocked  bool
}

'encrypted' indicates whether the wallet has been encrypted or not. If the
wallet has not been encrypted, then no data has been generated at all, and the
first time the wallet is unlocked, the password given will be used as the
password for encrypting all of the data. Encrypted will only be set to false if
the wallet has never been unlocked before (the unlocked wallet is still
encryped - but the encryption key is in memory).

'unlocked' indicates whether the wallet is currently locked or unlocked. Some
calls become unavailable when the wallet is locked.

/wallet/close [PUT]

Function: Locks and closes the wallet, preparing for shutdown. After being
closed, the keys are encrypted. Queries for the seed, to send siafunds, and
related queries become unavailable. Queries concerning transaction history and
balance are still available.

Parameters: none

Response: Standard.

/wallet/history [GET]

Function: Return a list of transactions related to the wallet.

Parameters:

struct {
    start int
    end   int
}

Response:

struct {
    transactions []WalletTransaction
}

'transactions' is a list of 'WalletTransactions'. Wallet transactions are
transactions that have been processed by the wallet and given more information,
such as a confirmation height and a timestamp. Each wallet transaction contins
information about only a single input or output. One network transaction with
many inputs an outputs can result in many wallet transactions. All of the
wallet transactions created by a network transaction are guaranteed to be
consecutive in history. Wallet transactions will always be returned in
chronological order.

A wallet transaction takes the
following form:

struct WalletTransaction {
    TransactionID         string
    ConfirmationHeight    int
    ConfirmationTimestamp uint64

    FundType       string
    OutputID       string
    RelatedAddress string
    Value          int
}

'TransactionID' is the id of the transaction from which the wallet transaction
was derived. The full transaction can be obtaied by calling
'/wallet/transaction/$(id)'

'ConfirmationHeight' is the height at which the transaction was confirmed. The
height will be set to 'uint64MAX' if the transaction has not been confirmed.

'ConfirmationTimestamp' is the time at which a transaction was confirmed. The
timestamp is a 64bit unix timestamp, and will be set to uint64MAX if the
transaction is unconfirmed.

'FundType' indicates what type of fund is represented by the wallet
transaction. The options are 'Siacoin Input', 'Siacoin Output', 'Siafund
Input', and 'Siafund Output', corresponding to whether the fund is related to
siacoins or siafunds, and to whether the fund is an input to the transaction or
an output of the transaction. When looking at transactions, it should be noted
that a 'Siacoin Input' actually represents outgoing siacoins, as they are an
input to the transaction; they are being spent. A 'Siacoin Output' represents
incoming coins, because the output is being created by the transaction and made
available to the wallet.

'OutputID' is the id of the output. OutputIDs will always be unique.

'RelatedAddress' is the address that is affected. For inputs (outgoing money),
the related address is usually not important because the wallet arbitrarily
selects which addresses will fund a transaction. For outputs (incoming money),
the related address field can be used to determine who has sent money to the
wallet.

'Value' indicates how much money has been moved in the input or output.

/wallet/history/$(addr) [GET]

Function: Return all of the transaction related to a specific address.

Parameters: none

Response:

struct {
    transactions []WalletTransaction
}

'transactions' is a list of 'WalletTransactions' that affect the input address.
See the documentation for '/wallet/history' for more information.

/wallet/seed [GET]

Function: Return the seed that is being used to generate addresses. This seed
can be used to derive secret keys, and must be kept safe. This call is
unavailable when the wallet is locked or closed.

Parameters:

struct {
    dictionary string
}

'dictionary' is the name of the dicitionary that should be used when encoding
the seed.

Response:

struct {
    primarySeed        string
    addressesRemaining int
    backupSeeds        []string
}

'primarySeed' is the seed that is actively being used to generate new addresses
for the wallet.

'addressesRemaining' is the number of addresses that remain in the primary seed
until exhaustion has been reached and no more addresses will be generated.

'backupSeeds' is a list of seeds that are no longer used to generate new
addresses, but are still tracked can have spendable outputs.

A seed is an encoded version of a 128 bit random seed. The output is 15 words
chosen from a small dictionary as indicated by the input. The most common
choice for the dictionary is going to be 'english'. The underlying seed is the
same no matter what dictionary is used for the encoding. The encoding also
contains a small checksum of the seed, to help catch simple mistakes when
copying. The library entropy-mnemonics is used when encoding.

/wallet/seed [PUT]

Function: Get a new address from the wallet generated by the primary seed. An
error will be returned if the seed has been exhausted (there is a limit on the
number of addresses that can be generated from one seed), or if the wallet is
locked/closed.

Parameters: none

Response:

struct {
    address string
}

'address' is the address that cna

/wallet/seed [POST]

Function: Fetch a new seed for the wallet. The old seed will be added to the
list of backup seeds. In general, this call should be avoided unless address
exhaustion is reached. Because making a new seed is considered a significant
action, there is a verification field that must be set to true. This is to
prevent mistakes e.g. calling POST instead of GET or PUT.

Parameters:

struct {
    verification bool
    dictionary   string
}

'verification' must be set to true, or an error is returned. This is to prevent
an implementation mistake where POST is called instead of GET or PUT.

'dictionary' is the name of the dictionary that should be used when encoding
the newly created seed. 'english' is the most common choice.

Response:

struct {
    newSeed string
}

'newSeed' is the new seed that will be used as the seed for generating new
addresses.

/wallet/siacoins [GET]

Function: get the siacoin balance (confirmed and unconfirmed) of the wallet.

Parameters: none

Response:

struct {
    confirmedSiacoinBalance     int
    unconfirmedOutgoingSiacoins int
    unconfirmedIncomingSiacoins int
}

'confirmedSiacoinBalance' is the number of siacoins available to the wallet as
of the most recent block in the blockchain.

'unconfirmedOutgoingSiacoins' is the number of siacoins that are leaving the
wallet according to the set of unconfirmed transactions. Often this number
appears inflated, because outputs are frequently larger than the number of
coins being sent, and there is a refund. These coins are counted as outgoing,
and the refund is counted as incoming. The difference in balance can be
claculated using 'unconfirmedIncomingSiacoins' - 'unconfirmedOutgoingSiacoins'

'unconfirmedIncomingSiacoins' is the number of siacoins are entering the wallet
according to the set of unconfirmed transactions. This number is often inflated
by outgoing siacoins, because outputs are frequently larger than the amount
being sent. The refund will be included in the unconfirmed incoming siacoins
balance.

/wallet/siacoins [PUT]

Function: Send siacoins to an address. The outputs are arbitrarily selected
from addresses in the wallet.

Parameters:

struct {
    amount      int
    destination string
}

'amount' is the number of siacoins being sent.

'destination' is the address that is receiving the coins.

Response: standard

/wallet/siafunds [GET]

Function: get the siafund balance of the wallet. Only the confirmed balance is
displayed for siafunds.

Parameters: none

Response:

struct {
    siafundBalance      int
    siacoinClaimBalance int
}

'siafundBalance' is the number of siafunds available to the wallet as
of the most recent block in the blockchain.

'siacoinClaimBalance' is the number of siacoins that can be claimed from the
siafunds as of the most recent block. Because the claim balance increases every
time a file contract is created, it is possible that the balance will increase
before any claim transaction is confirmed.

/wallet/siafunds [PUT]

Function: Send siafunds to an address. The outputs are arbitrarily selected
from addresses in the wallet. Any siacoins available in the siafunds being sent
(as well as the siacoins available in any siafunds that end up in a refund
address) will become available to the wallet as siacoins after 144
confirmations. To access all of the siacoins in the siacoin claim balance, send
all of the siafunds to an address in your control (this will give you all the
siacoins, while still letting you control the siafunds).

Parameters:

struct {
    amount      int
    destination string
}

'amount' is the number of siafunds being sent.

'destination' is the address that is receiving the funds.

Response: standard

/wallet/transaction/$(id) [GET]

Function: Get the transaction associated with a specific transaction id.

Parameters: none

Response:

struct {
    transaction Transaction
}

'transaction' is a 'types.Transaction'. The full transaction can be seen in
types.transaction.go. All hashes in the transaction are encoded as strings.

/wallet/unlock [PUT]

Function: Unlock the wallet. The wallet is capable of knowing whether the
correct password was provided. The first time that the wallet is unlocked ever,
the password used will become the permanent password. Any string can be used as
the key, though it is generally recommended that the encoded primary seed be
used.

Parameters:

struct {
    key string
}

Response: standard

[DavidVorick]

@poloniex @agundy @Mingling94 @lukechampine

the above comment contains the rough draft of the specification for the wallet api. If there is anything that is unusual, unusable, or you think could be improved, please let me know.

@poloniex, let me know if this adds all of the features that you will need to add Sia to an exchange.

The majority of this has been implemented already.

[poloniex]

The wallet transaction thing is a little cumbersome for exchange purposes. The API described is sufficient for an exchange implementation, but I will have to construct the list of transactions I described myself -- either by adding up all the wallet transactions, or fetching transactions individually by id (probably the former).

[Mingling94]

• So if I'm understanding it correctly, you're intending seeds to serve a dual purpose of generating addresses and (optionally) being the passwords to wallets? Won't most people just want to use their own passwords schemes?

• Also your description is incomplete for

  /wallet/seed [PUT]

  'address' is the address that cna

• Lastly,

I think the API could be less repetitious and more logically consistent by combining /wallet/siacoins [PUT] and /wallet/siafunds [PUT] into one

/wallet/transaction [POST]

struct {
    amount      int
    destination string
    currency    string
}

while also combining /wallet/siacoins [GET] and /wallet/siafunds [GET] into

/wallet/balance [GET]

struct {
    confirmedSiacoinBalance     int
    unconfirmedOutgoingSiacoins int
    unconfirmedIncomingSiacoins int
    siafundBalance              int
    siacoinClaimBalance         int
}

[DavidVorick]

People should not make their own passwords. People make grossly insecure passwords, and dictionary attacks are extremely powerful. While they could make their own password when locking the wallet, the vast majority of people make passwords (even at 24+ characters!) that can be guessed in a relatively short amount of time using a dictionary attack. That's why we recommend you use the seed password - it's got a full 128 bits of entropy and is immune to dictionary attacks.

I could put all of the balance stuff into the /wallet [GET] call.

I'm not sure that it makes sense to combine the /siacoins and /siafunds calls when sending money.

[Mingling94]

• That's the advisable method, my concern is more of a use-case thing. Will we even give the option of a custom password or will we require (at least on the UI side) that passwords be this 15 word seed? If we give the custom password option, people will likely take it. Thus the UI will probably auto-use the primary seed as a password.
• Balance stuff going into /wallet [GET] is fine. Nothing initially seems problematic about it.
• Hmm, it's a difference of string placement in the call parameters or in the api url itself. I just feel it makes more sense semantically and RESTfully to be 'POST'ing a transaction rather than 'PUT'ing an update to the siacoin or siafund balances.

[DavidVorick]

The new api functions have been implemented and merged into master. Documentation on the calls can be found in doc/API.md.

Further discussion/requests can be continued by opening a new issue.