Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improved Wallet State Managment #30

Open
wants to merge 3 commits into
base: master
from

Conversation

@yeastplume
Copy link
Member

yeastplume commented Nov 4, 2019

Link to Rendered Document

Edit by lehnberg: Correcting link to rendered document

@lehnberg

This comment has been minimized.

Copy link
Contributor

lehnberg commented Nov 4, 2019

Nice. Couple of nitpicky points:

  • Most RFCs will be "enhancements", and wallet-update-process sounds quite generic. What would a more descriptive title be? improved-wallet-state-management?

  • For better readability for future readers, write in present form with the assumption that the RFC has already been merged. So rather than proposing changes, describe the already accepted changes. Similarly current wallet procedures become previous wallet procedures, etc.

  • IIRC, largest doesn't lead to all outputs being locked, just the most possible. This to reduce UTXOs, but comes at a privacy cost. It might be worth noting that a smallest strategy might reduce output likability, but may also lead to increased transaction fees.

@yeastplume yeastplume changed the title Wallet Update Process Enhancements Improved Wallet State Managment Nov 4, 2019
@yeastplume

This comment has been minimized.

Copy link
Member Author

yeastplume commented Nov 4, 2019

Updated the tense and title as per suggestions.

selection-strategy=all will sweep as many outputs as it can, up to a hardcoded limit of 500. In most cases this will have the effect of locking all user funds.

There was some conversation a while back where someone was arguing that the default selection strategy didn't actually reduce overall UTXO set size effectively. I admit this isn't something I've studied in any particular detail, so would be interested in hearing more opinions here. The only thing I can say with relative certainty is the current default 'sweep' method is terrible for usability regardless of it's effect on UTXO set size.

@lehnberg

This comment has been minimized.

Copy link
Contributor

lehnberg commented Nov 5, 2019

Looks much better @yeastplume 👍

selection-strategy=all will sweep as many outputs as it can, up to a hardcoded limit of 500. In most cases this will have the effect of locking all user funds.

I did not know this. Seems I had completely misunderstood this strategy to be something more like largest.

@DavidBurkett

This comment has been minimized.

Copy link

DavidBurkett commented Nov 5, 2019

There was some conversation a while back where someone was arguing that the default selection strategy didn't actually reduce overall UTXO set size effectively.

Yea, I was the one that said that. While it results in a temporary reduction, as long as you're using a selection strategy that routinely spends "dust" (ie. don't use a "largest first" strategy), your current inputs will eventually be spent as you continue to use Grin, so you're mostly just inconveniencing yourself by spending them all everytime, and drawing a big circle around your inputs, reducing Grin to an accounts-based model which is bad for privacy.

@DavidBurkett

This comment has been minimized.

Copy link

DavidBurkett commented Nov 5, 2019

For better readability for future readers, write in present form with the assumption that the RFC has already been merged. So rather than proposing changes, describe the already accepted changes. Similarly current wallet procedures become previous wallet procedures, etc.

I actually find it more confusing now that the tenses have been updated. I like when the RFC clearly states what takes place today, and how things will work after the RFC is implemented. I wouldn't suggest changing this RFC again, but maybe it's something we can talk about elsewhere?

[unresolved-questions]: #unresolved-questions

- Do we limit the Kernel Lookup part of the process to just cases where there is no change output. Is there any benefit to calling it for all outstanding transactions on each lookup (or any particular downside to doing so?)
- Can we add the TTL field at consensus level? Making it consensus means much less scope for inconsistency between wallets overall, but how would it be commit to? (Could it be inclded in the kernel excess somehow, for example?)

This comment has been minimized.

Copy link
@DavidBurkett

DavidBurkett Nov 5, 2019

The only thing we could do is add it to the kernel and become part of the signature, but it could end up being really bad for privacy. For example, if Grin++ uses a different algo for calculating TTL than grin wallet, allowing you to quickly identify grin wallet kernels and Grin++ kernels, effectively splitting the anon set in half, I would recommend not supporting this at the consensus level for this reason.

Copy link

DavidBurkett left a comment

Great work!

@yeastplume

This comment has been minimized.

Copy link
Member Author

yeastplume commented Nov 6, 2019

Added more clarity on smallest selection process, fixed typos.

Copy link
Member

quentinlesceller left a comment

Looking good 👍 some minors typo fix below before entering FCP.


#### Transacting (Payer -> Payee) Output + Transaction Creation

Outputs and tranasctions are created during the transaction exchange process as follows. Note that the invoicing workflow is mostly identical with the roles of Payer and Payee reversed, so this workflow is not outlined separately.

This comment has been minimized.

Copy link
@quentinlesceller

quentinlesceller Nov 12, 2019

Member

Typo tranasctions


* The sender creates a new 'blank' transaction Slate, adding inputs and change outputs to the slate.
* The sender generates a random kernel offset. (The kernel offset is always generated by the slate creator)
* The sender sends the slate to the payee (via file or http). If sending synchronously, (e.g via http) the associated transaction is saved to the log after a response from the payee's listener. If sending asynchronously, (e.g. via file), the transaction is saved immediately.

This comment has been minimized.

Copy link
@quentinlesceller

quentinlesceller Nov 12, 2019

Member

e.g -> e.g.

This comment has been minimized.

Copy link
@quentinlesceller

quentinlesceller Nov 12, 2019

Member

plus 2 spaces after.


* For every unspent output in the wallet:
* If the output's status in the user's wallet is 'unconfirmed' and it is present in the UTXO set, change the output's status to 'confirmed', and update the associated transaction status to 'confirmed'.
* If the output's status in the user's wallet is 'confirmed' and it no longer appears in the UTXO set, set it's status to 'Spent'. Note these outputs will usually be locked so they cannot be selected for spending again.

This comment has been minimized.

Copy link
@quentinlesceller

quentinlesceller Nov 12, 2019

Member

it's -> its


The Update by Output process on its own is not enough to ensure the contents of a wallet is correct. There are many easily-encountered situations in the course of a wallet's operation where this process is insufficient and can potentially lead to an wallet state that's inconsistent with the UTXO set, including but not limited to:

* Manually cancelling transctions and unlocking outputs before they've had a chance to confirm on the chain.

This comment has been minimized.

Copy link
@quentinlesceller

quentinlesceller Nov 12, 2019

Member

transctions -> transactions

* If an output exists in the UTXO set and is not in the wallet, create an output for the wallet at the derivation path stored in the bullet proof, and create a new transaction crediting the amount
* If an output exists in the UTXO set and is marked as 'spent', mark it as unspent and cancel any associated transaction log entries.

Additionally, the check-repair process can take a flag instructing it to delete unconfirmed outputs and reset all outstanding transctions. If this flag is set:

This comment has been minimized.

Copy link
@quentinlesceller

quentinlesceller Nov 12, 2019

Member

transctions -> transactions


To rectify these problems, the following changes are made:

* A Time-To-Live (`TTL`) field is added to the Slate, which is defined as the last block height at which a wallet should attempt to complete a transaction. This field should be respected at the wallet-exchange level, but is not currently commit to at a consensus level. If a wallet detects a particlar transaction's TTL has expired, it will automatically cancel the transaction and unlock associated outputs. Note this doesn't prevent the other party sending the transaction to the node after this happens, but there is no guarantee the outputs will still be available. However, if this does happen the `scan` command will correct the wallet state.

This comment has been minimized.

Copy link
@quentinlesceller

quentinlesceller Nov 12, 2019

Member

particlar -> particular


The update process was previously run inline in a synchronous blocking fashion as part of individual wallet commands. With the `scan` command becoming part of normal operations, it's expected that a particular invocation of the overall update process could potentially take a long time. This is acceptable in single-use modes such as the single invocation model of `grin-wallet`, but is far less usable in environments where the wallet and Owner API stay resident and the caller may not have any particular insight as to why a call invoking the update process might be taking a long time to complete.

To address this, the wallet provides a method of calling the Overall Update Process in its own thread that attempts to mimimise its usage of the wallet lock. The Wallet's Owner API (V3 Only) is extended with the following functions:

This comment has been minimized.

Copy link
@quentinlesceller

quentinlesceller Nov 12, 2019

Member

mimimise -> minimize or minimise.

* `stop_update_thread()` => Stops the update thread
* `get_update_status()` => Returns the current status of the update thread, along with as much information as possible about what the thread is doing and, in the case of the `scan` process, the percentage complete.

Further, if the update thread is currently running, commands that previously always called the Overall Update Process (such as `info`, `txs`) etc do not call the update process. This means that when invoked via the command line, the behaviour is unchanged. However, wallets running the Owner API can instead choose to run and monitor the background update thread at the desired frequency while keeping the user informed of any long-running update processes.

This comment has been minimized.

Copy link
@quentinlesceller

quentinlesceller Nov 12, 2019

Member

(such as info, txs) etc -> (such as info, txs, etc...) ?

[unresolved-questions]: #unresolved-questions

- Do we limit the Kernel Lookup part of the process to just cases where there is no change output. Is there any benefit to calling it for all outstanding transactions on each lookup (or any particular downside to doing so?)
- Can we add the TTL field at consensus level? Making it consensus means much less scope for inconsistency between wallets overall, but how would it be commit to? (Could it be inclded in the kernel excess somehow, for example?)

This comment has been minimized.

Copy link
@quentinlesceller

quentinlesceller Nov 12, 2019

Member

inclded -> included

Coinbase outputs and their related transactions are created via the wallet listener as part of the mining process. The workflow is:

* When mining, create a potential coinbase output for the target block.
* If the block is accepted and the output is detected in the UTXO set (via the [Update by Output process](#)), create a transaction log entry of type 'Confirmed Coinbase', with 'confirmed' set to true.

This comment has been minimized.

Copy link
@quentinlesceller
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.