CIP-1856 | Collateral Key derivation

[SebastienGllmt]

This CIP extends CIP-1852 to add a special derivation path for inputs to be used as collateral for smart contract execution. See README for rationale.

This PR is part of a series of different ways we can improve collateral handling in Cardano. You can find the three options below

Collateral Key
Collateral Output
Collateral Reward

[nc6]

When we were discussing collateral inputs, we were thinking that there might indeed be only one collateral UTxO, and that most transactions would be over-collateralised. I don't think relying on this is particularly risky. Firstly, we expect that most transactions spending from script-locked addresses will be submitted by the PAB, and hence already verified. Secondly, with IntersectMBO/ouroboros-network#3230 we expect the local node to reject submitted transactions that fail phase-2 validation.

[SebastienGllmt]

Unless you are running your own node, there is no guarantee anybody is setting this flag.

[dcoutts]

I don't follow.

It's quite simple: if your wallet is working correctly then you will never ever have any collateral taken. The local node that you submit your transaction two is just an extra double-check on this. So it should be really very safe indeed.

[dcoutts]

I would put this much more strongly:

Wallets must run scripts locally to determine that they do not fail and how many execution units they need. This is a crucial part of the function of a wallet supporting scripts. They should not construct or submit invalid transactions.

[francisluz]

Would make sense, have to describe the best practices like you're mentioning about running your own node?

[michaelpj]

There is no reason why a good-faith user should ever end up submitting a transaction with failing scripts. So the only people who are ever at risk of losing their collateral are malicious users or people who've inexplicably decided to go to some trouble to not check their transactions before sending them.

[mmahut]

What about light wallets or hosted APIs, where the operator of the service does not want to consume its computational resources but instead wants to delegate this responsibility to the user?

[GeorgeFlerovsky]

Can we really categorically rule out that wallets and PAB will never send transactions with failing scripts? Should the user pay with an overly large collateral utxo in case of a wallet/PAB bug?

I've lost my fair share of collateral on the Alonzo Blue testnet (using Cardano CLI) :)

Currently, Plutus.evaluateScriptCounting is giving me an Ex Budget that is too low to succeed when submitted in a transaction to the testnet. I've had to multiply it by a factor of ~20X to stop losing collateral in the transaction.

https://github.com/input-output-hk/Alonzo-testnet/blob/8b4e51fb849ffc0fa2801d267a93438ba5f1e0eb/resources/plutus-sources/plutus-alwayssucceeds/app/plutus-alwayssucceeds.hs

[michaelpj]

The testnet node currently accepts transactions with failing scripts and posts them to the network as failures. That will be fixed before mainnet.

[BaldBodyBuilder]

Should this happen you would likely have to pay a small fee to transfer from collateral to main account? Whatever the protocol minimum parameters dictate?

[SebastienGllmt]

Yes, that's correct

[BaldBodyBuilder]

I feel like there could be simple improvements to handle this, maybe some mechanism to take an average of collateral within the network to help users better 'forecast' what collateral they might need so they can plan ahead and not use more than necessary.

[SebastienGllmt]

Yes, wallets should also make a suggestions for how much to put in for common use cases. Something like "use DEXs often? Put this much in and you'll usually be fine!". We can warn them before they make a smart contract call to add more collateral, but it's ideal if they just set some amount upfront for their usage pattern and forget about it

[vsubhuman]

The proposal implies wallets would always select all "pure" UTxO available on the collateral account?

[SebastienGllmt]

Probably best not to select more than required because otherwise you're increasing the tx fee for no reason. Additionally, there's a limit on the number of UTXO entries you can add as collateral.

[vsubhuman]

Later in this CIP, if I understood correctly, it is implied that light wallets have no way to know how much of collateral is actually needed so they have to select all utxo in collateral account and at best provide some guesses of how much of collateral would be needed - which then means here in the naive solution the step of accessing whether the selected collateral is sufficient or not is impossible at all.

[v-almonacid]

According to the specs collateral is a function of the tx fee, so it can be computed beforehand.

[SebastienGllmt]

Collateral is not an explicit field in transactions -- it's calculated based on the tx fee which depends on the amount you include as part of the witness for each redeemer script.

[vsubhuman]

Ah, but the problem then is that the correct fee would depend on the execution units then as well, so if I'm getting it right, rather than knowing the correct collateral from knowing the correct fee, a wallet that has no access to a local personal full-node would not be able to correctly get neither the fee, nor the collateral 🤔

[SebastienGllmt]

Yes, that's correct. That's why there is talk about trying to get the PAB to run in the browser so it can be included as part of light wallets. Otherwise the best you can do is trust some 3rd party to tell the truth about the cost of running the contract and risk losing collateral if they lied.

[dcoutts]

There are two independent issues:

1. A wallet does need some information from the chain to be able to construct a transaction. It needs to know the UTxOs that make up the user's wallet so it can do coin selection. It needs to know the value of a few protocol parameters etc. Based on that it can work out the executions and fees for the transaction it builds. The hash of the relevant parameters (price of execution units etc) is included in the tx body and is checked in phase one, and thus no collateral can be lost.

2. Collateral needs to be supplied to cover the fees. Note that these are the fees that are actually declared. So again, if you get the fees wrong then the transaction is simply rejected and no collateral is taken.

The only way to arrange to have your collateral taken is to calculate the execution units incorrectly, but to include the correct protocol parameters into the transaction body. That is you cannot get this wrong by 3rd parties lying to you about current protocol parameters. You have to do something really wrong.

Ah, but the problem then is that the correct fee would depend on the execution units then as well,

Yes.

so if I'm getting it right, rather than knowing the correct collateral from knowing the correct fee, a wallet that has no access to a local personal full-node would not be able to correctly get neither the fee, nor the collateral

A wallet needs some information from the chain like the wallet's UTxO, and the values of some of the current protocol parameters. If you're thinking about a light wallet and a compromised server lying to you, then there's no problem: because the hash of the relevant protocol parameters is included in the tx body and checked by the ledger rules in phase 1, you will not loose collateral if you were given the wrong protocol parameter values, or if the values changed before the tx is submitted.

Yes, that's correct. That's why there is talk about trying to get the PAB to run in the browser so it can be included as part of light wallets. Otherwise the best you can do is trust some 3rd party to tell the truth about the cost of running the contract and risk losing collateral if they lied.

No. Unless there's a design mistke here that I am not aware of then I don't think you can lose collateral if the server lies. Provided that your local wallet can run the scripts locally then the only reason to loose collateral is bugs in the wallet. You do not need the whole PAB to be able to run scripts. There are more lightweight libraries that can do that. For example the CLI will do this using the Cardano API library.

[v-almonacid]

@dcoutts what other lightweight libraries can do that? by "Cardano API" you mean cardano-wallet?

Provided that your local wallet can run the scripts locally

I must be missing something but I'm not sure how a pure JS wallet like Yoroi will be able to do this. If a light wallet needs to rely on a remote node every time it needs to recompute the fees and re-validate a smart-contract tx, this doesn't translates into a great UX.

I'm also confused because you mention that the wallet only needs to know its UTXO set and some protocol parameters in order to compute the execution units, but then you mention that the wallet needs to run the scripts locally, which sound like a big additional requirement.

[vsubhuman]

so if I'm getting it right, rather than knowing the correct collateral from knowing the correct fee, a wallet that has no access to a local personal full-node would not be able to correctly get neither the fee, nor the collateral

A wallet needs some information from the chain like the wallet's UTxO, and the values of some of the current protocol parameters. If you're thinking about a light wallet and a compromised server lying to you, then there's no problem: because the hash of the relevant protocol parameters is included in the tx body and checked by the ledger rules in phase 1

@dcoutts , yes, but the main point is specifically about whether it is 100% guaranteed already that all wallets will be able to run scripts locally to validate their execution and calculate the execution points?

You do not need the whole PAB to be able to run scripts. There are more lightweight libraries that can do that. For example the CLI will do this using the Cardano API library.

Is this planned to be compatible with light wallets, e.g. allowing to run any script from any user, without having an instance of a wallet created locally?

[dcoutts]

With my CIP editor's hat on, it's not my place to make judgements on the wisdom of CIPs, but to review if they are technically coherent. There's obviously a few comments by various people, but broadly speaking it's a technically coherent proposal.

With my Cardano architect's hat on I think this whole approach is unnecessary, and wallets that choose to take this approach will end up with a worse UX. I would recommend that wallets do not follow this proposal. My view is that the so-called naive solution is a better solution:

• It involves no new concepts to explain to users.
• The notion of the wallet's balance remains simple: a single number.
• The risk of loosing collateral is overblown, the risk is tiny.
• The concern about having to rearrange the UTxO in some corner cases is not a new thing: wallets must do this already for corner cases in ordinary UTxO input selection.
• Such rearrangement transactions are already easy to identify (as all inputs and outputs are within the wallet) and so it's easy to mark them as such in the transaction history. The same is true for hardware wallets that already have to sign such rearrangement transactions.
• Yes multi-sig wallets will have to fund collateral inputs from a personal address, rather than a shared address. This is of course only necessary when using Plutus scripts together with multisig. As far as I'm aware, all wallets that are adding multi-sig support already support personal accounts, and so this should not be a significant burden to have fees or collateral put up by one party.

[dcoutts]

I would put this much more strongly:

Wallets must run scripts locally to determine that they do not fail and how many execution units they need. This is a crucial part of the function of a wallet supporting scripts. They should not construct or submit invalid transactions.

[dcoutts]

We should mention the UX downside that this means users have to be aware of a whole new concept: a collateral address, and need to manually move funds into it which are not normally spendable. Are those collateral funds part of your balance or not?

Whereas with the "naive" solution, there are no new concepts for users. There are already rare corner cases where wallets have to rearrange the UTxO to be able to perform certain transactions, and this is visible in the transaction history and for hardware wallet users. Such rearranging transactions are (in principle) already easily identifiable by both software and hardware wallets because all the input and output addresses are within the wallet. So this is already in the UX, but is fortunately rare. There's no clear reason to think that the new corner cases here would be any more problematic.

[SebastienGllmt]

If we decide we don't want to explicitly tell the users about the collateral address existence, that's fine. It can be like how we have external & internal addresses without the user really knowing about their existence.

but is fortunately rare

I don't think this is that rare. When I was working at EMURGO we usually had a few support tickets about this topic per day due to NFTs and the presence of tokens will only increase going forward -- especially with Alonzo

[dcoutts]

I think you should elaborate on how this proposal interacts with multi-sig wallets since it is not clear.

Do the collateral addresses just belong to one party? So this is equivalent of the submitting party having a second personal account (which is of course the primary alternative solution).

[crptmppt]

This PR was briefly discussed during last week's Editor meeting - see notes.
PR104 is set as a REVIEW item for next week's 8/3 Editor meeting 27: if the issue is of relevance to you, consider attending the meeting.

[jonathanknowles]

I just wanted to share a little about how the cardano-wallet team is planning to handle the issue of collateral selection in the immediate future.

I've very interested to hear any feedback that anyone might have.

Initial release

Internally, within the wallet, we have defined a function called selectCollateral, with the following behaviour:

• When given an ordinary UTxO set, selectCollateral attempts to identify the smallest subset c of pure-ada entries such that:
  
  

  sum  c ≥ minCollateralPercent × txFee
  size c ≤ maxCollateralInputs
  

• If the search space is too large, selectCollateral falls back to a strategy of returning larger values (but the number of values will still be bounded by maxCollateralInputs).

When requested to create a transaction requiring collateral, the wallet will use this selectCollateral function to identify the smallest amount it can.

Issues and workarounds

Of course, this creates two potential issues for the user:

• Potential issue 1: the attempt to select collateral fails, because their wallet doesn't have any UTxO entries suitable for collateral.
• Potential issue 2: the attempt to select collateral succeeds, but selects a large amount that the user is uncomfortable with using for collateral. (Even if the risk of losing collateral is extremely low, I can imagine that risk-averse users will still want to minimize the amount used.)

In both of the above situations, the user can manually unblock themselves by using one of the following workarounds:

• Workaround 1: the user (aided by the UI) manually creates an ordinary transaction that adds an amount of ada to the wallet (that they are comfortable to use as collateral)
• Workaround 2: the user (aided by the UI) manually creates a "rebalancing" transaction (one that rearranges the wallet's UTxO, liberating one or more pure-ada entries of a size they are comfortable with).

After the user has added ada entries to their wallet of a size they are comfortable with, the user should be able to rely on the wallet to select them automatically, since the wallet's internal selectCollateral function will always identify the smallest amount it can. (We've worked quite hard to make this happen.)

Here we are also taking advantage of the fact that for any transaction requiring collateral, the set of ordinary inputs can intersect with the set of collateral inputs. So the user should never encounter the situation where they add ada to their wallet, or rebalance their wallet, only to find that the very next transaction (the one requiring collateral) is unable to select these pure-ada entries.

Future releases

The above solution is obviously still not ideal, because it requires the user to manually unblock themselves in the event that they don't have enough "free" ada to use as collateral, or in the event that their wallet only has entries that are "too large" for comfort.

In my personal opinion, it would be ideal if wallets could create these "rebalancing" transactions automatically.

Suppose the user requests the wallet to create a transaction t requiring collateral:

• If the wallet detects that there is enough collateral, and the amount is "reasonably-sized" (for some definition of "reasonably-sized" that is agreeable to the user), then it can just go ahead and return transaction t to the user for signing.
• Otherwise, the wallet can return an ordered pair of transactions [r, t] to the user for signing, where r is a preliminary "rebalancing" transaction that liberates a "reasonably-sized" quantity of ada to be used as collateral, and t is the requested transaction requiring collateral.

As others have pointed out, "rebalancing" transactions should be fairly easy to identify (by both software and hardware wallets) because all the input and output addresses are within the wallet.

I'm hoping that wallet UIs will be able to build a nice UX around these "rebalancing transactions".

[crptmppt]

tentatively CIP-0027

[crptmppt]

tentatively CIP-1856

[crptmppt]

This PR was REVIEWED last week's Editor meeting (27)- see notes.
Consensus is that it should be merged pending final conversations and discussions - Add conversation where needed (thanks @jonathanknowles)
PR104 is set as a Last Check item for next week's 8/17 Editor meeting 28.
If the issue is of relevance to you, consider attending the meeting.

[refi93]

I have one general concern regarding hw wallets - hw wallets don't have the context to safely assure the user what is the collateral amount of the input(s) included as it isn't part of the collateral field when serialized in the transaction and the hw wallet inherently can't trust the software wallet to not include a collateral input too big or from a non-collateral address. hw wallets neither have the capacity to be able to reasonably validate the script to assure that the collateral wouldn't be spent.

The only possible way I can think of to assure the user that the input they are including as collateral is of a certain amount would be letting the hw wallet parse the transaction from which the collateral input came which would be a significant complication as hw wallets would have to be able to parse all possible tx bodies with forward compatibility in mind

[lehins]

Suggested change

	1. Although the rearrange transaction can be created under the hood for in-software wallet, it will confused users when it shows up on their transaction history. For hardware wallets, it will require explicit approval from the user which is also possibly confusing.
	1. Although the rearrange transaction can be created under the hood for in-software wallet, it will confuse users when it shows up on their transaction history. For hardware wallets, it will require explicit approval from the user which is also possibly confusing.

[crptmppt]

This PR was discussed in last week's Editor meeting (28) - see notes.
the PR is tabled for now while the conversation continues.
If you want to advance this PR, consider adding to the conversation.

[refi93]

this requirement isn't currently feasible for Ledger (without the Ledger devs updating the firmware which is closed source) as the Cardano app doesn't have control over the master node derivation (and that's why it wasn't compliant with Icarus so far), or is this paragraph just meant to clarify that the collateral account key has to be derived in the same way as the standard account key was derived so far by given wallet type (Ledger/Trezor/Icarus wallets)? The latter would require significantly less effort given it would be consistent with the current status of key derivations

[janmazak]

I'm not sure this way of treating collateral is relevant any longer. Babbage will come with collateral outputs
https://hydra.iohk.io/build/11759383/download/1/babbage-changes.pdf
and then collateral does not need to be treated separately, any utxos (without datum hashes) would do as collateral inputs, even those including tokens.

[KtorZ]

Closing this one. Feel free to re-open if you want to re-submit as "rejected" CIP to keep record of the discussion.