Investigate possibility of reducing 10-blocks lock

[erciccione]

The 10-blocks lock is the most problematic aspect of Monero, which heavily impacts its usability as a currency and the user experience of people using services based on Monero. #95 explores the possibility of removing the lock, which would be the optimal solution, but the problem is not simple to resolve and the total removal might happen in a far future.

It worth considering if there are the premises for reducing the lock to a smaller amount of blocks, without impacting the security and stability of the network. A couple of questions to get the conversation started:

• Did we ever get close to the 10 blocks limit during a reorg?
• What's the highest number of blocks lower than 10 that could be acceptable for the lock?

Note that even a reduction from 10 to 8 blocks would be a significant UX improvements for Monero users.

[SChernykh]

I've just checked my node's logs as far back as 2021-08-31 (I don't have older logs), and the largest reorg was 3 blocks deep:

2022-04-10 17:26:24.193	[P2P9]	INFO	global	src/cryptonote_core/blockchain.cpp:2093	 alternative blockchain size: 3 with cum_difficulty 186060274402189254

[erciccione]

Thanks @SChernykh, would be very useful to know if that reflects the situation for the entire history of Monero or at least the last years. It's already very good that the largest reorg in the last year was only 3 blocks deep.

[SChernykh]

I don't have logs anymore, but I vaguely remember seeing some bigger reorgs before, but they never reached 10 blocks. Unfortunately I don't remember if they reached 8 blocks.

[SamsungGalaxyPlayer]

I'm okay with >0 reorgs that are 10+ blocks long. We can't avoid this issue without having an infinite funds lock time, which is impractical.

Thus, we're always picking a reasonable value.

Lowering the block time could have the following downsides:

1. Requires changing the default decoy selection algorithm, and requires sufficient testing for this to ensure it works properly.
2. Some clients could use an old algorithm, resulting in different selection models. MyMonero comes to mind, which has its own implementation. This is possible in any case, it's just exacerbated with the transition. Anyone who uses a custom client will need to do part 1 above also.
3. Setting too low a value could enable privacy downsides during re-orgs.
4. Setting a low value could reduce privacy for quick spends even with a good algorithm. Currently, everyone who wants to spend quickly spends around blocks 10-20, which should be easier to fit an accurate decoy selection model to than 5-20, for example.

Speaking completely without any numbers (:p), I would like to set a goal of cutting the lock time in half to 5.

[UkoeHB]

context from @hyc

fyi, my linode monerod has ~5 years worth of logs, 1082 reorgs. all but 4 had size 2, those 4 had size 3.

[Rucknium]

This issue was discussed at the MRL meeting:

monero-project/meta#706 (comment)

[erciccione]

fyi, my linode monerod has ~5 years worth of logs, 1082 reorgs. all but 4 had size 2, those 4 had size 3.

This is very encouraging, as we didn't get even close to half the current block limit in 5 years. I would say the first 3 years have relatively less importance, since the network was much smaller and not yet robust.

Are there attack vectors that could end up creating reorgs (beside a 50%+1 attack, which is out of scope)? That seems to be the main issue, given that during normal operation of the network reorgs were never more than 3 blocks deep.

This issue was discussed at the MRL meeting:

@moneromooo-monero mentions smooth was strongly against lowering the blocks to <10 some time ago. Do we know the reasoning?

[chaserene]

@hyc could you share an extract of your Linode logs that includes reorg events, their time and their depth? seeing how the size and the frequency of reorgs changed over time, maybe if there is a general trend or a correlation with hash rate/upgrades/etc, could help making an informed decision about what level of reorg probability we are comfortable with.

[hyc]

http://highlandsun.com/hyc/reorgs.txt currently contains 1219 lines, 160KB. All but four reorgs are size 2, four are size 3.

But I'd be dubious about using this as a metric in deciding how to change the lock. If we reduce it significantly below 10, that will incentivize attackers to try to double-spend.

[chaserene]

thank you!

from lines 395 to 1076 (both inclusive), there are only a few dates for all entries, which seems to be something other than when the reorg happened (maybe that's when the log file was exported). do you see a way to assign actual dates to those?

also, among those records (so the date is not indicative), I've found a real outlier, supposedly a 26-block reorg (L751):

bitmonero.log-2018-09-06-21-22-35: alternative blockchain size: 26 with cum_difficulty 14334003971616801�[0m

does anyone remember this occurrence? (I don't.)

I'd be dubious about using this as a metric in deciding how to change the lock. If we reduce it significantly below 10, that will incentivize attackers to try to double-spend.

I agree in that the maximum 3 block depth in this data, presuming there were malicious attempts among those, doesn't mean that a longer-range attack is uneconomic for current potential attackers. but they are always incentivized to some extent to doublespend, even now. only an infinite lock period can remove this incentive. it's that with a shorter one, they are incentivized more. 10 blocks has worked out so far, that's valuable empirical data. reducing it to let's say 4 would be clearly reckless, but I do wonder if it would be safe to go below 10.

it's possible that there is no way to know the answer. other than this data set, the only useful source I can think of right now is making a minor reduction in the lock time, collecting reorg data in that environment for an extended period, and then reassessing.

since in general there is no specialized mining hardware, I would assume the wide availability of hardware ready to enter mining, so this decision definitely needs to be made with caution.

[chaserene]

here are some quick-and-dirty visualizations that no one asked for from hyc's data. I know this reflects only a single node's view of the network, but this is the best I've encountered so far.

absent further clarification, I excluded reorgs that didn't have a definitive date to them (lines 395-1076, inclusive), because obviously I can't plot them on a time axis without knowing when they occurred. this set amounts to ~56% of the total registered reorgs (682 / 1219). however, I don't think this will skew the data in a significant way, because 1) the rest of the data that has proper dates begins at 2019-10 (for the first chart, I used only full months, which includes 531 reorgs), and 2) based on the log file names in the log, 605 of the "dateless" reorgs occurred before 2019-06-11, so they don't interfere with the visualized period. the other 77 "dateless" reorgs happened sometime between 2019-06-11 and 2020-12-25. this means that at most 77 / (531 + 77) ≈ 13% of the picked up reorgs are missing from the charts, and they are limited to the period up to 2020-12-25.

all reorgs with a definitive date have a reported depth of 2 blocks.

the first chart shows how many reorgs were picked up per month (2019-11--2023-03), overlaid with the hash rate from Bitinfocharts.

the amount of reorgs seems to strongly correlate with the hash rate. Carbon Chameleon's PoW change is responsible for the large hash rate jump there. a reduction in reorgs seems to co-occur with it. this can't be clearly stated because some of the "dateless" reorgs may have happened between Carbon Chameleon and 2020-12-25, though I'm not sure that even all 77 "dateless" reorgs could erase that significant reduction.

the correlation seems to weaken after Fluorine Fermi and the amount of reorgs falls significantly.

I wonder if these major reductions are due to networking updates in the forks or something else.

the second chart is a shot at visualizing all reorgs with a definitive date (2019-10-29--2023-04-20, inclusive), each reorg being a dot and the Y axis showing how many hours elapsed since the last reorg, sort of a "time between failures reorgs".

this chart may not be as useful, but it seems that the time between reorgs kept improving after Fluorine Fermi against a relatively stable hash rate.

these are all subjective observations, I'm sure that with proper statistical methods one can glean better insights.

[Gingeropolous]

the noncesense lab has stuff on re-orgs.

https://github.com/noncesense-research-lab/archival_network

I don't know where the actual data or results are though.

[chaserene]

thanks @Gingeropolous. I guess the relevant data is buried in /raw_log_dumps. it would be interesting to see if it matches up with hyc's data, but for now there are serious doubts that a reduction could be done safely, no matter how good the recent results are (see the chat log of today's MRL meeting).

edit (2024-03-29): I noticed the above logs have @hyc saying it's dangerous to go below 10 blocks, but his precise reasoning happened outside the meeting hours. I'll offer a summary from memory, which you may be able to find in the full MRL chat logs, unfortunately I couldn't (but it happened around those days):

he basically said the risk of a reorg will increase unpredictably by reducing the lock period. years of experience show that 10 is safe, but reducing even just to 9 may cause catastrophic failures. it's like you're standing blindfolded near a precipice, and you know that the edge is at most 10 steps away, but you can't see where exactly. any step forward may cause your downfall.

[r4v3r23]

has https://github.com/monero-project/research-lab/issues/95#issuecomment-1197336438 been looked into?

[chaserene]

@r4v3r23

(correct link: #95 (comment))

I'm far from having processed the whole problem space, but I think it's a smart proposal, at the cost of protocol complexity that's not negligible nor crushing. I encourage reading it and comments after it through 2022-12-29, they help understanding its strengths and weaknesses. the proposal is in the 10-blocks lock elimination thread, but in practice it can only reduce it, hence relevant here ("in practice you would only select decoys from blocks older than some threshold larger than the average time it takes to propagate a double-spend report").

I like that it can (almost always) work without affecting transaction uniformity. it still troubles me that it requires assumptions about network behavior that we can't estimate the probability of, nor do we have enough historical data/experience about. but my takeaway is that it may be impossible to reduce the length of the lock without such assumptions.

going that route can lead to breaking aspects of the network. to move forward, I think we'd need a way to quantify the probability of transaction invalidation through a reorg with the current protocol, and see if an alternative model can reliably bring the same level with certain parameters (if any).

off-topic: the conservative in me wants to see the network as robust as possible and solve fast spendability rather on a layer-2. on the other hand, payment channels on Monero are currently only theoretical (I know about five papers and that's it) and rollups haven't even been seriously theorized. it would also prove useful to not fragment the network into layers.

[chaserene]

potentially useful regarding quantification (h/t to Rucknium for finding it):
Isthmus: Framework for assessing safe lock times based on worst case plausible scenario (draft, October 2019)

[shortwavesurfer2009]

Probably a very dumb question, but as a regular user I'm going to ask it anyway. Why not move stage net or test net to say eight or nine blocks and run it and see what happens? After all, isn't that what they are their for, to test experiments with? Either that or maybe Wownero may be willing to try it out

[chaserene]

@shortwavesurfer2009 not a dumb question. in my view, that wouldn't help a lot for the following reasons:

• I haven't experimented with stagenet or testnet, but I would bet that the network load is different. block size is a factor in reorg frequency and depth.
• the actual network topology is different. not everyone who's running a mainnet node runs a stagenet/testnet node. Wownero probably also differs a lot.
• incentives are important, think of e.g. a selfish mining scenario. stagenet/testnet coins have no economic value, and WOW is a different asset, so the incentives are different.

[tevador]

has #95 (comment) been looked into?

FYI, this proposal should be considered obsolete because it can't work with full-chain membership proofs (FCMPs).

With FCMPs, the decoy set will be defined by a reference block and will contain all outputs created up to (and including) that block.

Zcash works nearly the same way (they call their reference block an "anchor"). Interestingly, they don't have a consensus-mandated lock time and simply expect users to resubmit any transactions invalidated by reorgs. Some zcash wallets have a voluntary lock. Here is the relevant discussion: zcash/zcash#1614