the RPC should have commands to specify the side of a fork considered 'current'

[kanoi]

As bitcoin stands at the moment, a standard orphan race is decided simply by the first block received above the 'current' height - of course this process can continue if both forks grow at the same rate, but the concept and the solution are the same for multi-block forks up to a point
When the orphan is resolved by the 'higher' block found, if that block is on a different valid fork to the what bitcoin considers the current fork, it will simply switch to the other fork.

It would be very useful to have an RPC command to switch which fork bitcoin considers is the current fork - probably by choosing the block hash at the top of the fork, if bitcoin knows multiple forks at the same height
In the case of network issues about choosing a fork, this would provide a very simple method to resolve which side of the fork a miner using bitcoin is mining on.
This would of course add complexity and require a different command if the fork chosen was at a lower level, but that could also be handled by being able to state that a specific block (in a fork) is invalid and should be ignored - and thus give a method to effectively invalidate the fork.

I'm not sure what the RPC submitblock does with a block when it receives one, on or creating a different fork, but of course it would need to accept a block on a different fork if it doesn't already.

[laanwj]

I'm not sure the use of this is in practice, besides testing,but you can effectively already achieve this with invalidateblock and reconsiderblock. Invalidate the forks that you don't want to be on - and switch back by validating the blocks again.

[kanoi]

Yep, indeed those 2 commands do have a method for doing what I've suggested.
Heh, they've only been around for a ... year ... guess I should look closer at the code changes :)

I'd guess that a switch from a 'valid' fork would be done by invalidating then revalidating a block in the 'other' fork, and an 'offending' fork switch (for a fork that was a network issue like the v3 fork and you wouldn't want to ever continue) would simply be to invalidate a block in the 'offending' fork.

[gmaxwell]

@laanwj thats a pretty dangerous suggestion (and the risk of miners doing that was a reason we considered leaving out out those RPCs in the first place)... because in the common case that the fork you heard first was the actual longer one if your software screws up and you fail to revalidate it, you'll continue on forking the network.

The normal reason for asks like this is because of a high-ish hashpower miner who found a tying block moments after they had already accepted a first longer block from the network (due to hardware and submission delays); and rather than accept their defeat gracefully they'd like to attempt a momentary majority hashpower attack and try to get lucky excluding the other block. If you simulate this behavior of selfishly mining on your own later blocks, for many hashrate distributions and latency choices it gives massively worse convergence times for the network. (I think for a toy example of two 50% miners and a 30 second submission latency it increased the frequency of 6+ deep reorgs 100 fold or something insane like that)

Perhaps it would be better to focus more on cutting down on the submission latency side of it, since asking miners to not behave in a way thats adversarial towards the network may not be sustainable-- but historically many hardware vendors have not cared too much about eliminating these latencies.

[gmaxwell]

Based on this thread I think we need to either remove these debugging RPCs in further releases or provide an additional RPC that twiddles the time ordering without invalidating. Having miners deploy half tested software to prefer their own blocks using invalidateblock would be a disaster waiting to happen which we should have nothing to do with.

[TheBlueMatt]

ACK. I would love to remove those RPCs, but I don't think we can, so a time-order-twiddling RPC (that only allows twiddling by 5 seconds or something similar) might be better than the alternative :(

On November 12, 2015 1:48:54 AM PST, Gregory Maxwell notifications@github.com wrote:

Based on this thread I think we need to either remove these debugging
RPCs in further releases or provide an additional RPC that twiddles the
time ordering without invalidating. Having miners deploy half tested
software to prefer their own blocks using invalidate block would be a
disaster waiting to happen which we should have nothing to do with.

---

Reply to this email directly or view it on GitHub:
#6995 (comment)

[kanoi]

Ah, so suddenly we have the implication that the person who allowed these RPC calls into bitcoin should not have? Interesting.
Though I do wonder why it is considered half tested software if it is part of the release code.

Yet I've stated a clear use case that is advantageous to the bitcoin network

I've no specific idea how bitmain and f2pool did resolve their v2/v3 bitcoin states in the biggest fork recently, but when I contacted bitmain and asked them to switch away from their v2 fork, it did indeed take a while for them to do that.
I wonder if they too (like me) didn't realise these commands existed and thus the problem lasted longer than if they had known about these ...

[sipa]

Would a patch to add an RPC 'preferblock' be acceptable, that makes it win from all other blocks with the same work?

[kanoi]

I'd guess in that case the problem would be if someone on an 'offending' fork wanting to switch to the 'considered correct' fork but the 'offending' fork was higher, then (as stated) it would indeed be possible with the current RPC

[gmaxwell]

@kanoi You mean me? As I was the big prodding force to get these in. No-- I didn't say they shouldn't have been done, but rather we put them in knowing that the network was currently vulnerable to at the time non-public forking vulnerability (the one that just split PPC recently) and I was concerned that given the difficulty of contacting miners and the problems with reindexing that happened with the March 2013 fork we needed a tool available that would allow switching forks manually without reindexing, but couldn't even disclose the need. Turns out they're super useful for testing too.

But now that they've been suggested for overriding the fastest convergence behavior that suggestion can't be taken back, and we need to provide a safe way of doing that (preferblock), because doing it via invalidate is not generally safe. Thats all, no big deal. Even 'preferblock' is not all that safe: as I mention, in many network typologies preferring your own blocks over tied earlier ones has really bad impacts on convergence; but we can't stop people from doing that. We can at least reduce provide a way that limits the harm to an increase in large reorgs rather than outright network forking.

In the case of f2pool, their bitcoind was on the correct chain; but other mining software which did not enforce any network validity rules was ignoring it and extending another chain because it had more blocks. This didn't help there.

[laanwj]

I was kind of wary to add these RPC calls in the first place. But it's impossible to keep people from doing bad stuff with them. Even if they were to be removed, miners could just patch their node software to include them. They could implement similar functionality even before it was considered for Bitcoin Core.

I think it is good to have this discussion though, at least now it is documented publicly that it is a bad idea and why.

[kanoi]

What is PPC?

I'm wondering why orphan races are considered to be such an issue that it would prompt suggestions to change the release code and thus try to ban access to an interface (that is already available)

I'd consider exaggeration for effect not really a good argument.
5 or 6 depth forks are ever so unlikely that they should only ever to be likely to occur when there is a network problem, and this interface would indeed be the fastest solution to that problem.

@gmaxwell - I'm not sure why you brought that up, but I've posted details on the forum and told many manufacturers about the importance of latency and block change handling in their miners for many years.
https://bitcointalk.org/index.php?topic=294499.0
Con and I implemented the original code for dealing with that in cgminer (a certain 'other' person wouldn't make his changes, he made for that, public after I asked him to implement it in his BFL driver ...)

[gmaxwell]

PPC is some altcoin based on the bitcoin core codebase; and like most others hasn't kept up with fixes.

If miners prefer their own blocks over earlier received ones, then for many combinations of network topology and miner latency this results in significant increases in moderate depth to long reorgs. This is because in a near tie the winning block doesn't just have to win early enough to overcome network and validation latency and be heard uniformly before a competitor, it has to to that by more than the mining stack latency (which can be rather large). When that doesn't happen the fork persists when it otherwise would have resolved, and the result is a longer reorg. The effect is as big as it is, due to the exponential distribution of times between blocks means that a lot of blocks are 'fast'.

The comment here was not on that but that now that breaking ties with invalidation was suggested I think we really need to provide a better interface or get rid of the dangerous one---- we probably should have all along but it hadn't come up; even though it hurts convergence people will do it anyways and they might as well do it safely. Though possible to do via invalidateblock it is quite tricky to get that right, and the result will be miners that refuse to mine on the longest chain even once theirs has fallen behind (e.g. due to state management and details like if they've invalidated they won't even see the longer extension, so they won't know they've fallen behind and will continue to try to mine a fork), which will leave lite clients more vulnerable to double spending and create other issues. Since it sounds like people are going to do that we should provide an option (preferblock) which is less likely to shoot miners (and the network as a whole) in the foot and completely break things. :)

[kanoi]

Actually I'm not at all convinced by your lack of numbers and trying to convince people with vague mathematics.

Concentrating on some tiny picture and ignoring the big picture isn't gonna help much.

You'd have to show what the 'full' current situation is (i.e. not ignore relevant issues and numbers) and what the expected increase is and how that increase is so terrible.
Vaguely it looks like a tiny effect on the network due to a rare to occur event that would result in success only the % size of the pool that used it, and then result in an extra 2 block reorg like any current orphan does, in those rare occasions.
Anything beyond that would be so much more rare that you should argue the things that are actually quite large and have been ignored for years and do something about them instead.

The hardware issue itself is minor in comparison and also over time will only get better since no one is out there trying to make slower worse hardware ... and as block sizes increase and nothing is done about the important issues, those stale shares from hardware will become less and less significant.

Amazing being able to suggest you want this RPC removal change and make it sound like it could happen quickly when there are much larger issues directly related to this that have been known for years and very little done about them.

The big issues would be block confirmation times in bitcoin, getblocktemplate times, internet latency (that there is no direct solution to) and also block distribution.
Interestingly I discussed with gavin, back in 2012 I think it was, ideas about block distribution and wasted re-validation of transactions and transmission of the entire block and all it's transactions unnecessarily ... and it only took ... until 0.10.0 for similar such changes to finally occur ... without hard forks.
The issues with getblocktemplate have been known for many years and nothing done about it.
Block confirmations ... well you're considered to know about the maths in that, show that your worthy of that title and make it faster :)

[sipa]

See #6996 for an RPC to pick a block to prefer in case there are competing blockchain tips.

[laanwj]

The issues with getblocktemplate have been known for many years and nothing done about it.

You don't understand how an open source project works. If you want something to be done, you have to either do it yourself, or possibly pay/hire someone to do it. Just talking about it or waiting expectantly won't help. Everyone, including @gmaxwell, decides for themselves what they want to work on, despite what you think priorities should be.

[sipa]

In fact, something is being done about it: #6898.

[kanoi]

@laanwj Indeed they do, just as I do on the opensource project of the most used miner (cgminer) and my other open source project. Yet I still do have an opinion on what needs to be done and what problems there are in bitcoin for a long time and have been ignored :) Luckily I'm also free to have that opinion ... I hope :) (and my decisions about what I think are best for my opensource projects aren't decided by who pays)

[kanoi]

So, anyone considering trying the invalidateblock followed by a reconsiderblock:
It would indeed appear to be that the code wasn't tested all that well after all.

There was an interesting orphan race today ... between AntPool and Bitfury ... where the outcome was worthy of conspiracy theories:
383291 BWPool
383292 an orphan race between AntPool and Bitfury -> AntPool won ... but ...
383293 Bitfury (the one that confirmed the orphan race block ... of AntPool not Bitfury)
383294 Bitfury

Anyway, while this was happening I tried "invalidateblock ...00005df7..." on a home bitcoind with my home test environment of CKPool running to see what would happen. As expected the 'inactive' fork became the active one (after a VERY long delay of about 4 miutes). However, I left it like that when another block arrived on the ...00005df7... fork to see what would happen - and indeed it stayed where it should have. 3 blocks later after 383295 appeared, I tried "reconsiderblock ...00005df7..." but it didn't correct the confirmation counts of a transaction in qt and "getinfo" still reported 383292 as the current block. Fortunately a bitcoind restart fixed it, so it must have been the internal state of bitcoind was messed up.

Reason for mentioning that here is so that anyone considering it should test it well before using it and might find it doesn't work.

However, the important use of invalidateblock (to solve a network problem) seems to work ok.

[gmaxwell]

I think that is NotABug; it's specifically part of the behavior that I was worrying about when thinking of people trying this for preferences.

I attempted to explain that behavior in my post above "if they've invalidated they won't even see the longer extension, so they won't know they've fallen behind": My expectation is that it would have stayed on 383292 until a new block caused to to actually go fetch the children of 00005df7 which it currently wasn't fetching because they were extensions of an invalid chain.

[maflcko]

The feature request didn't seem to attract much attention in the past. Also, the issue seems not important enough right now to keep it sitting around idle in the list of open issues.

Closing due to lack of interest. Pull requests with improvements are always welcome.

[MentalNomad]

Agree with the closure; wanted to comment that the lack of activity likely stems from the fact that the feature request seems to have been addressed in Add preciousblock RPC #6996, as referenced by Sipa here:

See #6996 for an RPC to pick a block to prefer in case there are competing blockchain tips.