Testnet4 consensus failure due to timewarp related "softfork"

[Sjors]

#30647 was effectively a soft-fork on testnet4 with no activation plan.

It reduced the timewarp mitigation allowance from 7200 to 600 seconds.

It's enforced by folks running v28.0rc1 and recent master, but not by miners.

This causes new nodes to be stuck at block 42,335: https://mempool.space/testnet4/block/00000000542792e54a720567ba66157d48cdae7bfd01c1b678d0f07a2ed56e99

As observed on BitcoinTalk: https://bitcointalk.org/index.php?topic=5499150.msg64488234#msg64488234 and pointed out by @garlonicon in #30614 (comment)

The simplest solution would for a majority of testnet4 hash power to upgrade to the release candidate and call:

• bitcoin-cli -testnet4 invalidateblock 00000000000000263393ce5f648afd53676f13d360cc9f264b89351623bf1242
• bitcoin-cli -testnet4 reconsiderblock 00000000000000263393ce5f648afd53676f13d360cc9f264b89351623bf1242

This would cause a 16+ hour reorg for any older testnet4 node that's not stuck. Users who upgraded their node after this block was created, and accepted by their node under the old rules, can either wait for this reorg to happen, or also use the above commands.

[Sjors]

Note that the block that upgraded nodes will get stuck on - 42,335 - has difficulty 1. It was probably mined by setting the machine clock 20 minutes in the future. The original testnet4 rules allowed the next block to use the real time. But under the new rules it needs to be 10 minutes in the future (which #30681 takes care of).

[Sjors]

The invalidateblock -> reconsiderblock block approach doesn't work, because the timewarp attack is only checked in ContextualCheckBlockHeader. According to the code documentation -reindex-chainstate won't work either.

bitcoind -reindex does the trick, and only takes a minute on this tiny chain fortunately.

[Sjors]

cc @mempool

[1440000bytes]

cc @mempool

I don't think this notified anyone

[inyellowbus]

bitcoind -reindex does the trick, and only takes a minute on this tiny chain fortunately.

Reindex doesn't work for me. After starting the node, everything still stacks on block 42335. Maybe I need to do something else before starting the reindex? Or register a connection to some explicit node?

[Sjors]

everything still stacks on block 42335.

This is expected. You'll have to wait for a miner to do the same. Welcome to the testnet4 UASF :-)

[achow101]

Anyone have an asic to reorg that?

[Sjors]

It's too hot to turn on my S9 :-)

[russeree]

Anyone have an asic to reorg that?

MARA hash on the way! Luxor also said they are willing to help as well. Shout out to Nick to even turning me onto this problem.

[hashbender]

You have my hashpower 🫡

[1440000bytes]

You have my hashpower 🫡

This will help

[wiz]

ok will do

[hashbender]

Here's my last chain tip in case you want to verify the correct chain. This is from Bitcoin Core version v28.0rc1

hashbender@macbook-pro-124: bitcoin-cli -testnet4 getchaintips
[
  {
    "height": 42375,
    "hash": "00000000000000306de15dd9ee1e6af0461e5bdef12d6dc6be212ca7f5296fa1",
    "branchlen": 0,
    "status": "active"
  }
]

[russeree]

Here's my last chain tip in case you want to verify the correct chain. This is from Bitcoin Core version v28.0rc1

hashbender@macbook-pro-124: bitcoin-cli -testnet4 getchaintips
[
  {
    "height": 42375,
    "hash": "00000000000000306de15dd9ee1e6af0461e5bdef12d6dc6be212ca7f5296fa1",
    "branchlen": 0,
    "status": "active"
  }
]

Got 500TH/s pointed toward T4 UASF

[garlonicon]

I think there is a small risk, that we will see a huge double-spending attempt in testnet4, because after block 00000000000000263393ce5f648afd53676f13d360cc9f264b89351623bf1242, there are transactions in the old chain, which are not picked by nodes, mining the new chain. Which means, that suddenly, when the deep reorg will revert hundreds of blocks, all of those transactions will suddenly become unconfirmed.

So, if anyone has any coins, which were received after block number 42335, then they can be easily double-spent on the new chain, by anyone, just because all of those transactions will vanish. Which means, that as long, as the chain is not yet reorged, people can submit their double spending transactions to the new chain, get them confirmed, and successfully reverse a payment, which had hundreds of confirmations, when a huge chain reorg will kick in.

Also, there is some incentive to download transactions from old node, and pass it secretly to the new node (without broadcasting to the network), because in this way, a miner can get bigger coinbase amount, than other miners.

[remcoros]

I think there is a small risk, that we will see a huge double-spending attempt in testnet4, because after block 00000000000000263393ce5f648afd53676f13d360cc9f264b89351623bf1242, there are transactions in the old chain, which are not picked by nodes, mining the new chain. Which means, that suddenly, when the deep reorg will revert hundreds of blocks, all of those transactions will suddenly become unconfirmed.

So, if anyone has any coins, which were received after block number 42335, then they can be easily double-spent on the new chain, by anyone, just because all of those transactions will vanish. Which means, that as long, as the chain is not yet reorged, people can submit their double spending transactions to the new chain, get them confirmed, and successfully reverse a payment, which had hundreds of confirmations, when a huge chain reorg will kick in.

Also, there is some incentive to download transactions from old node, and pass it secretly to the new node (without broadcasting to the network), because in this way, a miner can get bigger coinbase amount, than other miners.

I don't think there's an incentive problem. Coins have no value and it's easy enough to cpu mine some future blocks if you need tn4 coins.

While it might be annoying for users/developers to have their transactions rolled back, that's a small price to pay for living on the edge and running a node with unmerged code, while there were still discussions going on about a consensus rule change. Bitcoin Core 28.0-RC1 is the first 'official' release with testnet4 support and that includes the time-warp fix.

[russeree]

I think there is a small risk, that we will see a huge double-spending attempt in testnet4, because after block 00000000000000263393ce5f648afd53676f13d360cc9f264b89351623bf1242, there are transactions in the old chain, which are not picked by nodes, mining the new chain. Which means, that suddenly, when the deep reorg will revert hundreds of blocks, all of those transactions will suddenly become unconfirmed.
So, if anyone has any coins, which were received after block number 42335, then they can be easily double-spent on the new chain, by anyone, just because all of those transactions will vanish. Which means, that as long, as the chain is not yet reorged, people can submit their double spending transactions to the new chain, get them confirmed, and successfully reverse a payment, which had hundreds of confirmations, when a huge chain reorg will kick in.
Also, there is some incentive to download transactions from old node, and pass it secretly to the new node (without broadcasting to the network), because in this way, a miner can get bigger coinbase amount, than other miners.

I don't think there's an incentive problem. Coins have no value and it's easy enough to cpu mine some future blocks if you need tn4 coins.

While it might be annoying for users/developers to have their transactions rolled back, that's a small price to pay for living on the edge and running a node with unmerged code, while there were still discussions going on about a consensus rule change. Bitcoin Core 28.0-RC1 is the first 'official' release with testnet4 support and that includes the time-warp fix.

My question is why did we merge code into Core that forked T4 without contacting any of the larger swaths of hashrate and or having the HR to outrun the previously canonical chain?

[achow101]

My question is why did we merge code into Core that forked T4 without contacting any of the larger swaths of hashrate and or having the HR to outrun the previously canonical chain?

There was no indication at the time that anyone would mine blocks that were outside of MAX_TIMEWARP of 600. And generally, the assumption is that if someone is running code from an unmerged PR, they should be following it and be aware that it might majorly change before it is merged.

Also, it's testnet 🤷. We can just make another one.

[benthecarman]

I am on 28.0rc1 and seem to be on a different chain. Block is 00000000000044a7284978db5161b2f4ca012b3fb981a9405a4ce2473f3e4a5f for height 42545, unsure how to get to the main chain

[garlonicon]

unsure how to get to the main chain

You are on the main chain. But the deep reorg didn't kick in yet, so if you use for example mempool.space to see, if you are on the main chain or not, then that page is still showing the old chain. I guess it can take a few days to get there, but we are getting closer and closer, to trigger that reorg.

[russeree]

My question is why did we merge code into Core that forked T4 without contacting any of the larger swaths of hashrate and or having the HR to outrun the previously canonical chain?

There was no indication at the time that anyone would mine blocks that were outside of MAX_TIMEWARP of 600. And generally, the assumption is that if someone is running code from an unmerged PR, they should be following it and be aware that it might majorly change before it is merged.

Also, it's testnet 🤷. We can just make another one.

"the assumption is that if someone is running code from an unmerged PR, they should be following it and be aware that it might majorly change before it is merged."

I thought about this earlier and agree with this statement. There is no difference other than subtlety of timestamp validation and just using a new genesis block. Either way I would have been forked.

[garlonicon]

Also, we can now clearly see, why weak blocks are needed: mempool.space shows more than 500 transactions, waiting for being confirmed, but only a few of them are really mined, and the rest is waiting patiently, because nobody rebroadcasted them.

And there is more: it is possible to easily double-spend them, because it will take some time, before they will be shared in P2P way.

It is also something to consider, in case when mempools are congested: if you can see a miner, sending blocks with a low health, it doesn't always mean censorship. It can also mean, that it was just recently started, and then, you can pass a low fee transaction, directly to this node, and get it confirmed, even though there will be a lot of higher paying transactions. Because if they are not re-broadcasted, then they are "invisible" for newly started nodes.

[Sjors]

@garlonicon I think you're going a bit off topic here, a weak blocks proposals seems more suitable for the mailinglist.

[achow101]

My nodes' logs indicate this was resolved at 2024-09-05 02:06:40 UTC when the new chain reorg'd the timewarped one.

[garlonicon]

I guess some ASIC miners still didn't upgrade their nodes:

$ getchaintips
  ...
  {
    "height": 50831,
    "hash": "000000000060a808a1067bed2feb0b22261383bb0f9b6e3f0dee2436eb8e5048",
    "branchlen": 432,
    "status": "headers-only"
  },
  {
    "height": 50826,
    "hash": "000000000035411cdce9394ab799205c7dd0ce99fd23b23f9481e591635fd0f2",
    "branchlen": 427,
    "status": "headers-only"
  },
  {
    "height": 50817,
    "hash": "0000000073a34762fc9f48419a7fba80c17783398848d8627b620cef7cd61bb4",
    "branchlen": 1,
    "status": "valid-headers"
  },
  {
    "height": 50811,
    "hash": "00000000000105aeeb41726a8cca80aa0515cce83fbbb6042d90129173bdd2ca",
    "branchlen": 412,
    "status": "headers-only"
  },
  ...

Over 400 different blocks. So, it seems that some forks are still there. This time at block number 50400.

[murchandamus]

@wiz, @russeree: Are you still mining on Testnet4?

[wiz]

I figured the community had started mining it by now - will take a look

[russeree]

@wiz, @russeree: Are you still mining on Testnet4?

Currently no, I assumed honest HR had started mining on T4 now so I pointed my miners back at Mainnet. Should I reconsider?

[murchandamus]

I have the impression that the person that was selling testnet3 coins may be the one that has monopolized mining testnet4 now