Websocket: newBlockHeaders does not return canonical blocks

[ankitchiplunkar]

Hi there,

System information

Geth version: 1.8.17-stable-8bbe7207
OS & Version: Ubuntu 18.04

Expected behaviour

The websocket when subscribed using newBlockHeaders should return the forked blocks and also subsequently the canonical blocks.

Actual behaviour

The websocket when subscribed using newBlockHeaders does not return the canonical blocks (for almost every ~200 blocks)

Steps to reproduce the behaviour

I have a local geth node, am using web3.js for subscribing to newBlockHeaders, below is the code to get new blocks.

var Web3 = require('web3')
const fs = require('fs');
var web3 = new Web3();

web3.setProvider('ws://localhost:8546');

var subscription = web3.eth.subscribe('newBlockHeaders', function(error, result){
    if (!error) {
        console.log(result['hash']);

        return;
    }

    console.error(error);
}).on("data", function(blockHeader){
    console.log(blockHeader['number']);
    strData = "data, " + blockHeader['hash'] +', ' + blockHeader['number'] + ', '+ blockHeader['parentHash'] + "\n"
    fs.appendFile('test_pubsub.txt', strData , function (err) {
        if (err) throw err;
      });
}).on("changed", function(blockHeader){
    console.log(blockHeader['number']);
    strData = "change, " + blockHeader['hash'] +', ' + blockHeader['number'] + ', '+ blockHeader['parentHash'] + "\n"
    fs.appendFile('test_pubsub.txt', strData , function (err) {
        if (err) throw err;
      });
}).on("error", console.error);

most of the time the code works as expected, but sometimes (~ 1 block in 200 blocks) the subscription returns non-canonical blocks. Below is an example where the blockhash of 6646702 is different than parenthash of 6646703.

data, 0xaf203a61a22c187155feb4923bb82f88ae2476362c1c6b1d6076770cc604fdfc, 6646702, 0x4116633d815399f3a361207b60c689d1b824d987b5c4ecb0ecd6f8ea846ecd9b
data, 0x42181846ab8ba4195b88226515e6cb6af5720e87094e2ed8d31bbc383c405971, 6646703, 0x51e788e53ca7c434441a4fc6f26cd2b6cf8e6da77dc6d71b0842a3bbf6699910

This is similar to an earlier reported issue #18026

Is it possible to get canonical blocks from newBlockHeaders method without relying on using delayed methods like eth_getBlockByNumber?

[jpzk]

This is an important issue, I believe many companies rely on this. It's also a major differentiator if it works in Geth, also Parity is unreliable.

[karalabe]

What you're probably experiencing is a reorg. Your chain gets a new head block 6646702, but at the same time a miner also mines a side fork at the same height. The network however decides to continue on the later mined version of 6646702, and produces 6646703. The parent will be rightfully different because a mini reorg happened.

The subscription only reports on new heads. If the parent is not the old head, it's up to you to handle the reorg. Does this explain the oddity you are experiencing?

[ankitchiplunkar]

I agree that we are experiencing a reorg. But for some blocks we never see the canonical block in the stream.

Most of the time, the subscription returns canonical and non-canonical (forked) blocks, for example block 6646960 (below) where we see 3 different blocks being returned.

data, 0x73ac9cdef479466757e65e6a08d6d11b6d8a85b9f1840fd407515548d6a0bb1c, 6646959, 0x587954140c6aac0a3d77a615df0e665b9673c72d80107a19fb8800db6d6ee837
data, 0x2dd9db2b96a58b66105b180f1c9b4230389fe75a88a4be40430f0e1518337801, 6646960, 0x73ac9cdef479466757e65e6a08d6d11b6d8a85b9f1840fd407515548d6a0bb1c
data, 0x9dc85008ec18b31e4230cd7f4dca4088be84e939316392b19d40dced2a875f0c, 6646960, 0x73ac9cdef479466757e65e6a08d6d11b6d8a85b9f1840fd407515548d6a0bb1c
data, 0xc7e8f9a67efb4443e69fd99fcecb4a02fbf65e587daa37ed421ad9c448b805bc, 6646960, 0x73ac9cdef479466757e65e6a08d6d11b6d8a85b9f1840fd407515548d6a0bb1c
data, 0xaebf65fb2942daadb47d6ad11cdfc4e90dbbc6c7d8be040b24d1976e1b50dd17, 6646961, 0x2dd9db2b96a58b66105b180f1c9b4230389fe75a88a4be40430f0e1518337801

But for some cases subscription never returns the canonical block. This is the main problem.

The subscription only reports on new heads

If this is the case then newBlockHeaders should return all valid (correct nonce, correct difficulty, correct root hashes etc) block headers that is gets from the p2p network. Or atleast return the canonical block when geth discovers that network has started building on another chain. I am guessing when the node discovers that it is on a forked chain it would import the canonical block from the p2p network.

[jpzk]

FYI The parity team is on it, flagged as ASAP and will be fixed shortly.

openethereum/parity-ethereum#9865

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[holiman]

Afaict, this is by design.

See https://github.com/ethereum/go-ethereum/blob/master/core/blockchain.go#L1519 , we only fire one event for new heads. So if we import 100 blocks in a batch, it will only fire one event.

However, we will fire side-chain events for all the old blocks that were removed: https://github.com/ethereum/go-ethereum/blob/master/core/blockchain.go#L2056 . Note, though, that that info is sent on another channel: bc.chainSideFeed.Send(ChainSideEvent{Block: oldChain[i]})

[holiman]

The subscription newBlockHeaders is not a native geth-subscription, seems to be a web3-specific thing which uses something else from geth under the hood.
Docs: https://web3js.readthedocs.io/en/v1.2.0/web3-eth-subscribe.html#subscribe-newblockheaders

Subscribes to incoming block headers. This can be used as timer to check for changes on the blockchain.

It does not say that every block/header will be delivered, but that it can be used to react to changes (new heads)

[adamschmideg]

We'll need to update Real-time Events | Go Ethereum