Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Websocket: newBlockHeaders does not return canonical blocks #18032

Open
ankitchiplunkar opened this issue Nov 5, 2018 · 4 comments

Comments

@ankitchiplunkar
Copy link

commented Nov 5, 2018

Hi there,

System information

Geth version: 1.8.17-stable-8bbe7207
OS & Version: Ubuntu 18.04

Expected behaviour

The websocket when subscribed using newBlockHeaders should return the forked blocks and also subsequently the canonical blocks.

Actual behaviour

The websocket when subscribed using newBlockHeaders does not return the canonical blocks (for almost every ~200 blocks)

Steps to reproduce the behaviour

I have a local geth node, am using web3.js for subscribing to newBlockHeaders, below is the code to get new blocks.

var Web3 = require('web3')
const fs = require('fs');
var web3 = new Web3();

web3.setProvider('ws://localhost:8546');

var subscription = web3.eth.subscribe('newBlockHeaders', function(error, result){
    if (!error) {
        console.log(result['hash']);

        return;
    }

    console.error(error);
}).on("data", function(blockHeader){
    console.log(blockHeader['number']);
    strData = "data, " + blockHeader['hash'] +', ' + blockHeader['number'] + ', '+ blockHeader['parentHash'] + "\n"
    fs.appendFile('test_pubsub.txt', strData , function (err) {
        if (err) throw err;
      });
}).on("changed", function(blockHeader){
    console.log(blockHeader['number']);
    strData = "change, " + blockHeader['hash'] +', ' + blockHeader['number'] + ', '+ blockHeader['parentHash'] + "\n"
    fs.appendFile('test_pubsub.txt', strData , function (err) {
        if (err) throw err;
      });
}).on("error", console.error);

most of the time the code works as expected, but sometimes (~ 1 block in 200 blocks) the subscription returns non-canonical blocks. Below is an example where the blockhash of 6646702 is different than parenthash of 6646703.

data, 0xaf203a61a22c187155feb4923bb82f88ae2476362c1c6b1d6076770cc604fdfc, 6646702, 0x4116633d815399f3a361207b60c689d1b824d987b5c4ecb0ecd6f8ea846ecd9b
data, 0x42181846ab8ba4195b88226515e6cb6af5720e87094e2ed8d31bbc383c405971, 6646703, 0x51e788e53ca7c434441a4fc6f26cd2b6cf8e6da77dc6d71b0842a3bbf6699910

This is similar to an earlier reported issue #18026

Is it possible to get canonical blocks from newBlockHeaders method without relying on using delayed methods like eth_getBlockByNumber?

@ankitchiplunkar ankitchiplunkar changed the title Websocket: newBlockHeaders does not returh canonical blocks Websocket: newBlockHeaders does not return canonical blocks Nov 5, 2018

@jpzk

This comment has been minimized.

Copy link

commented Nov 5, 2018

This is an important issue, I believe many companies rely on this. It's also a major differentiator if it works in Geth, also Parity is unreliable.

@karalabe

This comment has been minimized.

Copy link
Member

commented Nov 7, 2018

What you're probably experiencing is a reorg. Your chain gets a new head block 6646702, but at the same time a miner also mines a side fork at the same height. The network however decides to continue on the later mined version of 6646702, and produces 6646703. The parent will be rightfully different because a mini reorg happened.

The subscription only reports on new heads. If the parent is not the old head, it's up to you to handle the reorg. Does this explain the oddity you are experiencing?

@ankitchiplunkar

This comment has been minimized.

Copy link
Author

commented Nov 7, 2018

I agree that we are experiencing a reorg. But for some blocks we never see the canonical block in the stream.

Most of the time, the subscription returns canonical and non-canonical (forked) blocks, for example block 6646960 (below) where we see 3 different blocks being returned.

data, 0x73ac9cdef479466757e65e6a08d6d11b6d8a85b9f1840fd407515548d6a0bb1c, 6646959, 0x587954140c6aac0a3d77a615df0e665b9673c72d80107a19fb8800db6d6ee837
data, 0x2dd9db2b96a58b66105b180f1c9b4230389fe75a88a4be40430f0e1518337801, 6646960, 0x73ac9cdef479466757e65e6a08d6d11b6d8a85b9f1840fd407515548d6a0bb1c
data, 0x9dc85008ec18b31e4230cd7f4dca4088be84e939316392b19d40dced2a875f0c, 6646960, 0x73ac9cdef479466757e65e6a08d6d11b6d8a85b9f1840fd407515548d6a0bb1c
data, 0xc7e8f9a67efb4443e69fd99fcecb4a02fbf65e587daa37ed421ad9c448b805bc, 6646960, 0x73ac9cdef479466757e65e6a08d6d11b6d8a85b9f1840fd407515548d6a0bb1c
data, 0xaebf65fb2942daadb47d6ad11cdfc4e90dbbc6c7d8be040b24d1976e1b50dd17, 6646961, 0x2dd9db2b96a58b66105b180f1c9b4230389fe75a88a4be40430f0e1518337801

But for some cases subscription never returns the canonical block. This is the main problem.

The subscription only reports on new heads

If this is the case then newBlockHeaders should return all valid (correct nonce, correct difficulty, correct root hashes etc) block headers that is gets from the p2p network. Or atleast return the canonical block when geth discovers that network has started building on another chain. I am guessing when the node discovers that it is on a forked chain it would import the canonical block from the p2p network.

@jpzk

This comment has been minimized.

Copy link

commented Nov 19, 2018

FYI The parity team is on it, flagged as ASAP and will be fixed shortly.

paritytech/parity-ethereum#9865

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.