Immediately broadcast blocks on arrival

[zhyatt]

As part of the V21.3 service release the rebroadcasting of blocks upon immediate arrival was removed (specifically commit c21ec84) but this change reduces block propagation and may be contributing to delays in non-PR nodes receiving blocks initially. This rebroadcasting of blocks immediately on arrival should be added back in to improve block propagation.

[Exxenoz]

If I publish blocks on other nodes than my own (for example MyNanoNinja using the process RPC) it takes around 1-5 minutes until my node marks them as confirmed. I even setup a second Nano node on a fresh server - same behaviour. Seems like all non-PR nodes are currently affected by this. Confirmations are almost instant if I publish the blocks directly on my node, except for open blocks, which confirm somehow as slow as if I had published them using an external node (1-5mins).

[dsiganos]

@Exxenoz , if you wish to help debug this, the next step would be to enable detailed logs and repeat your test case.
To enable detailed logs , you can add these entries to your config-node.toml file.

[node.logging]
active_update = true
bulk_pull = true
election_expiration = true
election_fork = true
ledger = true
ledger_rollback = true
ledger_duplicate = true
log_ipc = false
log_to_cerr = false
max_size = 4294967296
min_time_between_output = 0
network = true
network_keepalive = true
network_message = true
network_packet = true
network_publish = true
network_rejected = true
network_telemetry = true
network_timeout = true
node_lifetime_tracing = false
rotation_size = 268435456
single_line_record = true
stable_log_filename = true
timing = true
upnp_details = false
vote = true
flush = true

[zhyatt]

@Exxenoz We would expect you to have less impacts now, as we are now temporarily rebroadcasting blocks to help improve these situations (until a fix can be deployed broadly). This just started consistently about an hour or so ago. Let us know if you keep experiencing these same delays.

[Exxenoz]

@zhyatt

except for open blocks, which confirm somehow as slow as if I had published them using an external node (1-5mins).

^ This was my fault. I accidentally published all open blocks to MyNanoNinja yesterday.

Now:
Test1: Send & Open block published to node1. -> Send & Open block got instantly confirmed on node1.
Test2: Send & Open block published to node1. -> Send & Open block got instantly confirmed on node1.
Test3: Send & Open block published to node1. -> Send & Open block got instantly confirmed on node1.

Test4: Send & Open block published to node2 (NanoCrawler). -> Send and Open block got both confirmed with a delay of around 40-50 seconds on node1.
Test5: Send & Open block published to node2 (NanoCrawler). -> Send block got almost instantly confirmed on node1; Open block did show up after around 2 minutes, but didn't get marked as confirmed until after another 2 minutes on node1.
Test6: Send & Open block published to node2 (NanoCrawler). -> Send block did show up after around 30 seconds, got marked as confirmed after another 3 minutes. Open block did show up after 8 minutes, got marked as confirmed after another 4.5 minutes.

Test7: Send & Receive block published to node2 (NanoCrawler). -> Send block got confirmed after around 10 seconds on node1, the receive block after around 30 seconds.
Test8: Send & Receive block published to node2 (NanoCrawler). -> Send block got almost instantly confirmed on node1. Receive block showed up after around 5.5 minutes, got marked as confirmed after another 2 more minutes.

@dsiganos I added your config entries to my node-config.toml after test 8, stopped the node, deleted all existing log files and started the node again.

Test9: Send & Receive block published to node2 (NanoCrawler). -> Send block did show up after around 20 seconds on node1, got marked as confirmed after ~5 minutes. Receive block showed up after less than a minute, got marked as confirmed after ~5 minutes.

Send block hash: 728CBC805CD07A53F84976872EA778B326B094721D371B7072A86292129B9A42
Receive block hash: 768104B8B439EDAD6AD8A953D25F8422F545196EBE23E09B04294FBCF9F399B5

Test9 Nano node1 log file: https://mega.nz/file/vcJi3ZSZ#z9KbHBuECpfC5GKcWuFYyG334N6bum_jBwQcnzmnqiM

[zhyatt]

@Exxenoz Thanks for confirming. We are reviewing these details and doing further investigation into the delays.

[Exxenoz]

I just published around 1600 (800 send- & 800 receive-) blocks to my Nano node using the "process" rpc and the "async: true" option. Block confirmations are really damn slow. Maybe around 5-10 blocks / minute get confirmed currently.

The publishing process was finished at 1:55 AM UTC+2, but it looks like it will take hours for all of them to get confirmed: https://nanocrawler.cc/explorer/account/nano_15m1ggywjfbgw8uwi5ix6ma69kgqzrm9q6iccrbuua817txyima6e6x4hz9i/history

I don't know if this was already the case two weeks ago. Didn't test it with this many blocks before. But in either case this doesn't seem right and is probably related to the same issue.

[zhyatt]

I just published around 1600 (800 send- & 800 receive-) blocks to my Nano node using the "process" rpc and the "async: true" option. Block confirmations are really damn slow. Maybe around 5-10 blocks / minute get confirmed currently.

The publishing process was finished at 1:55 AM UTC+2, but it looks like it will take hours for all of them to get confirmed: https://nanocrawler.cc/explorer/account/nano_15m1ggywjfbgw8uwi5ix6ma69kgqzrm9q6iccrbuua817txyima6e6x4hz9i/history

I don't know if this was already the case two weeks ago. Didn't test it with this many blocks before. But in either case this doesn't seem right and is probably related to the same issue.

Regardless of the issues above, when publishing lots of blocks on a single account you will naturally hit a confirmation limit, depending on the network confirmation times. So if confirmations take 250ms, then you max out at 4/s. This is because nodes won't vote on a block until the previous is confirmed. If you are trying to perform tests, especially with large amounts of blocks like this, please setup a node on the beta network to do load testing. If it is a reasonable amount of activity, the test network is available (load testing should not be done there).

[Exxenoz]

I just published around 1600 (800 send- & 800 receive-) blocks to my Nano node using the "process" rpc and the "async: true" option. Block confirmations are really damn slow. Maybe around 5-10 blocks / minute get confirmed currently.
The publishing process was finished at 1:55 AM UTC+2, but it looks like it will take hours for all of them to get confirmed: https://nanocrawler.cc/explorer/account/nano_15m1ggywjfbgw8uwi5ix6ma69kgqzrm9q6iccrbuua817txyima6e6x4hz9i/history
I don't know if this was already the case two weeks ago. Didn't test it with this many blocks before. But in either case this doesn't seem right and is probably related to the same issue.

Regardless of the issues above, when publishing lots of blocks on a single account you will naturally hit a confirmation limit, depending on the network confirmation times. So if confirmations take 250ms, then you max out at 4/s. This is because nodes won't vote on a block until the previous is confirmed. If you are trying to perform tests, especially with large amounts of blocks like this, please setup a node on the beta network to do load testing. If it is a reasonable amount of activity, the test network is available (load testing should not be done there).

This makes sense, but confirmations are still too slow. :> I will look into setting up a beta node very soon. Thanks 👍

[zhyatt]

Resolved by #3507