Byron-Proxy failing to sync from clean slate

[oneEdoubleD]

Issue

Cardano-Byron-Proxy is failing to sync when it's started from a clean slate. This is happening for both tags/1.0.3 and master on both staging and mainnet.

The first error that occurs once is:

[cardano_byron_proxy.main.client:Info:62] [2020-01-20 10:51:17.77 UTC] Attempting to download chain with hash AbstractHash 0bdaaa4fe38239c8fa64dfb208ff80886ef8f5dd49d9b532948809682487c85a from NodeId 13.250.233.196:3000:0
[cardano_byron_proxy.diffusion.outboundqueue.self:Debug:62] [2020-01-20 10:51:17.77 UTC] inFlight = fromList []
[cardano_byron_proxy.diffusion.outboundqueue.self:Debug:62] [2020-01-20 10:51:17.77 UTC] message MsgRequestBlocks (fromList [NodeId 13.250.233.196:3000:0]) enqueued to [NodeId 13.250.233.196:3000:0]
[cardano_byron_proxy.diffusion.outboundqueue.self:Debug:75] [2020-01-20 10:51:17.78 UTC] sending MsgRequestBlocks (fromList [NodeId 13.250.233.196:3000:0]) to NodeId 13.250.233.196:3000:0
[cardano_byron_proxy.diffusion:Debug:78] [2020-01-20 10:51:18.59 UTC] streamBlocks: Requesting stream of blocks from [d241cc87ef9d4ac5] to 0bdaaa4f

[cardano_byron_proxy.diffusion:Error:80] [2020-01-20 10:51:19.17 UTC] MsgStreamNoBlock handleStreamStart:strean Failed to find lca

[cardano_byron_proxy.diffusion.outboundqueue.self:Warning:75] [2020-01-20 10:51:19.17 UTC] sending MsgRequestBlocks (fromList [NodeId 13.250.233.196:3000:0]) to NodeId 13.250.233.196:3000:0 failed with BlockNetLogicException: DialogUnexpected "MsgStreamNoBlock handleStreamStart:strean Failed to find lca" :: SomeException

and then the proxy loops through these errors:

[cardano_byron_proxy.diffusion.outboundqueue.self:Error:60] [2020-01-20 10:39:42.72 UTC] enqueue instruction EnqueueOne [NodeRelay,NodeCore] (MaxAhead 3) PHigh failed to enqueue message MsgRequestBlocks (fromList [NodeId 13.250.233.196:3000:0]) to forwarding sets [(NodeRelay,[NodeId 13.250.233.196:3000:0])]
[cardano_byron_proxy.main.client:Info:60] [2020-01-20 10:39:42.72 UTC] Attempting to download chain with hash AbstractHash 2be956e62cb1a52334a82ade9de04841ddac2082606ed1fa740d8bb039548381 from NodeId 13.250.233.196:3000:0
[cardano_byron_proxy.diffusion.outboundqueue.self:Debug:60] [2020-01-20 10:39:42.72 UTC] inFlight = fromList []
[cardano_byron_proxy.diffusion.outboundqueue.self:Debug:60] [2020-01-20 10:39:42.72 UTC] Rejected alternative NodeId 13.250.233.196:3000:0 as it has a recent failure
[cardano_byron_proxy.diffusion.outboundqueue.self:Error:60] [2020-01-20 10:39:42.72 UTC] enqueue instruction EnqueueOne [NodeRelay,NodeCore] (MaxAhead 3) PHigh failed to enqueue message MsgRequestBlocks (fromList [NodeId 13.250.233.196:3000:0]) to forwarding sets [(NodeRelay,[NodeId 13.250.233.196:3000:0])]

@jbgi noticed a difference in the way logs are written which suggests some behaviour has changed:

1.0.3:

Rollback Origin
Rollforward (Block {blockPointSlot = SlotNo {unSlotNo = 0}, blockPointHash = ByronHash {unByronHash = AbstractHash d241cc87ef9d4ac5d4359d22d2db1c83102affd1220dda7e3c1895f47b33c8e4}})
startNode, we are NodeId de1dcb38-6ba7-4b18-969d-e82f537f7c73
dnsSubscriptionWorker: valency 1
subscriptionWorker: subscribing to NodeId 52.68.30.216:3000:0

< 1.0.3

Rollback Origin
startNode, we are NodeId 74ab47b9-a48a-4f0e-95d4-a07c169b6f33
dnsSubscriptionWorker: valency 1
Seeding database with genesis
subscriptionWorker: subscribing to NodeId 52.68.30.216:3000:0
Rollforward (Block {blockPointSlot = SlotNo {unSlotNo = 0}, blockPointHash = ByronHash {unByronHash = AbstractHash b365f1be6863b453f12b93e1810909b10c79a95ee44bf53414888513fe172c90}})

[karknu]

This ticket should be against https://github.com/input-output-hk/cardano-byron-proxy/ .

[karknu]

The bug is related to the new version of ouroboros-network included with version 1.0.3

[avieth]

Very strange that the apparent genesis hash changes with the new release. At release 1.0.2 (commit 9bae138919c31b5aeac8f7244e2b28e5f31d9f4f) I see 89d9b5a5b8ddc8d7e5a6795e9774d97faf1efea59b2caf7eaf9f8c5b32059df4, but with release 1.0.3 I see 6c6e21d568c6a45b4aa9d5342e950c8f85ed3d28bb45270247d76c79f58df05d.

cardano-byron-proxy does not compute these hashes itself; they come from the ChainDB, a part of ouroboros-network. Those Rollforward (Block {blockPointSlot ... log entries are of the Point values given by a ChainDB reader that the index uses to keep itself in sync.

[mrBliss]

Karl found out that the problem started as of IntersectMBO/ouroboros-network#1450. In that PR, I automatically create an EBB when the ChainDB is empty, because the cardano-ledger forces us to start the chain with an EBB. If we don't create this EBB, then no newly forged block will ever fit onto our chain. The benchmarking team were running into this problem when benchmarking

When starting with an existing chain, this change won't affect you.

When the node starts with an empty chain and thus creates the EBB automatically, we'll simply switch to the longer chain once we start downloading blocks.

A proxy started with an empty chain, however, tries to roll forward from the tip of the ChainDB, which is now that automatically created EBB instead of genesis. That's what's going wrong.

[erikd]

@mrBliss is someone working on a fix for that? I use the proxy while testing my explorer and I recently blew away the proxy state to test something and now I am stuck.

[jbgi]

@erikd as a workaround you can start an older version (1.0.2) for a few seconds and then switch back to master branch.

[mrBliss]

@erikd Yes, I'm working on it.

[mrBliss]

I'm trying to build the proxy with the latest ouroboros-network, but the network team bumped the dependency on the network package to network-3.1.0.1, causing conflicts in the build plan. Can somebody from the network team help with this? cc: @coot @karknu
In the meantime, I'll use an older version of ouroboros-network.