fix: ensure new withdrawalCredentials in switchToCompoundingValidator()#7478
Merged
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## unstable #7478 +/- ##
============================================
- Coverage 50.48% 50.47% -0.01%
============================================
Files 602 602
Lines 40606 40610 +4
Branches 2230 2230
============================================
Hits 20499 20499
- Misses 20067 20071 +4
Partials 40 40 |
Member
as pointed out on discord, we explicitly convert to lodestar/packages/api/src/utils/server/parser.ts Lines 9 to 12 in a25ea77 however, as a sanity check, we could consider asserting that when we pass the block to state-transition |
Contributor
Performance Report✔️ no performance regression detected Full benchmark results
|
nflaig
reviewed
Feb 19, 2025
nflaig
approved these changes
Feb 20, 2025
ensi321
reviewed
Feb 20, 2025
| const newWithdrawalCredentials = validator.withdrawalCredentials.slice(); | ||
| const newWithdrawalCredentials = Uint8Array.prototype.slice.call( | ||
| validator.withdrawalCredentials, | ||
| 0, |
Member
There was a problem hiding this comment.
Is the second and third param of call() necessary? Can we do
Uint8Array.prototype.slice.call(validator.withdrawalCredentials);
Member
There was a problem hiding this comment.
I don't think it is necessary, but don't mind either way
Member
Author
@nflaig this |
Member
yeah noticed that as well, was pointing that out because only the node that proposed the block has the issue, which is still unclear to me why that is the case |
Member
this happens because the proposer calls |
nflaig
added a commit
that referenced
this pull request
Feb 24, 2025
**Motivation** This is related to - #7487 - #7478 and adds a additional safety measure to avoid issues related to `Buffer`, specifically we wanna avoid unexpected behavior such as `Buffer.prototype.slice` not copying memory. **Description** This changes ensures loaded state is `typeof Uint8Array` by converting loaded bytes. This operation is pretty much free based on benchmarks in #6688 as it does not copy the underlying memory, just updates the view. ``` Buffer utils ✔ Buffer.from - copy 7.286412 ops/s 137.2418 ms/op x1.024 38 runs 5.82 s ✔ Buffer.from - no copy 1117318 ops/s 895.0000 ns/op x0.982 723013 runs 1.11 s ✔ Buffer.from - no copy with offset 1179245 ops/s 848.0000 ns/op - 2699558 runs 3.85 s ```
philknows
pushed a commit
that referenced
this pull request
Feb 24, 2025
…() (#7478) **Motivation** - got this issue in Gnosis Tested this but on first consolidation same target and source validator it apparently worked fine. However, the erigon-lodestar validator node that proposed the consolidation request transaction rejected all peers and forked. It is stuck on this slot: [link](http://45.79.201.144:8083/slot/68121) Lodestar shows these errors: ``` Feb-17 07:11:09.621[chain] debug: Block error slot=68121, code=BLOCK_ERROR_INVALID_STATE_ROOT, slot=68121, root=0xb8362da32e9ba60bb4e76aa3dae3cf3722404f97be7d674a1ba2d61140bae2f0, expectedRoot=0xb3e18d8a01cb01ac43a0e2fc4d11f6f9298d74ed51ed8520f0597001e4499516 Error: BLOCK_ERROR_INVALID_STATE_ROOT at verifyBlocksStateTransitionOnly (file:///usr/app/packages/beacon-node/src/chain/blocks/verifyBlocksStateTransitionOnly.ts:71:13) at BeaconChain.verifyBlocksInEpoch (file:///usr/app/packages/beacon-node/src/chain/blocks/verifyBlock.ts:114:7) at BeaconChain.processBlocks (file:///usr/app/packages/beacon-node/src/chain/blocks/index.ts:74:7) at JobItemQueue.runJob (file:///usr/app/packages/beacon-node/src/util/queue/itemQueue.ts:102:22) Feb-17 07:11:09.622[sync] verbose: Batch process error id=Finalized, startEpoch=4257, status=Processing, code=BLOCK_ERROR_INVALID_STATE_ROOT, slot=68121, root=0xb8362da32e9ba60bb4e76aa3dae3cf3722404f97be7d674a1ba2d61140bae2f0, expectedRoot=0xb3e18d8a01cb01ac43a0e2fc4d11f6f9298d74ed51ed8520f0597001e4499516 Error: BLOCK_ERROR_INVALID_STATE_ROOT at verifyBlocksStateTransitionOnly (file:///usr/app/packages/beacon-node/src/chain/blocks/verifyBlocksStateTransitionOnly.ts:71:13) at BeaconChain.verifyBlocksInEpoch (file:///usr/app/packages/beacon-node/src/chain/blocks/verifyBlock.ts:114:7) at BeaconChain.processBlocks (file:///usr/app/packages/beacon-node/src/chain/blocks/index.ts:74:7) at JobItemQueue.runJob (file:///usr/app/packages/beacon-node/src/util/queue/itemQueue.ts:102:22) ``` The rest of the nodes are working the fine, even nethermind-lodestar, and the validator seems to have updated to 0x02 withdrawal credentials type. [link](http://45.79.201.144:8083/validator/2000) closes #7487 **Description** - although it's not proved to fix the issue, we should ensure we always create a new `withdrawalCredentials` by enforcing the use of `Uint8Array.slice()`, just in case backed data is a Buffer - the similar "Invalid state root" was found in the past due to this code place, as noted in the comment - this pattern was used in a lot of places anyway **Update** just found this ```typescript describe("Buffer", () => { it("slice", () => { const buffer = Buffer.alloc(32, 1); const sliced = Uint8Array.prototype.slice.call(buffer, 0, buffer.length); sliced[0] = 2; expect(sliced[0]).toBe(2); expect(buffer[0]).toBe(1); expect(sliced instanceof Buffer).toBe(false); }); }); ``` ``` - Expected + Received - false + true ``` even we used this pattern a lot `Uint8Array.prototype.slice.call` in ssz to deserialize state it still return a Buffer when we deserialize state from leveldb, the state bytes there is really the Buffer that's one possibilities how this issue happened **Conclusion** The state root mismatch happens because the proposer calls `stateTransition` twice for the same block, first time when [computeNewStateRoot](https://github.com/ChainSafe/lodestar/blob/9a2cf748d3a2d63180fdfca6a948731e976b75d8/packages/beacon-node/src/chain/chain.ts#L726) is called and second time when processing / importing the block when it is publishing. Based on these observations it seems like the condition is that - lodestar needs to be proposer of block - the block needs to contain consolidation request switching validator to compounding - the validator being switched needs to be part of state that was loaded from the db / genesis ssz file Co-authored-by: Tuyen Nguyen <twoeths@users.noreply.github.com>
Closed
philknows
pushed a commit
that referenced
this pull request
Feb 24, 2025
…() (#7478) **Motivation** - got this issue in Gnosis Tested this but on first consolidation same target and source validator it apparently worked fine. However, the erigon-lodestar validator node that proposed the consolidation request transaction rejected all peers and forked. It is stuck on this slot: [link](http://45.79.201.144:8083/slot/68121) Lodestar shows these errors: ``` Feb-17 07:11:09.621[chain] debug: Block error slot=68121, code=BLOCK_ERROR_INVALID_STATE_ROOT, slot=68121, root=0xb8362da32e9ba60bb4e76aa3dae3cf3722404f97be7d674a1ba2d61140bae2f0, expectedRoot=0xb3e18d8a01cb01ac43a0e2fc4d11f6f9298d74ed51ed8520f0597001e4499516 Error: BLOCK_ERROR_INVALID_STATE_ROOT at verifyBlocksStateTransitionOnly (file:///usr/app/packages/beacon-node/src/chain/blocks/verifyBlocksStateTransitionOnly.ts:71:13) at BeaconChain.verifyBlocksInEpoch (file:///usr/app/packages/beacon-node/src/chain/blocks/verifyBlock.ts:114:7) at BeaconChain.processBlocks (file:///usr/app/packages/beacon-node/src/chain/blocks/index.ts:74:7) at JobItemQueue.runJob (file:///usr/app/packages/beacon-node/src/util/queue/itemQueue.ts:102:22) Feb-17 07:11:09.622[sync] verbose: Batch process error id=Finalized, startEpoch=4257, status=Processing, code=BLOCK_ERROR_INVALID_STATE_ROOT, slot=68121, root=0xb8362da32e9ba60bb4e76aa3dae3cf3722404f97be7d674a1ba2d61140bae2f0, expectedRoot=0xb3e18d8a01cb01ac43a0e2fc4d11f6f9298d74ed51ed8520f0597001e4499516 Error: BLOCK_ERROR_INVALID_STATE_ROOT at verifyBlocksStateTransitionOnly (file:///usr/app/packages/beacon-node/src/chain/blocks/verifyBlocksStateTransitionOnly.ts:71:13) at BeaconChain.verifyBlocksInEpoch (file:///usr/app/packages/beacon-node/src/chain/blocks/verifyBlock.ts:114:7) at BeaconChain.processBlocks (file:///usr/app/packages/beacon-node/src/chain/blocks/index.ts:74:7) at JobItemQueue.runJob (file:///usr/app/packages/beacon-node/src/util/queue/itemQueue.ts:102:22) ``` The rest of the nodes are working the fine, even nethermind-lodestar, and the validator seems to have updated to 0x02 withdrawal credentials type. [link](http://45.79.201.144:8083/validator/2000) closes #7487 **Description** - although it's not proved to fix the issue, we should ensure we always create a new `withdrawalCredentials` by enforcing the use of `Uint8Array.slice()`, just in case backed data is a Buffer - the similar "Invalid state root" was found in the past due to this code place, as noted in the comment - this pattern was used in a lot of places anyway **Update** just found this ```typescript describe("Buffer", () => { it("slice", () => { const buffer = Buffer.alloc(32, 1); const sliced = Uint8Array.prototype.slice.call(buffer, 0, buffer.length); sliced[0] = 2; expect(sliced[0]).toBe(2); expect(buffer[0]).toBe(1); expect(sliced instanceof Buffer).toBe(false); }); }); ``` ``` - Expected + Received - false + true ``` even we used this pattern a lot `Uint8Array.prototype.slice.call` in ssz to deserialize state it still return a Buffer when we deserialize state from leveldb, the state bytes there is really the Buffer that's one possibilities how this issue happened **Conclusion** The state root mismatch happens because the proposer calls `stateTransition` twice for the same block, first time when [computeNewStateRoot](https://github.com/ChainSafe/lodestar/blob/9a2cf748d3a2d63180fdfca6a948731e976b75d8/packages/beacon-node/src/chain/chain.ts#L726) is called and second time when processing / importing the block when it is publishing. Based on these observations it seems like the condition is that - lodestar needs to be proposer of block - the block needs to contain consolidation request switching validator to compounding - the validator being switched needs to be part of state that was loaded from the db / genesis ssz file Co-authored-by: Tuyen Nguyen <twoeths@users.noreply.github.com>
Closed
nflaig
pushed a commit
that referenced
this pull request
Feb 24, 2025
…() (#7478) **Motivation** - got this issue in Gnosis Tested this but on first consolidation same target and source validator it apparently worked fine. However, the erigon-lodestar validator node that proposed the consolidation request transaction rejected all peers and forked. It is stuck on this slot: [link](http://45.79.201.144:8083/slot/68121) Lodestar shows these errors: ``` Feb-17 07:11:09.621[chain] debug: Block error slot=68121, code=BLOCK_ERROR_INVALID_STATE_ROOT, slot=68121, root=0xb8362da32e9ba60bb4e76aa3dae3cf3722404f97be7d674a1ba2d61140bae2f0, expectedRoot=0xb3e18d8a01cb01ac43a0e2fc4d11f6f9298d74ed51ed8520f0597001e4499516 Error: BLOCK_ERROR_INVALID_STATE_ROOT at verifyBlocksStateTransitionOnly (file:///usr/app/packages/beacon-node/src/chain/blocks/verifyBlocksStateTransitionOnly.ts:71:13) at BeaconChain.verifyBlocksInEpoch (file:///usr/app/packages/beacon-node/src/chain/blocks/verifyBlock.ts:114:7) at BeaconChain.processBlocks (file:///usr/app/packages/beacon-node/src/chain/blocks/index.ts:74:7) at JobItemQueue.runJob (file:///usr/app/packages/beacon-node/src/util/queue/itemQueue.ts:102:22) Feb-17 07:11:09.622[sync] verbose: Batch process error id=Finalized, startEpoch=4257, status=Processing, code=BLOCK_ERROR_INVALID_STATE_ROOT, slot=68121, root=0xb8362da32e9ba60bb4e76aa3dae3cf3722404f97be7d674a1ba2d61140bae2f0, expectedRoot=0xb3e18d8a01cb01ac43a0e2fc4d11f6f9298d74ed51ed8520f0597001e4499516 Error: BLOCK_ERROR_INVALID_STATE_ROOT at verifyBlocksStateTransitionOnly (file:///usr/app/packages/beacon-node/src/chain/blocks/verifyBlocksStateTransitionOnly.ts:71:13) at BeaconChain.verifyBlocksInEpoch (file:///usr/app/packages/beacon-node/src/chain/blocks/verifyBlock.ts:114:7) at BeaconChain.processBlocks (file:///usr/app/packages/beacon-node/src/chain/blocks/index.ts:74:7) at JobItemQueue.runJob (file:///usr/app/packages/beacon-node/src/util/queue/itemQueue.ts:102:22) ``` The rest of the nodes are working the fine, even nethermind-lodestar, and the validator seems to have updated to 0x02 withdrawal credentials type. [link](http://45.79.201.144:8083/validator/2000) closes #7487 **Description** - although it's not proved to fix the issue, we should ensure we always create a new `withdrawalCredentials` by enforcing the use of `Uint8Array.slice()`, just in case backed data is a Buffer - the similar "Invalid state root" was found in the past due to this code place, as noted in the comment - this pattern was used in a lot of places anyway **Update** just found this ```typescript describe("Buffer", () => { it("slice", () => { const buffer = Buffer.alloc(32, 1); const sliced = Uint8Array.prototype.slice.call(buffer, 0, buffer.length); sliced[0] = 2; expect(sliced[0]).toBe(2); expect(buffer[0]).toBe(1); expect(sliced instanceof Buffer).toBe(false); }); }); ``` ``` - Expected + Received - false + true ``` even we used this pattern a lot `Uint8Array.prototype.slice.call` in ssz to deserialize state it still return a Buffer when we deserialize state from leveldb, the state bytes there is really the Buffer that's one possibilities how this issue happened **Conclusion** The state root mismatch happens because the proposer calls `stateTransition` twice for the same block, first time when [computeNewStateRoot](https://github.com/ChainSafe/lodestar/blob/9a2cf748d3a2d63180fdfca6a948731e976b75d8/packages/beacon-node/src/chain/chain.ts#L726) is called and second time when processing / importing the block when it is publishing. Based on these observations it seems like the condition is that - lodestar needs to be proposer of block - the block needs to contain consolidation request switching validator to compounding - the validator being switched needs to be part of state that was loaded from the db / genesis ssz file Co-authored-by: Tuyen Nguyen <twoeths@users.noreply.github.com>
Merged
Member
|
🎉 This PR is included in v1.27.1 🎉 |
Member
|
🎉 This PR is included in v1.28.0 🎉 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Tested this but on first consolidation same target and source validator it apparently worked fine. However, the erigon-lodestar validator node that proposed the consolidation request transaction rejected all peers and forked. It is stuck on this slot: link
Lodestar shows these errors:
The rest of the nodes are working the fine, even nethermind-lodestar, and the validator seems to have updated to 0x02 withdrawal credentials type. link
closes #7487
Description
withdrawalCredentialsby enforcing the use ofUint8Array.slice(), just in case backed data is a BufferUpdate
just found this
even we used this pattern a lot
Uint8Array.prototype.slice.callin ssz to deserialize state it still return a Bufferwhen we deserialize state from leveldb, the state bytes there is really the Buffer
that's one possibilities how this issue happened
Conclusion
The state root mismatch happens because the proposer calls
stateTransitiontwice for the same block, first time when computeNewStateRoot is called and second time when processing / importing the block when it is publishing.Based on these observations it seems like the condition is that