Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AssertionError in get_crosslink_committees_at_slot after slot 64 #101

Closed
mratsim opened this issue Feb 11, 2019 · 2 comments
Closed

AssertionError in get_crosslink_committees_at_slot after slot 64 #101

mratsim opened this issue Feb 11, 2019 · 2 comments

Comments

@mratsim
Copy link
Contributor

mratsim commented Feb 11, 2019 •
edited

On Linux x86-64

Similar to #89 #97 we have an issue during epoch transition, this times after 64 slots, for epoch 144115188075855873

Full logs available in https://gist.github.com/mratsim/a6539c049dfb49a638327b9db5cec44a.
There are 2 logs, one with multiple nodes desynced, one with only a single node desynced.

The following log is after merging #103 for epochs printing.

beacon_node.nim(435)     beacon_node
beacon_node.nim(399)     processBlocks
asyncloop.nim(756)       runForever
asyncloop.nim(238)       poll
beacon_node.nim(328)     :anonymous
beacon_node.nim(306)     scheduleEpochActions
validator.nim(95)        get_crosslink_committees_at_slot
system.nim(3879)         failedAssertImpl
system.nim(3872)         raiseAssert
system.nim(2918)         sysFatal
Error: unhandled exception: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(95, 10) `epoch <= next_epoch` Previous epoch: 144115188075855872, epoch: 144115188075855874, Next epoch: 144115188075855873 [AssertionError]
INF 2019-02-11 16:09:08+01:00 Scheduling block proposal                  thread=0 fromNow=338 slot=121 validator=96d7e599 node=0
INF 2019-02-11 16:09:08+01:00 Scheduling block proposal                  thread=0 fromNow=296 slot=114 validator=8e724e86 node=5
beacon_node.nim(435)     beacon_node
beacon_node.nim(399)     processBlocks
asyncloop.nim(756)       runForever
asyncloop.nim(238)       poll
beacon_node.nim(328)     :anonymous
beacon_node.nim(306)     scheduleEpochActions
validator.nim(95)        get_crosslink_committees_at_slot
system.nim(3879)         failedAssertImpl
system.nim(3872)         raiseAssert
system.nim(2918)         sysFatal
Error: unhandled exception: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(95, 10) `epoch <= next_epoch` Previous epoch: 144115188075855872, epoch: 144115188075855874, Next epoch: 144115188075855873 [AssertionError]
INF 2019-02-11 16:09:08+01:00 Scheduling block proposal                  thread=0 fromNow=326 slot=119 validator=0a89ffbe node=2
beacon_node.nim(435)     beacon_node
beacon_node.nim(399)     processBlocks
asyncloop.nim(756)       runForever
asyncloop.nim(238)       poll
beacon_node.nim(328)     :anonymous
beacon_node.nim(306)     scheduleEpochActions
validator.nim(95)        get_crosslink_committees_at_slot
system.nim(3879)         failedAssertImpl
system.nim(3872)         raiseAssert
system.nim(2918)         sysFatal
Error: unhandled exception: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(95, 10) `epoch <= next_epoch` Previous epoch: 144115188075855872, epoch: 144115188075855874, Next epoch: 144115188075855873 [AssertionError]
beacon_node.nim(435)     beacon_node
beacon_node.nim(399)     processBlocks
asyncloop.nim(756)       runForever
asyncloop.nim(238)       poll
beacon_node.nim(328)     :anonymous
beacon_node.nim(306)     scheduleEpochActions
validator.nim(95)        get_crosslink_committees_at_slot
system.nim(3879)         failedAssertImpl
system.nim(3872)         raiseAssert
system.nim(2918)         sysFatal
Error: unhandled exception: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(95, 10) `epoch <= next_epoch` Previous epoch: 144115188075855872, epoch: 144115188075855874, Next epoch: 144115188075855873 [AssertionError]
INF 2019-02-11 16:09:08+01:00 Scheduling block proposal                  thread=0 fromNow=350 slot=123 validator=91792a76 node=7
beacon_node.nim(435)     beacon_node
beacon_node.nim(399)     processBlocks
asyncloop.nim(756)       runForever
asyncloop.nim(238)       poll
beacon_node.nim(328)     :anonymous
beacon_node.nim(306)     scheduleEpochActions
validator.nim(95)        get_crosslink_committees_at_slot
system.nim(3879)         failedAssertImpl
system.nim(3872)         raiseAssert
system.nim(2918)         sysFatal
Error: unhandled exception: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(95, 10) `epoch <= next_epoch` Previous epoch: 144115188075855872, epoch: 144115188075855874, Next epoch: 144115188075855873 [AssertionError]
beacon_node.nim(435)     beacon_node
beacon_node.nim(399)     processBlocks
asyncloop.nim(756)       runForever
asyncloop.nim(238)       poll
beacon_node.nim(328)     :anonymous
beacon_node.nim(306)     scheduleEpochActions
validator.nim(95)        get_crosslink_committees_at_slot
system.nim(3879)         failedAssertImpl
system.nim(3872)         raiseAssert
system.nim(2918)         sysFatal
Error: unhandled exception: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(95, 10) `epoch <= next_epoch` Previous epoch: 144115188075855872, epoch: 144115188075855874, Next epoch: 144115188075855873 [Assert
beacon_node.nim(435)     beacon_node
beacon_node.nim(399)     processBlocks
asyncloop.nim(756)       runForever
asyncloop.nim(238)       poll
beacon_node.nim(328)     :anonymous
beacon_node.nim(306)     scheduleEpochActions
validator.nim(95)        get_crosslink_committees_at_slot
system.nim(3879)         failedAssertImpl
system.nim(3872)         raiseAssert
system.nim(2918)         sysFatal
Error: unhandled exception: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(95, 10) `epoch <= next_epoch` Previous epoch: 144115188075855872, epoch: 144115188075855874, Next epoch: 144115188075855873 [Assert
INF 2019-02-11 16:09:08+01:00 Scheduling block proposal                  thread=0 fromNow=344 slot=122 validator=842ef0e2 node=8
INF 2019-02-11 16:09:08+01:00 Scheduling block proposal                  thread=0 fromNow=356 slot=124 validator=861b6b0b node=8
INF 2019-02-11 16:09:08+01:00 Scheduling block proposal                  thread=0 fromNow=362 slot=125 validator=928a8a8e node=8
INF 2019-02-11 16:09:08+01:00 Scheduling block proposal                  thread=0 fromNow=374 slot=127 validator=9090c4be node=1
beacon_node.nim(435)     beacon_node
beacon_node.nim(399)     processBlocks
asyncloop.nim(756)       runForever
asyncloop.nim(238)       poll
beacon_node.nim(328)     :anonymous
beacon_node.nim(306)     scheduleEpochActions
validator.nim(95)        get_crosslink_committees_at_slot
system.nim(3879)         failedAssertImpl
system.nim(3872)         raiseAssert
system.nim(2918)         sysFatal
Error: unhandled exception: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(95, 10) `epoch <= next_epoch` Previous epoch: 144115188075855872, epoch: 144115188075855874, Next epoch: 144115188075855873 [Assert
beacon_node.nim(435)     beacon_node
beacon_node.nim(399)     processBlocks
asyncloop.nim(756)       runForever
asyncloop.nim(238)       poll
beacon_node.nim(328)     :anonymous
beacon_node.nim(306)     scheduleEpochActions
validator.nim(95)        get_crosslink_committees_at_slot
system.nim(3879)         failedAssertImpl
system.nim(3872)         raiseAssert
system.nim(2918)         sysFatal
Error: unhandled exception: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(95, 10) `epoch <= next_epoch` Previous epoch: 144115188075855872, epoch: 144115188075855874, Next epoch: 144115188075855873 [Assert
INF 2019-02-11 16:09:08+01:00 Scheduling block proposal                  thread=0 fromNow=380 slot=128 validator=968a8996 node=8
beacon_node.nim(435)     beacon_node
beacon_node.nim(399)     processBlocks
asyncloop.nim(756)       runForever
asyncloop.nim(238)       poll
beacon_node.nim(328)     :anonymous
beacon_node.nim(306)     scheduleEpochActions
validator.nim(95)        get_crosslink_committees_at_slot
system.nim(3879)         failedAssertImpl
system.nim(3872)         raiseAssert
system.nim(2918)         sysFatal
Error: unhandled exception: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(95, 10) `epoch <= next_epoch` Previous epoch: 144115188075855872, epoch: 144115188075855874, Next epoch: 144115188075855873 [Assert
Terminated
@mratsim mratsim mentioned this issue Feb 18, 2019
Merged
@mratsim
Copy link
Contributor Author

mratsim commented Feb 19, 2019

It seems like we are overwriting slots between epoch updates.

With several added debug traces in scheduleEpochActions

deepinscreenshot_select-area_20190219150512

This was referenced Feb 19, 2019
Closed
Closed
@arnetheduck
Copy link
Member

state replay again should fix these, haven't seen anything like it in a few days

etan-status pushed a commit that referenced this issue May 12, 2022
@cheatfate
Fix cancellation race when connect() call is getting cancelled. (#101)
d6d0084
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix yet another epoch slot issue status-im/nimbus-eth2
2 participants
@arnetheduck @mratsim