Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Timing issue in Jormungandr for slot leader signing blocks #1143

Closed
rickymac68 opened this issue Nov 17, 2019 · 2 comments · Fixed by #1151
Closed

Timing issue in Jormungandr for slot leader signing blocks #1143

rickymac68 opened this issue Nov 17, 2019 · 2 comments · Fixed by #1151

Comments

@rickymac68
Copy link

@rickymac68 rickymac68 commented Nov 17, 2019

Epoch-16-1

Epoch-18

Describe the bug
My node timing is correct as verified by 3 different time sources and automatic time sync is turned on. When my node is supposed to sign a block I intermittently get this error in the log:
Nov 16 13:46:36.917 ERRO Eek... we missed an event schedule, system time might be off?, event_end: 2019-11-16T18:46:39+00:00, event_start: 2019-11-16T18:46:37+00:00, event_date: 12.1468, leader_id: 1, task: leadership

The problem may be more obvious when delegated stake %, but a detailed look into the problem shows it is more likely related to time. "Missed the deadline to compute the schedule" shows up in the leader logs. However a closer look at some of the time shows the system clock was 1 second ahead of what the protocol is expecting. Screen cap for Epoch 18 has red dots showing the discrepancy. The problem of blocks not being signed in time shows up between blocks that were signed correctly so that tells me system time is good in addition to my manual comparisons to 3 other time sources.

Mandatory Information

  1. jcli --full-version output;
    jcli 0.7.0 (HEAD-a93d4f67, release, macos [x86_64]) - [rustc 1.39.0 (4560ea788 2019-11-04)]
  2. jormungandr --full-version output;
    jormungandr 0.7.0 (HEAD-a93d4f67, release, macos [x86_64]) - [rustc 1.39.0 (4560ea788 2019-11-04)]

To Reproduce
Steps to reproduce the behavior:

  1. Operate Jormungandr on the nightly test net.
  2. Have a low % stake <1% to make more obvious
  3. jcli rest v0 leaders logs get -h http://127.0.0.1:3101/api
  4. See error

Expected behavior
Blocks get signed correctly.

Additional context
Using Mac OSX Catalina 10.15.1 on a Mac Pro server with auto time sync turned on.
Logs and screen captures attached. Log and screen cap for Epoch 18 when stake % is higher. Screen caps from previous epochs when stake was less than 1%
Epoch-9
Epoch-10
Epoch-10-1
Epoch-11
Epoch-13
Epoch-14

leader-logs-epoch-18-rick.log

@maneav

This comment has been minimized.

Copy link

@maneav maneav commented Nov 17, 2019

@NicolasDP NicolasDP self-assigned this Nov 18, 2019
NicolasDP added a commit that referenced this issue Nov 18, 2019
fix #1143
@rickymac68

This comment has been minimized.

Copy link
Author

@rickymac68 rickymac68 commented Nov 23, 2019

Looks like this issue is resolved. Text logs are attached. Thank you!
Epoch-49
Epoch-50
leader-log-epoch-49-Rick.txt
leader-log-epoch-50-Rick.txt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.