Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Delayed message propagation on network setup #6396

Closed
Tracked by #6185
travisperson opened this issue Jun 4, 2021 · 8 comments · Fixed by #6453
Closed
Tracked by #6185

[BUG] Delayed message propagation on network setup #6396

travisperson opened this issue Jun 4, 2021 · 8 comments · Fixed by #6453
Assignees
@travisperson @magik6k @arajasek
Labels
kind/bug Kind: Bug P1 P1: Must be resolved
Projects
Lotus+Actors Board
Milestone
Network Hyperdrive

Comments

@travisperson
Copy link
Contributor

Ran into this issue when deploying a network based on feat/nv13-1.11 (40cc29d)

During a network setup I observed a prolonged initialization of the non-genesis miners (t01001, t01002) after the genesis miner (t01000) was up and running. After some inspection I found that the initialization message for t01001 and t01002 were not showing up in t01000 mempool, or each others mempools. The messages were only being propagated to the bootstrap peers on the network and no other peers.

All nodes were fully connected, all peer scores were non-negative, and all nodes were fully synced with miner t01000.

➜  ansible git:(master) ✗ ansible -i $hostfile -b -m shell -a 'lotus mpool pending' lotus_daemon
bootstrap-0.interop.fildev.network | CHANGED | rc=0 >>
{
  "Message": {
    "Version": 0,
    "To": "t01002",
    "From": "t3xctj5ltrh2endxj3aj4xd3cxawbjsysfpcg7pfy5irfd767loqbamtu5ybfiopmxqso7frqdj56tzxzpgv4q",
    "Nonce": 0,
    "Value": "0",
    "GasLimit": 1086678,
    "GasFeeCap": "807321212",
    "GasPremium": "100107",
    "Method": 4,
    "Params": "gVgmACQIARIgFfYPuSrRq5XmXSY6BHqX/vWhF5R3IP6ko+T6t4oaHVE=",
    "CID": {
      "/": "bafy2bzacecybsjpgmhajryalpjw6cobr5dogbfgvway455a7gfnhtk6wljf3u"
    }
  },
  "Signature": {
    "Type": 2,
    "Data": "tHT2pz0wvHGdTGEpPXlr6ElmCvL+ySsvR5EAGMM2IODqww7mLP1eAHU8TWvJhOleA2b/4obWdgHZs5C06nxWeRagry/R9fwkCjBstS9Xprx5CXuZFWR/xij9RbTrm+/a"
  },
  "CID": {
    "/": "bafy2bzacecybsjpgmhajryalpjw6cobr5dogbfgvway455a7gfnhtk6wljf3u"
  }
}
{
  "Message": {
    "Version": 0,
    "To": "t01001",
    "From": "t3uvqe3el3itf3q7cx6nxq5crvurohe76hvszldtjqltvdjdsguaomklxipasdzpip7gj43x5w444tchmigyba",
    "Nonce": 0,
    "Value": "0",
    "GasLimit": 1086678,
    "GasFeeCap": "131087",
    "GasPremium": "124541",
    "Method": 4,
    "Params": "gVgmACQIARIgHsTJ+86FX6jHQET9Q62ekxVnoxsZ3lWdanNOzUopR4g=",
    "CID": {
      "/": "bafy2bzacea4uo56wdksh4ek7aqd4lujierrrghm24mbqq3lq6gyus6usksf36"
    }
  },
  "Signature": {
    "Type": 2,
    "Data": "pb9hI2wHpBljzctcCMv9TCfukBbk4fOtNE638ovVvcvCm8ie/Lpeit2JWNetZDmGGEZcoFX2InJ5WvY3VmQwfIOS73IJAxeR6khCBwpvppblfEo/lJ253vgKY80R1SWP"
  },
  "CID": {
    "/": "bafy2bzacea4uo56wdksh4ek7aqd4lujierrrghm24mbqq3lq6gyus6usksf36"
  }
}
scratch-1.interop.fildev.network | CHANGED | rc=0 >>
bootstrap-1.interop.fildev.network | CHANGED | rc=0 >>
{
  "Message": {
    "Version": 0,
    "To": "t01002",
    "From": "t3xctj5ltrh2endxj3aj4xd3cxawbjsysfpcg7pfy5irfd767loqbamtu5ybfiopmxqso7frqdj56tzxzpgv4q",
    "Nonce": 0,
    "Value": "0",
    "GasLimit": 1086678,
    "GasFeeCap": "807321212",
    "GasPremium": "100107",
    "Method": 4,
    "Params": "gVgmACQIARIgFfYPuSrRq5XmXSY6BHqX/vWhF5R3IP6ko+T6t4oaHVE=",
    "CID": {
      "/": "bafy2bzacecybsjpgmhajryalpjw6cobr5dogbfgvway455a7gfnhtk6wljf3u"
    }
  },
  "Signature": {
    "Type": 2,
    "Data": "tHT2pz0wvHGdTGEpPXlr6ElmCvL+ySsvR5EAGMM2IODqww7mLP1eAHU8TWvJhOleA2b/4obWdgHZs5C06nxWeRagry/R9fwkCjBstS9Xprx5CXuZFWR/xij9RbTrm+/a"
  },
  "CID": {
    "/": "bafy2bzacecybsjpgmhajryalpjw6cobr5dogbfgvway455a7gfnhtk6wljf3u"
  }
}
{
  "Message": {
    "Version": 0,
    "To": "t01001",
    "From": "t3uvqe3el3itf3q7cx6nxq5crvurohe76hvszldtjqltvdjdsguaomklxipasdzpip7gj43x5w444tchmigyba",
    "Nonce": 0,
    "Value": "0",
    "GasLimit": 1086678,
    "GasFeeCap": "131087",
    "GasPremium": "124541",
    "Method": 4,
    "Params": "gVgmACQIARIgHsTJ+86FX6jHQET9Q62ekxVnoxsZ3lWdanNOzUopR4g=",
    "CID": {
      "/": "bafy2bzacea4uo56wdksh4ek7aqd4lujierrrghm24mbqq3lq6gyus6usksf36"
    }
  },
  "Signature": {
    "Type": 2,
    "Data": "pb9hI2wHpBljzctcCMv9TCfukBbk4fOtNE638ovVvcvCm8ie/Lpeit2JWNetZDmGGEZcoFX2InJ5WvY3VmQwfIOS73IJAxeR6khCBwpvppblfEo/lJ253vgKY80R1SWP"
  },
  "CID": {
    "/": "bafy2bzacea4uo56wdksh4ek7aqd4lujierrrghm24mbqq3lq6gyus6usksf36"
  }
}
scratch-0.interop.fildev.network | CHANGED | rc=0 >>
toolshed-0.interop.fildev.network | CHANGED | rc=0 >>
preminer-1.interop.fildev.network | CHANGED | rc=0 >>
{
  "Message": {
    "Version": 0,
    "To": "t01001",
    "From": "t3uvqe3el3itf3q7cx6nxq5crvurohe76hvszldtjqltvdjdsguaomklxipasdzpip7gj43x5w444tchmigyba",
    "Nonce": 0,
    "Value": "0",
    "GasLimit": 1086678,
    "GasFeeCap": "131087",
    "GasPremium": "124541",
    "Method": 4,
    "Params": "gVgmACQIARIgHsTJ+86FX6jHQET9Q62ekxVnoxsZ3lWdanNOzUopR4g=",
    "CID": {
      "/": "bafy2bzacea4uo56wdksh4ek7aqd4lujierrrghm24mbqq3lq6gyus6usksf36"
    }
  },
  "Signature": {
    "Type": 2,
    "Data": "pb9hI2wHpBljzctcCMv9TCfukBbk4fOtNE638ovVvcvCm8ie/Lpeit2JWNetZDmGGEZcoFX2InJ5WvY3VmQwfIOS73IJAxeR6khCBwpvppblfEo/lJ253vgKY80R1SWP"
  },
  "CID": {
    "/": "bafy2bzacea4uo56wdksh4ek7aqd4lujierrrghm24mbqq3lq6gyus6usksf36"
  }
}
preminer-2.interop.fildev.network | CHANGED | rc=0 >>
{
  "Message": {
    "Version": 0,
    "To": "t01002",
    "From": "t3xctj5ltrh2endxj3aj4xd3cxawbjsysfpcg7pfy5irfd767loqbamtu5ybfiopmxqso7frqdj56tzxzpgv4q",
    "Nonce": 0,
    "Value": "0",
    "GasLimit": 1086678,
    "GasFeeCap": "807321212",
    "GasPremium": "100107",
    "Method": 4,
    "Params": "gVgmACQIARIgFfYPuSrRq5XmXSY6BHqX/vWhF5R3IP6ko+T6t4oaHVE=",
    "CID": {
      "/": "bafy2bzacecybsjpgmhajryalpjw6cobr5dogbfgvway455a7gfnhtk6wljf3u"
    }
  },
  "Signature": {
    "Type": 2,
    "Data": "tHT2pz0wvHGdTGEpPXlr6ElmCvL+ySsvR5EAGMM2IODqww7mLP1eAHU8TWvJhOleA2b/4obWdgHZs5C06nxWeRagry/R9fwkCjBstS9Xprx5CXuZFWR/xij9RbTrm+/a"
  },
  "CID": {
    "/": "bafy2bzacecybsjpgmhajryalpjw6cobr5dogbfgvway455a7gfnhtk6wljf3u"
  }
}


preminer-0.interop.fildev.network | CHANGED | rc=0 >>
➜  ansible git:(master) ✗ ansible -i $hostfile -b -m shell -a 'lotus net scores' lotus_daemon
bootstrap-1.interop.fildev.network | CHANGED | rc=0 >>
12D3KooWQJyXf1RWs9xafT6JhdydZDNYqTPwJcw7mH1V2PoLiznd, 2500.000190
12D3KooWQBA2s4KX6kBFMHnMvXZDbkReMZSkjFzJpGDHxK3Tw66Z, 0.000190
12D3KooWMefPq6oyBbavPwVJsc3VFpYuGBReAJ95k5z2se7tZy5N, 0.000190
12D3KooWEyusHDDmqqZ6bk4zE73HfDnZCzE1oBs6UGfy7ox9ZTsw, 0.000190
12D3KooWEJjNJmhACb5ak8LztgoAYP9bnxmX9f1owCxgeYpCSBqa, 0.004965
12D3KooWC4mWJhM5psdpWkZSKYED3UYB8QvMRd1FCJ2FGmq64WYL, 0.000190
12D3KooWADsRekqnpHHaKg7rNhVQHKMTRrFAktoxUfAxWAqq8d7V, 28.715898
bootstrap-0.interop.fildev.network | CHANGED | rc=0 >>
12D3KooWQBA2s4KX6kBFMHnMvXZDbkReMZSkjFzJpGDHxK3Tw66Z, 0.000190
12D3KooWMefPq6oyBbavPwVJsc3VFpYuGBReAJ95k5z2se7tZy5N, 0.000190
12D3KooWFbvG7af6eJZ9VDogFYpLDaQzmKmPNWiFMBtpAA5rb8cU, 2500.000190
12D3KooWEyusHDDmqqZ6bk4zE73HfDnZCzE1oBs6UGfy7ox9ZTsw, 0.000190
12D3KooWEJjNJmhACb5ak8LztgoAYP9bnxmX9f1owCxgeYpCSBqa, 0.004965
12D3KooWC4mWJhM5psdpWkZSKYED3UYB8QvMRd1FCJ2FGmq64WYL, 0.000190
12D3KooWADsRekqnpHHaKg7rNhVQHKMTRrFAktoxUfAxWAqq8d7V, 28.734709
scratch-0.interop.fildev.network | CHANGED | rc=0 >>
12D3KooWQJyXf1RWs9xafT6JhdydZDNYqTPwJcw7mH1V2PoLiznd, 2500.000190
12D3KooWMefPq6oyBbavPwVJsc3VFpYuGBReAJ95k5z2se7tZy5N, 0.000190
12D3KooWFbvG7af6eJZ9VDogFYpLDaQzmKmPNWiFMBtpAA5rb8cU, 2500.141470
12D3KooWEyusHDDmqqZ6bk4zE73HfDnZCzE1oBs6UGfy7ox9ZTsw, 0.000190
12D3KooWEJjNJmhACb5ak8LztgoAYP9bnxmX9f1owCxgeYpCSBqa, 0.025442
12D3KooWC4mWJhM5psdpWkZSKYED3UYB8QvMRd1FCJ2FGmq64WYL, 0.025442
12D3KooWADsRekqnpHHaKg7rNhVQHKMTRrFAktoxUfAxWAqq8d7V, 28.591451
scratch-1.interop.fildev.network | CHANGED | rc=0 >>
12D3KooWQJyXf1RWs9xafT6JhdydZDNYqTPwJcw7mH1V2PoLiznd, 2500.000190
12D3KooWQBA2s4KX6kBFMHnMvXZDbkReMZSkjFzJpGDHxK3Tw66Z, 0.000190
12D3KooWFbvG7af6eJZ9VDogFYpLDaQzmKmPNWiFMBtpAA5rb8cU, 2500.141109
12D3KooWEyusHDDmqqZ6bk4zE73HfDnZCzE1oBs6UGfy7ox9ZTsw, 0.000190
12D3KooWEJjNJmhACb5ak8LztgoAYP9bnxmX9f1owCxgeYpCSBqa, 0.025249
12D3KooWC4mWJhM5psdpWkZSKYED3UYB8QvMRd1FCJ2FGmq64WYL, 0.025249
12D3KooWADsRekqnpHHaKg7rNhVQHKMTRrFAktoxUfAxWAqq8d7V, 28.550551
toolshed-0.interop.fildev.network | CHANGED | rc=0 >>
12D3KooWQJyXf1RWs9xafT6JhdydZDNYqTPwJcw7mH1V2PoLiznd, 2500.000190
12D3KooWQBA2s4KX6kBFMHnMvXZDbkReMZSkjFzJpGDHxK3Tw66Z, 0.000190
12D3KooWMefPq6oyBbavPwVJsc3VFpYuGBReAJ95k5z2se7tZy5N, 0.000190
12D3KooWFbvG7af6eJZ9VDogFYpLDaQzmKmPNWiFMBtpAA5rb8cU, 2500.141470
12D3KooWEJjNJmhACb5ak8LztgoAYP9bnxmX9f1owCxgeYpCSBqa, 0.025637
12D3KooWC4mWJhM5psdpWkZSKYED3UYB8QvMRd1FCJ2FGmq64WYL, 0.025637
12D3KooWADsRekqnpHHaKg7rNhVQHKMTRrFAktoxUfAxWAqq8d7V, 28.601379
preminer-2.interop.fildev.network | CHANGED | rc=0 >>
12D3KooWQJyXf1RWs9xafT6JhdydZDNYqTPwJcw7mH1V2PoLiznd, 2500.142558
12D3KooWQBA2s4KX6kBFMHnMvXZDbkReMZSkjFzJpGDHxK3Tw66Z, 0.000190
12D3KooWMefPq6oyBbavPwVJsc3VFpYuGBReAJ95k5z2se7tZy5N, 0.000190
12D3KooWFbvG7af6eJZ9VDogFYpLDaQzmKmPNWiFMBtpAA5rb8cU, 2500.000190
12D3KooWEyusHDDmqqZ6bk4zE73HfDnZCzE1oBs6UGfy7ox9ZTsw, 0.000190
12D3KooWEJjNJmhACb5ak8LztgoAYP9bnxmX9f1owCxgeYpCSBqa, 0.026632
12D3KooWADsRekqnpHHaKg7rNhVQHKMTRrFAktoxUfAxWAqq8d7V, 28.312855
preminer-1.interop.fildev.network | CHANGED | rc=0 >>
12D3KooWQJyXf1RWs9xafT6JhdydZDNYqTPwJcw7mH1V2PoLiznd, 2500.000190
12D3KooWQBA2s4KX6kBFMHnMvXZDbkReMZSkjFzJpGDHxK3Tw66Z, 0.000190
12D3KooWMefPq6oyBbavPwVJsc3VFpYuGBReAJ95k5z2se7tZy5N, 0.000190
12D3KooWFbvG7af6eJZ9VDogFYpLDaQzmKmPNWiFMBtpAA5rb8cU, 2500.142558
12D3KooWEyusHDDmqqZ6bk4zE73HfDnZCzE1oBs6UGfy7ox9ZTsw, 0.000190
12D3KooWC4mWJhM5psdpWkZSKYED3UYB8QvMRd1FCJ2FGmq64WYL, 0.026836
12D3KooWADsRekqnpHHaKg7rNhVQHKMTRrFAktoxUfAxWAqq8d7V, 28.318649
preminer-0.interop.fildev.network | CHANGED | rc=0 >>
12D3KooWQJyXf1RWs9xafT6JhdydZDNYqTPwJcw7mH1V2PoLiznd, 2500.000190
12D3KooWQBA2s4KX6kBFMHnMvXZDbkReMZSkjFzJpGDHxK3Tw66Z, 0.000190
12D3KooWMefPq6oyBbavPwVJsc3VFpYuGBReAJ95k5z2se7tZy5N, 0.000190
12D3KooWFbvG7af6eJZ9VDogFYpLDaQzmKmPNWiFMBtpAA5rb8cU, 2500.000190
12D3KooWEyusHDDmqqZ6bk4zE73HfDnZCzE1oBs6UGfy7ox9ZTsw, 0.000190
12D3KooWEJjNJmhACb5ak8LztgoAYP9bnxmX9f1owCxgeYpCSBqa, 0.024679
12D3KooWC4mWJhM5psdpWkZSKYED3UYB8QvMRd1FCJ2FGmq64WYL, 0.024679


➜  ansible git:(master) ✗ ansible -i $hostfile -b -m shell -a 'lotus chain list --count 1' lotus_daemon
scratch-0.interop.fildev.network | CHANGED | rc=0 >>
129: (Jun  3 18:59:30) [ bafy2bzacea6eft7vcm7cpm3pni5u6slgx73u4po3dfhoeuhze5o3px7no7gue: t01000, ]
bootstrap-0.interop.fildev.network | CHANGED | rc=0 >>
129: (Jun  3 18:59:30) [ bafy2bzacea6eft7vcm7cpm3pni5u6slgx73u4po3dfhoeuhze5o3px7no7gue: t01000, ]
scratch-1.interop.fildev.network | CHANGED | rc=0 >>
129: (Jun  3 18:59:30) [ bafy2bzacea6eft7vcm7cpm3pni5u6slgx73u4po3dfhoeuhze5o3px7no7gue: t01000, ]
bootstrap-1.interop.fildev.network | CHANGED | rc=0 >>
129: (Jun  3 18:59:30) [ bafy2bzacea6eft7vcm7cpm3pni5u6slgx73u4po3dfhoeuhze5o3px7no7gue: t01000, ]
toolshed-0.interop.fildev.network | CHANGED | rc=0 >>
129: (Jun  3 18:59:30) [ bafy2bzacea6eft7vcm7cpm3pni5u6slgx73u4po3dfhoeuhze5o3px7no7gue: t01000, ]
preminer-0.interop.fildev.network | CHANGED | rc=0 >>
129: (Jun  3 18:59:30) [ bafy2bzacea6eft7vcm7cpm3pni5u6slgx73u4po3dfhoeuhze5o3px7no7gue: t01000, ]
preminer-2.interop.fildev.network | CHANGED | rc=0 >>
129: (Jun  3 18:59:30) [ bafy2bzacea6eft7vcm7cpm3pni5u6slgx73u4po3dfhoeuhze5o3px7no7gue: t01000, ]
preminer-1.interop.fildev.network | CHANGED | rc=0 >>
129: (Jun  3 18:59:30) [ bafy2bzacea6eft7vcm7cpm3pni5u6slgx73u4po3dfhoeuhze5o3px7no7gue: t01000, ]

Shortly after message started to propagate and t01002 mine its first block.

129: (Jun  3 18:59:30) [ bafy2bzacea6eft7vcm7cpm3pni5u6slgx73u4po3dfhoeuhze5o3px7no7gue: t01000, ]
130: (Jun  3 19:00:00) [ bafy2bzacecykuetthfuvlhvmiyfluyceizruu44i32cqyfjllcrkf55kwh4yo: t01000, ]
131: (Jun  3 19:00:30) [ bafy2bzacebjvo327sbsd77f5cdpqojgqudvt46qlaafxbksnex4nm5wtydrrg: t01000, ]
132: (Jun  3 19:01:00) [ bafy2bzaceaffb2psjocdvubcvvhhvluxlhzlffa4kxphhlgiy5fbqa6txgme2: t01000, ]
133: (Jun  3 19:01:30) [ bafy2bzaceaa5niwhhf5pwvd345lt5wl7ce6y4r2fflmyekyafxuh4tukavjym: t01000, ]
134: (Jun  3 19:02:00) [ bafy2bzacect2v6jp6vfo2wdvpraidcr5l3y4xplqq4q4yeivfrhrd6bma4b5w: t01002,bafy2bzacedepyrq6izumx3lvgta7tidyyq5cssk2od4tgxndvemqhxt3nmeva: t01000, ]

I reset the network after this, and observed a much shorter setup time, with t01001 and t01002 joining at tipset height 42.
@BigLep BigLep added this to the Network Hyperdrive milestone Jun 4, 2021
@BigLep BigLep added this to Need Analysis in Lotus+Actors Board Jun 4, 2021
@BigLep BigLep added P1 P1: Must be resolved and removed hint/needs-triaging labels Jun 4, 2021
@arajasek
Copy link
Contributor

arajasek commented Jun 4, 2021

I reset the network after this, and observed a much shorter setup time, with t01001 and t01002 joining at tipset height 42.

@travisperson Was this much shorter setup time (tipset 42) close to normal for when you've done this in the past? Or is there still a noticeable difference?

@travisperson
Copy link
Contributor Author

Generally the additional miners will be setup within blocks 11-20. The genesis miner will initialize in ~10 tipsets, and the rest of the miners will come in just a bit after. So 42 is longer than normal, but not by a lot.

For example here is the beginning of the calibration network:

0: (Feb 19 23:10:00) [ bafy2bzaceapb7hfdkewspic7udnogw4xnhjvhm74xy5snwa24forre5z4s2lm: t00, ]
1: (Feb 19 23:10:30) [ bafy2bzacebeudsbqx7ex3zhrlgzt7uifphgianr5kg556uswpu6qbutbz3ap4: t01000, ]
2: (Feb 19 23:11:00) [ bafy2bzacedfuh7zetzkrn4h45nijs2dygy2y5qj2qdlc3n6jyynw54oqru5xy: t01000, ]
3: (Feb 19 23:11:30) [ bafy2bzacedqvyu3m6jcei2lv5ngpbttmepenzjtpy4lhi44hcon2ab4lglvao: t01000, ]
4: (Feb 19 23:12:00) [ bafy2bzaceamobsm355wazipptahajt2zo7h6bvr4akwsrcwrstzsex2fgw4kw: t01000, ]
5: (Feb 19 23:12:30) [ bafy2bzacedylr3yc27vzjsr3lln6hfsh3snaifmwvgtvywywphx4pfe6gzala: t01000, ]
6: (Feb 19 23:13:00) [ bafy2bzaceaqa3rau46mbuup7nffl2uypafludry6fotcfpe2sfb645dodywlq: t01000, ]
8: (Feb 19 23:14:00) [ bafy2bzaced2f5rz7gt64aony7bj2ojcoiowdm56lxaxv2h6sv2y5ad4xgbqum: t01000, ]
11: (Feb 19 23:15:30) [ bafy2bzacecwwjcpljbrlcuw3trhie6vpdn2zxlxe5lzsplw4fzxxi26l54zdc: t01000, ]
12: (Feb 19 23:16:00) [ bafy2bzacebyds5cxreau46k7azmfgln67s2tp67xwozjxks2lpifmsplrdruk: t01000, ]
14: (Feb 19 23:17:00) [ bafy2bzaceaffjw62dmqviutnnohs5rur265mjwocy2dd7jbqmymobupiwwhmu: t01000, ]
15: (Feb 19 23:17:30) [ bafy2bzacebqvsks6hw5z2dm7t3e4q4rxhitjzm26iwrbwb56bkryywftbp3i6: t01000,bafy2bzacedpirpqjtpvf3ainqk2ytpadigoro4zfnluhbxhmmtikniaii75r6: t01001, ]
17: (Feb 19 23:18:30) [ bafy2bzacea2ny3a6lxkoazcez3hym42nugaeyohkjoimjhq54axkrfpfkvfz6: t01001, ]
18: (Feb 19 23:19:00) [ bafy2bzacedf2xttxy6udwpbhj2hizflihkqst62afzv2po6p42ffzr3ndjm6c: t01001, ]
19: (Feb 19 23:19:30) [ bafy2bzaced4r5dareg55yxu5iofsd44hhvuypv7sybkcf3zyiojgxwshzn3fw: t01001, ]
20: (Feb 19 23:20:00) [ bafy2bzacece4tctkfo5sxdksbjfpqoom363bkz3yb7vbuxuwwupzpxjx7xkky: t01001, ]
21: (Feb 19 23:20:30) [ bafy2bzacea53ah4fahx3cxc5gnsqq4ez4pt2qny3gvktieuxzjrlryecsywny: t01001, ]
23: (Feb 19 23:21:30) [ bafy2bzaced6qdcvvghqjqzi4m4bvejcvdwv5m43ofth3tbnafzh2cz3i4ggti: t01001, ]
24: (Feb 19 23:22:00) [ bafy2bzacednlhu2xla77cuzfcgbqomx2jqeq6b5ijy72d5d5xvtkyffz6jrcm: t01001,bafy2bzacebdi4whcpmckn3v327ejthp6pzinjhc2s5trbbgbgvyefxx6b7vaq: t01000,bafy2bzacebvl7ulcyzyxu3zzzvutrtp2p5ixcbav7tbuofzgbcvcvsdcjdnbm: t01002, ]
25: (Feb 19 23:22:30) [ bafy2bzacecj27vr4ompayxnojfzgymkcpzqfvlendcz5frsxwxtsnqjh24nfo: t01001,bafy2bzacedf7gb6xyz64guqvxryitjrckkym4ghlybptcsew5gmmawjuenrrg: t01000,bafy2bzacedgxurwumuc66fa54hs3p6oln3svdjb3x376rmxa34fevudy3bajo: t01002, ]
26: (Feb 19 23:23:00) [ bafy2bzacea2hzal3x5fgbngjh3ln72kvgdtp4hzxau4e7zbkqlrml3f4mdnhu: t01001,bafy2bzacebtwgiakmgelzdmtwicrfwgpja5rin2ktrogdp7rktrfd3inlubww: t01002,bafy2bzacechwepcdn6wy3xzp7mo4qzunfs6rqxwbd5cwewn22ln2elfqfvwec: t01000, ]

@BigLep BigLep mentioned this issue Jun 4, 2021
Closed
80 tasks
@magik6k magik6k mentioned this issue Jun 11, 2021
Merged
@BigLep BigLep linked a pull request Jun 11, 2021 that will close this issue
mpool: Add more metrics #6453
Merged
@BigLep BigLep moved this from Need Analysis to In Progress in Lotus+Actors Board Jun 11, 2021
@jennijuju
Copy link
Member

related filecoin-project/specs-actors#1453

@BigLep
Copy link
Member

BigLep commented Jun 11, 2021

Next step is to investigate butterfly nodes that are manifesting this issue.

@arajasek
Copy link
Contributor

Update to:

Next step is to investigate butterfly nodes that are manifesting this issue.

The butterfly nodes were running into a Bellperson deadlock, which has since been fixed. That may have been the rootcause here, but it's unlikely -- the nodes were manifesting different symptoms.

However, we have since launched multiple networks and aren't seeing reports of this issue. If the performance metrics added in #6453 don't show anything concerning, I'd be comfortable closing this and moving forward.

@BigLep
Copy link
Member

BigLep commented Jun 18, 2021

Next step is for @magik6k to look at performance metrics on his node.

@travisperson
Copy link
Contributor Author

travisperson commented Jun 18, 2021 •
edited

On the first reset of the calibration network it doesn't appear this issue occurred at all. Even got some messages into blocks during the initial setup phase.

1: (Jun 18 23:06:30) [ bafy2bzacea4jwiohzt7trznxcxdvucl7wdysi3v7xwilsgtsc33pj7cgpvjus: t01000, ]
2: (Jun 18 23:07:00) [ bafy2bzacec7rbsuv4bu4q7egvvcmdt3qfezvf3d6c5lk3dzfkjyoq6t4leh4y: t01000, ]
4: (Jun 18 23:08:00) [ bafy2bzacedg4quagnb3kjcwc6vxxomuae37ai2jgbn273hd4cr63ngqh3koza: t01000, ]
5: (Jun 18 23:08:30) [ bafy2bzacebpppe76pmuyifwt5hifrau2cchsoloow5qmqzoin7vl7qyeatfau: t01000, ]
6: (Jun 18 23:09:00) [ bafy2bzacebe4fnwmxhvnxsl3cou5ybc5anol3jfcryhit7hrv4lekkfrbt4uc: t01000, ]
7: (Jun 18 23:09:30) [ bafy2bzacea6xx2hv4ifvmrff227xuf7b6gqv3fd4p6qoe76n3xrweldtbwha2: t01000, ]
8: (Jun 18 23:10:00) [ bafy2bzacebtk5dp3kztdieofohdhyp6navhevgrb32auz6wditmkzlzkavkuw: t01001,bafy2bzacec7yz7q2bnll4jrsmx4xlvriirvbzfsrcqdwusfnnk5xkpoyjvkxy: t01002, ]
9: (Jun 18 23:10:30) [ bafy2bzacecyxlzjepewvbgsbfhp3xk5xs7oec25agldh4tbvusm5rhum4tfbi: t01001,bafy2bzacebckt4dlvzpesybor2w4ci2wkj6oy234mamhjbj6k4hppn7ljikh4: t01002, ]
10: (Jun 18 23:11:00) [ bafy2bzacec45rwfydxbl5xhasspe5xxxongstkoeebdumsjhvryx25sox4ouk: t01001,bafy2bzacea64sqivurzwwqnl7ty6pn64ihybbibmz2wcddzvku3ojimwxf5r2: t01002, ]
11: (Jun 18 23:11:30) [ bafy2bzacebfohnnnhwldbw5f32iqacgnfcgkrbsds2g253dpkfdpqecf4kbm4: t01001, ]
12: (Jun 18 23:12:00) [ bafy2bzaceb67ir4bwxrccxqg72xwkepcgm7i5ksibald5oly76wvr3hf2liuw: t01001,bafy2bzacecbsazgtxvgrd3fbvbmm62bptqeqyumfyhxacr4eoi4nolkmlztsa: t01000, ]

@arajasek
Copy link
Contributor

Closing because this seems to have been a mysterious occurrence.

Lotus+Actors Board automation moved this from In Progress to Closed Jun 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@travisperson travisperson

@magik6k magik6k

@arajasek arajasek

Labels
kind/bug Kind: Bug P1 P1: Must be resolved
Projects
Lotus+Actors Board
Closed
Milestone
Network Hyperdrive
Development

Successfully merging a pull request may close this issue.

mpool: Add more metrics filecoin-project/lotus
5 participants
@BigLep @travisperson @magik6k @arajasek @jennijuju