wss://02.tfchain.grid.tf/ is behind acceptable delay - Context dead line exceeded.

[mahendravarmayadala93]

A client with farm 1141 and the node IDs 3452 and 3679 are repeatedly encountering "Context deadline exceeded" even after multiple reboots and LAN port changes.

[TullysInc]

@muhamadazmy - Can we have your thoughts on these errors, please?

As per Scott's recommendation - we suggested setting the clock by ntp over the network, but the farmer states after several reboots, and using new boot images these errors continue to persist. Any suggestions for a work around?

[muhamadazmy]

I think there were 2 issues here:

• For some reason the real time clock of the machine is really behind. This caused SSL connections to be rejected. This should be fixed by NTP server running on the node. NTP on the otherhand can give up if the skew is too great and has then to be set manually. A possible solution was to first fix the time manually in the bios and then let ntp server fix the accuracy.
• The last error message is the node just waiting to get an IPv6, it will keep trying for a while, but then it should give up and continue after 4 minutes. So your node should show that it's waiting for slaac to give ipv6 for around 4 minutes before it continues. During the wait time the networkd/test process will keep failing waiting in networkd to finish setup.

How long does the networkd/test persist ? is it longer than 4 minutes? otherwise this can be an issue in the mechanism that causes the nodes to get stuck on boot

[TullysInc]

Thanks for the detailed insight into the 2 issues, Azmy.

How long does the networkd/test persist ? is it longer than 4 minutes? otherwise this can be an issue in the mechanism that causes the nodes to get stuck on boot

As I understand from the farmer's description - it does appear longer than 4 minutes, and has occurred on several reboot attempts. Is there a suggestive workaround for the mechanism issue?

The NTP issue seems to be specific to just one Node 3679.

[Cameron1986labeyrie]

@muhamadazmy - Is there any update on this? The farmer has returned and reported that the issue remains unresolved

" four of my nodes are already affected by this problem. 3032, 3452, 2596 ( Dell R620 ) and 3679 ( HP DL360 Gen9 ) "

[Cameron1986labeyrie]

@muhamadazmy - Do we have any updates for this Farmer? The farmer came back again stating "Just want to let you know that 6 of my nodes except for 905, 1745, and 3784 are experiencing the same problem, hanging on the blank screen and waiting for IPV6 yggdrasil connection."

[Cameron1986labeyrie]

@muhamadazmy - Any positive update for this ticket?

"Most of my nodes are affected except for 905,1745 and 4008."

[muhamadazmy]

Hi I did some deep investigation of the problem and here is what i found:

• The networkd is not "stuck" on waiting for ipv6 although this shows the most on the logs this is eventually skipped by the system and networkd moves on.
• shortly after that part is skipped networkd hits another fatal error which makes it exit, and restarted.
• networkd then start the process again, waiting another 2 minutes for ipv6, and so on.
• So it was important to figure out what causes the crash later on, I scanned a lot of logs until i find this out.

• you can see that, networkd when tries to contact the chain for some network related configs, it fail with this error. and it looks like that the node is running ahead of the chain by around 18 seconds!

• 18 seconds is a big time difference and hence the request is rejected, after retrying few times, it then goes into fatal error, networkd exits, then restarted by zos to start over again

• The next question was, why the node is running ahead of the correct time. ALL zos nodes run NTPD which makes sure the localtime is in sync with accurate ntp daemons.

• Then i checked the ntp logs for the same time when we are having this error and this is what I found
  

• NTP is having hard time connectiong to global ntp daemons/server. This brings us back to farmer networkd

• The only thing i can think of is that

  • farm router somehow is blocking ntp traffic
  • other issue with farm router (bad gateway, bad dns, etc...)
  • ISP issues (ntp traffic is blocked, or filtered ...) don't think it's the case but worth investigating.

• The farmer need to test NTP traffic from his own network, they can run a live ubuntu server on the same hardware, and see (may be manually run ntp) to see if it gives any similar errors) then find out way.

[muhamadazmy]

I really think this is not an issue from our side but is probably a network on the farmer side that causes ntp to not synchronise as expected causing this issue. Hence I will move it to blocked until we have more info