Node Crash [ DELTA 8 ]

[matsuro-hadouken]

Based on:

I've found the "scheduler_queue_total_count" to be a more useful one tbh. It shows the current total backlog of these kind of events waiting to be processed. We aim for it to always be very low - ideally single or double digits. When we saw the recent OOM issues, before the node crashed, that value would invariably skyrocket to hundreds of thousands - meaning we had a huge backlog of events

Here is the moment of actual crash: ( 2021-01-10 14:11:52 UTC )

Since we sure this is OOM most luckily:

Here is no doubts, the reason for node to loose ability providing us with HTTP and RPC end point is coming from the same source. We can observe it on mostly every graph, as example I post graphs below.

From here we can see, issue start accumulating before the actual catastrophe ( 2021-01-10 06:27:00 UTC )

• Update 1

I been tracking network activity, and here is some in case can be usefull.

From here we can see, since my crawler still crawling after node colapse, new IP addresses still joining chain.

At moment of crash my node have 69 peers connected, auction order book have 65 bids and network coun 132 unique IP addresses. My validator crash happen when 133 addresses been count, it was exactly on the moment when new IP join network, not sure if this can be related to some specific missconfigured node joining.

• Update 2

Last IP's joined network:

130 95.216.224.235
131 95.217.0.233
132 95.217.154.113
133 95.217.154.200
134 95.217.213.84
135 95.217.84.84
136 95.217.94.83

• Update 3

List of current "still alive peers":

 104.197.195.68  Era: 101       Height: 2647  
 104.236.61.140  Era: 101       Height: 2647  
 107.22.247.79   Era: 101       Height: 2647  
 116.202.106.191 Era: 83        Height: 2277  
 116.202.24.91   Era: 80        Height: 2205  
 116.203.157.37  Era: 101       Height: 2647  
 116.203.206.169 Era: 101       Height: 2647  
 116.203.214.160 Era: 101       Height: 2647  
 116.203.254.245 Era: 101       Height: 2647  
 116.203.69.88   Era: 101       Height: 2647  
 129.146.102.188 Era: 101       Height: 2647  
 129.146.102.28  Era: 101       Height: 2647  
 129.146.137.190 Era: 101       Height: 2647  
 129.146.166.34  Era: 101       Height: 2647  
 129.146.17.202  Era: 101       Height: 2647  
 129.146.243.139 Era: 101       Height: 2647  
 129.146.58.40   Era: 101       Height: 2647  
 129.146.83.137  Era: 101       Height: 2647  
 134.209.16.172  Era: 101       Height: 2647  
 135.181.129.210 Era: 101       Height: 2647  
 135.181.143.233 Era: 97        Height: 2605  
 135.181.163.79  Era: 101       Height: 2647  
 135.181.192.149 Era: 101       Height: 2647  
 135.181.25.45   Era: 101       Height: 2647  
 135.181.30.191  Era: 101       Height: 2647  
 135.181.35.187  Era: 101       Height: 2647  
 135.181.40.95   Era: 101       Height: 2647  
 135.181.89.76   Era: 101       Height: 2647  
 135.181.97.186  Era: 101       Height: 2647  
 13.68.180.213   Era: 101       Height: 2647  
 138.197.65.212  Era: 101       Height: 2647  
 143.110.225.98  Era: 101       Height: 2647  
 144.76.199.44   Era: 101       Height: 2647  
 144.91.109.12   Era: 101       Height: 2647  
 157.90.19.142   Era: 101       Height: 2647  
 157.90.26.194   Era: 101       Height: 2647  
 158.101.3.185   Era: 101       Height: 2647  
 158.247.204.136 Era: 101       Height: 2647  
 161.97.124.55   Era: 101       Height: 2647  
 161.97.152.70   Era: 101       Height: 2647  
 161.97.82.77    Era: 101       Height: 2647  
 161.97.96.132   Era: 101       Height: 2647  
 167.86.75.50    Era: 101       Height: 2647  
 168.119.123.213 Era: 101       Height: 2647  
 168.119.173.204 Era: 101       Height: 2647  
 168.119.179.150 Era: 101       Height: 2647  
 168.119.59.64   Era: 101       Height: 2647  
 172.105.13.176  Era: 101       Height: 2647  
 173.212.205.2   Era: 101       Height: 2647  
 194.19.235.78   Era: 101       Height: 2647  
 195.201.148.78  Era: 96        Height: 2565  
 195.201.220.118 Era: 101       Height: 2647  
 207.180.198.56  Era: 101       Height: 2647  
 207.180.245.251 Era: 101       Height: 2647  
 3.19.76.184     Era: 101       Height: 2647  
 34.121.43.83    Era: 101       Height: 2647  
 34.65.104.178   Era: 101       Height: 2647  
 35.198.176.85   Era: 101       Height: 2647  
 35.233.165.80   Era: 101       Height: 2647  
 40.75.89.48     Era: 101       Height: 2647  
 46.4.15.204     Era: 101       Height: 2647  
 49.12.109.102   Era: 101       Height: 2647  
 49.12.66.192    Era: 97        Height: 2605  
 49.12.96.112    Era: 101       Height: 2647  
 52.247.126.146  Era: 101       Height: 2647  
 52.247.219.167  Era: 101       Height: 2647  
 78.46.121.69    Era: 101       Height: 2647  
 78.46.186.163   Era: 101       Height: 2647  
 78.47.10.119    Era: 101       Height: 2647  
 78.47.110.240   Era: 101       Height: 2647  
 78.47.9.189     Era: 101       Height: 2647  
 88.198.227.113  Era: 101       Height: 2647  
 88.99.120.29    Era: 101       Height: 2647  
 88.99.36.131    Era: 101       Height: 2647  
 94.130.224.158  Era: 101       Height: 2647  
 95.216.147.88   Era: 101       Height: 2647  
 95.216.164.122  Era: 101       Height: 2647  
 95.217.154.113  Era: 101       Height: 2647  
 95.217.154.200  Era: 101       Height: 2647  
 95.217.213.84   Era: 101       Height: 2647  
 95.217.84.84    Era: 101       Height: 2647 

Dead peers:

103.212.121.38                               
116.202.17.49                                
116.203.157.213                              
116.203.182.37                               
135.181.118.39                               
135.181.147.7                                
135.181.152.29                               
135.181.42.149                               
13.57.200.251                                
144.76.111.74                                
144.91.123.104                               
144.91.83.59                                 
157.90.21.155                                
157.90.26.198                                
158.101.12.238                               
159.146.54.127                               
159.69.218.20                                
161.97.136.153                               
161.97.143.67                                
167.99.74.47                                 
168.119.126.225                              
168.119.169.105                              
168.119.177.31                               
168.119.177.37                               
168.119.250.90                               
168.119.49.220                               
168.119.59.12                                
173.212.200.158                              
173.249.31.34                                
18.144.176.168                               
18.217.179.179                               
20.57.161.70                                 
206.189.4.64                                 
207.180.199.157                              
207.180.213.119                              
207.180.243.57                               
34.122.124.107                               
34.123.112.124                               
34.75.153.42                                 
35.236.56.221                                
35.239.228.18                                
45.61.52.142                                 
46.4.91.24                                   
62.171.165.37                                
78.46.198.83                                 
78.46.200.17                                 
78.47.131.222                                
78.47.18.35                                  
78.47.218.195                                
88.198.200.113                               
88.99.171.225                                
88.99.227.165                                
95.216.224.235                               
95.217.0.233                                 
95.217.94.83

Last delta-8 geomap:

Geodata CSV attached
delta-8_geo_data.txt

[goral09]

Thanks @matsuro-hadouken , we're already looking into this and have some turnovers. Apologies for not getting back to you.

[sacherjj]

Many updates were made for Delta-10, including network changes and optimization to eliminate OOM crashes. With heavy load in Delta-10 with up to 100 wasm deploys we are not longer seeing this occur.