Lost all connections, out of nothing

[xeroc]

Hey there,

As you can read from
https://bitsharestalk.org/index.php?topic=6003.0;all
my delegate missed several blocks. Reason for it was that all connections were lost and the client did not try to (our could not) connection to other hosts.

Once I restarted the client (twice) the connection to the seed node was made again and I have 40+ connections.

You cann find all logs over here:
http://bitshares.xeroc.org/logs/

[wackou]

same happened to me (right after I went so sleep, lucky me... 😢 ). I don't have the logs (shitty internet and no time to investigate right now), but this might still be related to a file descriptor leak, as I exited the client, restarted it immediately, and got 1 connection but no more. I then rebooted the machine, relaunched the client, and got all connections immediately.

[wackou]

If that helps, I'm running on Debian Wheezy on a VPS.

[wackou]

Another note: maybe there could be a limit to the number of connections per second that are made? Because my client keeps connecting and disconnecting real fast:

--- there are now 27 active connections to the p2p network
--- there are now 28 active connections to the p2p network
--- there are now 27 active connections to the p2p network
--- there are now 26 active connections to the p2p network
--- there are now 27 active connections to the p2p network
--- there are now 26 active connections to the p2p network
--- there are now 27 active connections to the p2p network
--- there are now 28 active connections to the p2p network
--- there are now 27 active connections to the p2p network
--- in sync with p2p network
--- there are now 28 active connections to the p2p network
--- there are now 27 active connections to the p2p network
--- there are now 26 active connections to the p2p network
--- there are now 27 active connections to the p2p network
--- there are now 26 active connections to the p2p network
--- there are now 27 active connections to the p2p network
--- in sync with p2p network
--- there are now 28 active connections to the p2p network
--- there are now 27 active connections to the p2p network
--- there are now 26 active connections to the p2p network
--- there are now 27 active connections to the p2p network
--- there are now 26 active connections to the p2p network
--- there are now 27 active connections to the p2p network
--- there are now 28 active connections to the p2p network
--- there are now 29 active connections to the p2p network
--- there are now 28 active connections to the p2p network
--- syncing with p2p network, 506 blocks left to fetch
--- there are now 27 active connections to the p2p network
--- there are now 28 active connections to the p2p network
--- there are now 27 active connections to the p2p network
--- there are now 26 active connections to the p2p network
--- there are now 27 active connections to the p2p network

This happened in maybe 2-3 seconds, and it keeps doing it...

[maqifrnswa]

same thing happened to me:
I had 100 connections, then suddenly 0get_info
{
"blockchain_head_block_num": 36910,
"blockchain_head_block_age": "6 minutes old",
"blockchain_head_block_timestamp": "20140723T154130",
"blockchain_average_delegate_participation": 71.63120567375887,
"blockchain_delegate_pay_rate": 213568,
"blockchain_blocks_left_in_round": 56,
"blockchain_confirmation_requirement": 1,
"blockchain_accumulated_fees": "258,332.52201 BTSX",
"blockchain_share_supply": 199974166729722,
"blockchain_random_seed": "2db3d621b4f99521b2e756cb319e24dc6980bd88",
"blockchain_database_version": 115,
"blockchain_version": 109,
"network_num_connections": 0,
"network_num_connections_max": 100,
"network_protocol_version": 104,
"ntp_time": "20140723T154758",
"ntp_error": -0.015782000000000001,
"wallet_open": true,
"wallet_unlocked": true,
"wallet_unlocked_until": "3 years 2 months in the future",
"wallet_unlocked_until_timestamp": "20170922T023602",
"wallet_block_production_enabled": true,
"wallet_next_block_production_time": "16 minutes in the future",
"wallet_next_block_production_timestamp": "20140723T160340",
"wallet_version": 100
}

everyone seems to be on TIME_WAIT
showard@maqifrnswa-delegate:$ netstat
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 162.243.220.142:ssh 110.45.244.147:65245 SYN_RECV
tcp 0 0 162.243.220.142:1776 180.157.15.102:54901 TIME_WAIT
tcp 0 0 162.243.220.142:1776 loft7678.serverlo:36793 TIME_WAIT
tcp 0 0 162.243.220.142:1776 19.Red-81-34-98.d:56837 TIME_WAIT
tcp 0 0 162.243.220.142:ssh REDACTED:38563 ESTABLISHED
tcp 0 0 162.243.220.142:1776 minecraft.die-sch:57164 TIME_WAIT
tcp 0 0 162.243.220.142:1776 ec2-54-191-20-243:46927 TIME_WAIT
tcp 0 0 162.243.220.142:1776 222.249.152.189:58966 TIME_WAIT
tcp 0 0 162.243.220.142:1776 127.149.33.58.broa:1776 TIME_WAIT
tcp 0 0 162.243.220.142:1776 242.22.156.27.bro:17956 TIME_WAIT
tcp 0 0 162.243.220.142:1776 loft7678.serverlo:36801 TIME_WAIT
tcp 0 0 162.243.220.142:1776 ec2-54-77-78-162.e:1776 TIME_WAIT
tcp 0 0 162.243.220.142:1776 115.215.37.93:16304 TIME_WAIT
tcp 0 0 162.243.220.142:1776 srn-027-223.ddi.s:50752 TIME_WAIT
tcp 0 0 162.243.220.142:1776 119.254.110.18:41110 TIME_WAIT
tcp 0 0 162.243.220.142:1776 aftr-37-24-156-10:11430 TIME_WAIT
tcp 0 0 162.243.220.142:1776 223.199.16.113:56790 TIME_WAIT
tcp 0 433 162.243.220.142:1776 59.61.216.123:50428 ESTABLISHED
tcp 0 0 162.243.220.142:1776 175.154.203.195:2090 TIME_WAIT
tcp 0 0 162.243.220.142:1776 cpc67367-blbn9-2-:51135 TIME_WAIT
tcp 0 0 162.243.220.142:1776 101.26.123.45:18194 TIME_WAIT
tcp 0 433 162.243.220.142:1776 175.154.203.195:2240 ESTABLISHED
Active UNIX domain sockets (w/o servers)
Proto RefCnt Flags Type State I-Node Path
unix 4 [ ] DGRAM 7566 /dev/log
unix 3 [ ] DGRAM 7289
unix 3 [ ] STREAM CONNECTED 7545 /var/run/dbus/system_bus_socket
unix 3 [ ] DGRAM 7288
unix 3 [ ] STREAM CONNECTED 7476 /var/run/dbus/system_bus_socket
unix 3 [ ] STREAM CONNECTED 7475
unix 3 [ ] STREAM CONNECTED 7544
unix 3 [ ] STREAM CONNECTED 7474 @/com/ubuntu/upstart
unix 3 [ ] STREAM CONNECTED 8166
unix 2 [ ] DGRAM 8806
unix 2 [ ] DGRAM 153581
unix 3 [ ] STREAM CONNECTED 153757
unix 3 [ ] STREAM CONNECTED 7237
unix 3 [ ] STREAM CONNECTED 7248 @/com/ubuntu/upstart
unix 3 [ ] STREAM CONNECTED 8441 @/com/ubuntu/upstart
unix 3 [ ] STREAM CONNECTED 7430
unix 3 [ ] STREAM CONNECTED 7456
unix 3 [ ] STREAM CONNECTED 153766 /var/run/dbus/system_bus_socket
unix 3 [ ] STREAM CONNECTED 153765
unix 3 [ ] STREAM CONNECTED 7455
unix 3 [ ] STREAM CONNECTED 153758
showard@maqifrnswa-delegate:$

[dnotestein]

There's a known fix for the connecting/reconnecting problem that will be in the next release.

[maqifrnswa]

xeroc, do you use digitalocean nyc2 as your vps by any chance?

[vikramrajkumar]

This seems to be gone after the latest round of networking+threading fixes. Delegate uptime is higher than ever.