Reconnect breaks connectivity since 1.2.4, 1.2.5

[xnoreq]

Since #4325 was closed but this issue was not resolved I'm opening a new issue for this.

Setup:

• I'm using qbittorrent
• bound to any interface/IP address
• my "eth0" interface has very limited connectivity to specific networks, no Internet access
• vpn connected through tun0 with Internet access

How to reproduce:

• reconnect vpn while qbittorrent is running

Outcome:

• trackers stop working
• DHT nodes drop to 0

Expected outcome:

• trackers reconnect after vpn reconnection
• DHT nodes don't drop

It looks like on vpn disconnect libtorrent disables and removes everything (such as DHT nodes) associated with the vpn0 IP address bound socket instead of leaving these things in place to time out or get picked up again if a reconnection happens in time.

[arvidn]

Please provide more information on exactly how trackers stop working. What’s the error message in the tracker error alert? Or do trackers stop attempting to announce?

A verbose log with torrent log notifications enabled would also probably be very helpful.

After you stop and start your VPN, how long do you wait for peers, trackers and DHT nodes to reconnect?

The thing with the DHT is that the routing table depends on your node id, which depends on your external IP. If your IP changes, the routing table needs to be changed. The nodes aren’t dropped, but most of them may no longer have a place in your new routing table. (And if they can’t be contacted, they will likely be dropped).

[arvidn]

I cannot reproduce this. I use private internet access on linux and I disconnect it and reconnect it. Using client_test and a bunch of torrents.

I can confirm that the DHT routing table is cleared out immediately when changing out the network interfaces, but it starts to build up again within a few seconds (say, 5-10 seconds).

What kind of VPN do you have? OpenVPN or wireguard?

[arvidn]

I've tested with both private internet access (OpenVPN) and nordnet (OpenVPN and wireguard) on ubuntu 19.10, without encountering this problem.

Which operating system are you on?

[xnoreq]

simple_client binds to 0.0.0.0:6881 just like qbittorrent did with libtorrent before 1.2.4 so it probably doesn't care if the interfaces goes down or the IP address is removed from it.

What's curious is that after reconnecting the VPN and keeping qbittorrent running but removing and re-adding the torrent it started downloading again but it seems to have rediscovered some peers and starting downloading again:

DHT nodes also stay 0.

Restarting qbittorrent results in:

[xavier2k6]

@xnoreq

Which operating system are you on?

you may have missed this...

[xnoreq]

The vpn uses openvpn with a script that executes on up:

ip link set dev "tun0" up mtu "$tun_mtu"
# setting up network schedulers
# setting up firewall rules
ip addr add dev "tun0" "${ip_local}/${netmask}"

And executes this on route-up:

ip route add 0.0.0.0/1 via "$gateway" dev "tun0"
ip route add 128.0.0.0/1 via "$gateway" dev "tun0"

I'm on Arch Linux, kernel 5.5.8, OpenVPN 2.4.8.

[mgaulton]

I'm experiencing something like this potentionally.
Opensuse Tumbleweed, libtorrent 1.2.5 and deluge, openvpn tun0 in a linux namespace for isolation.
No connectivity, lots of failures to talk to trackers showing in the logs, no activity other than DNS.
Downgrading to 1.2.3 fixed it for a time, but even with locking the package, when tumbleweed moved to python 3.8, it broke again for other reasons.
Hoping the package maintainers for opensuse get 1.2.5 soonish :)

[arvidn]

@mgaulton would you mind posting the errors you see in the log?

[arvidn]

Even without a VPN, when you lose connection

By "losing connection", you mean unplugging and plugging back in the ethernet cable? I imagine you have to lose connection for at least one announce cycle for the trackers to be affected, is that right?

I suspect this change is responsible for this. I should revert that.

Any chance someone could give that a try?

[arvidn]

well, actually, a failure because the DNS server cannot be reached is a different error than the one I'm checking for in that patch. There are two kinds of "host not found" errors, authoritative and non-authoritative. The former (the error I check for) means the DNS server responded with NXDOMAIN error, i.e. the host name does not exist.

You shouldn't be getting an NXDOMAIN error by just losing connection. But, perhaps there are routers or ISPs in widespread use that get this wrong, and it's unreliable.

[arvidn]

@An0n666 if you experience this, could you test dig www.google.com for instance, when you have lost connectivity?

[swejuggalo]

I did a test with freshly built libtorrent-git and qbittorrent-git on Manjaro.
Qbittorrent 4.2.2 (official build with 1.2.5) on Windows resumes as normal.
On Manjaro with the self built versions via AUR it does not resume and DHT goes at 0. I can see a temporary fragment of an upload but that is about it.

I can probably catch some logs (just let me now what and how I should catch them for you) and rebuild with extra options if nessesary.

[xnoreq]

Anyway, I'm stuck on af12f5d for the time being until these issues are fully resolved.

[arvidn]

trying to reproduce this issue, I'm on ubuntu with private internet access VPN. I'm using head of RC_1_2, which does have a few fixes since 1.2.5.

./client_test -f client-test.log -s . --alert_mask=error,connect,tracker --outgoing_interfaces=tun0 --listen_interfaces=tun0:12345 ubuntu-19.10-desktop-amd64.iso.torrent

When disabling the VPN, trackers fail with "No route to host" (as expected). When I reconnect the VPN and re-announce, trackers work and peers reconnect. The DHT routing table starts to build up again.

I could use some help understanding what other settings may affect this behavior.

[arvidn]

@xnoreq that commit is very recent on RC_1_2. Are you saying the issue is fixed in RC_1_2 then? and the "fully resolved" refers to a new release being made, is that right?

[swejuggalo]

trying to reproduce this issue, I'm on ubuntu with private internet access VPN. I'm using head of RC_1_2, which does have a few fixes since 1.2.5.

./client_test -f client-test.log -s . --alert_mask=error,connect,tracker --outgoing_interfaces=tun0 --listen_interfaces=tun0:12345 ubuntu-19.10-desktop-amd64.iso.torrent

When disabling the VPN, trackers fail with "No route to host" (as expected). When I reconnect the VPN and re-announce, trackers work and peers reconnect. The DHT routing table starts to build up again.

I could use some help understanding what other settings may affect this behavior.

Loosing internet, like in my case when rebooting router trigger the same loss of DHT and never recover it after reconnecting. I assume trackers fail too. And in my case no VPN involved.
But 1.2.5 in windows with qbittorrent 4.2.2 does not have that behavior. Find that odd.

I have never used that command. Running it from a libtorrent-rasterbar installation directory I assume?

[xnoreq]

@arvidn No, I initially pasted the wrong hash. I guess you're reading the mails and corrections through edits are not sent as mails again.

The commit is from 11th of January.

[xnoreq]

I think the issue is with qbittorrent(-nox). Not sure what needs to be implemented to get the reconnection working on the side that uses libtorrent.

[xnoreq]

Update: latest libtorrent and qbittorrent master doesn't even connect to peers anymore.

[arvidn]

@xnoreq under what conditions?

[swejuggalo]

@xnoreq under what conditions?

Replicated the same thing with libtorrent and qbittorrent (latest git) built earlier today. Simply adding a torrent and the traffic does not start at all. I don't remember the details of the commits in both projects... I'll take a look an see if I can guess what change that might have broken something :)

Qbittorrent did some recent peer related changes, but to webui...
I'm not 100% from what date it broke...but I think it worked before today commits.

Rolling back to stable git version of Qbittorrent fixes the problem. Download starts. Going back to latest git breaks it again

[arvidn]

"latest git" means head of RC_1_2 in libtorrent?

[xnoreq]

@arvidn Yes, RC_1_2 branch of libtorrent, qbittorrent master.

Going back to af12f5d works also with qbittorrent master.

[xnoreq]

Still doesn't work and I don't see how that commit would fix the described issue.

[arvidn]

@xnoreq could you provide logs from client_test, with alert_mask=all?

The issue I can reproduce on MacOS when disabling and re-enabling the network interface, is that the process is notified of the network coming up before the routing table is set up. The notification causes the listen sockets to be opened up again, and part of that logic is to figure out which networks it makes sense to run a DHT node on. For this, it looks at routing table for routes to the internet.

On MacOS the routing table can be empty right at that point. So some of the logic that looks at the routing table is disabled in case it's empty. That's what this patch does.

[ghost]

@xnoreq
Someone is claiming that listening only on IPv4 solves the problem.
Can you test and confirm? qbittorrent/qBittorrent#15253 (comment)
You'll have to choose your specific VPN interface and then set optional IP address to bind to option to "All IPv4 Addresses".

Also the claim in this comment qbittorrent/qBittorrent#13794 (comment) refers to a line of code that indicates a change related to IPv6 which might have introduced this issue.

[arvidn]

I can't reproduce this problem with this patch applied #6338

I tested with a VPN as well, listening to a specific interface.

[ghost]

@sledgehammer999 can you please provide a qBt 4.3.7 win x64 build with latest RC_1_2? I want to test if this issue is fixed on Windows with the latest commit.

[arvidn]

looking over this thread again, scratch my comment about the PR "fixing it".

It sounds more likely the problem is:

• the initial DHT bootstrap happens while there is no network connectivity

• the router node can't be found, any saved DHT nodes are tried and fail and are discarded

• bootstrap as a whole fails, we have no more known peers

• network connectivity is restored but the DHT bootstrap doesn't trigger. If network connectivity is happening away from the main computer (say, by restarting the router), there is no trigger on the machine running libtorrent to notify it of there being a route to the internet.

[Wolfie713]

Rather interesting thing happened for me earlier. Had to relaunch qBit because of a connection reset and a couple of torrents that were pending for download started up right away (connected to trackers just fine) while those for seeding took awhile to finally start working. Not sure if it was just a fluke/coincidence, or if there might be something about it that makes a difference. To the same trackers, for the record, so not like the downloads worked because of being from a different source.

[gothicserpent]

Just wanted to chime in here. I had this issue for a few versions of qBit, and I have been resolving it when it happened by simply pausing and resuming all torrents which fixed it.

The issue happens for me when my internet cable modem is restarted / my ISP shuts it down for maintenance for a few minutes, thereby triggering a VPN re-connection. It does not happen for me when I restart my VPN or change VPN servers.

Even when my modem is restarted, the DHT table gets spun back up in a few seconds, its just the trackers that show errors such as not contacted yet not not working. Again, pausing all torrents and resuming all torrents seems to resolve it.

As a strong test method, I would recommend trying to restart your modems / routers / ISP connections and see if it lets you reproduce this.

[ghost]

Just wanted to chime in here. I had this issue for a few versions of qBit, and I have been resolving it when it happened by simply pausing and resuming all torrents which fixed it.

The issue happens for me when my internet cable modem is restarted / my ISP shuts it down for maintenance for a few minutes, thereby triggering a VPN re-connection. It does not happen for me when I restart my VPN or change VPN servers.

Even when my modem is restarted, the DHT table gets spun back up in a few seconds, its just the trackers that show errors such as not contacted yet not not working. Again, pausing all torrents and resuming all torrents seems to resolve it.

As a strong test method, I would recommend trying to restart your modems / routers / ISP connections and see if it lets you reproduce this.

This issue hasn't been fixed yet.

[gothicserpent]

Yes, it still occurs for me if my ISP has a maintenance outage thereby causing a vpn restart. I just workaround by pausing and resuming all torrents.

This issue may be more easy to reproduce while testing if the cable modem is reset thereby causing a restart of the VPN.

Resetting my modem (which in my case is holding the modem power button down until there's a red led indicator) always lets me reproduce it.

[ghost]

I have upgraded from Deluge 1.3.15(libtorrent 1.1.14) to Deluge 2.0.5(libtorrent 1.2.15) 6 days ago.

I just noticed that after running for 5-6 days, none of the trackers were working on any of my seeding torrents with the error code:
No route to host.

However some torrents were still uploading to peers.

Even force reannounce didn't fix the trackers. Had to restart Deluge and it started working immediately.

[ghost]

@xnoreq can you please test if the latest commits fix the tracker not working issue?

[ghost]

The issue with DHT not recovering is still present on release 2.0.8.

[gothicserpent]

No longer happens for me as of 2.0.7.0 onwards.

Once my VPN cycles to a new server and the connection is reset, trackers reconnect and DHT nodes re-establish.

Haven't had this issue in a while actually.

[ghost]

No longer happens for me as of 2.0.7.0 onwards.

Once my VPN cycles to a new server and the connection is reset, trackers reconnect and DHT nodes re-establish.

Haven't had this issue in a while actually.

You should also pause all torrents and then stop/start the VPN. See if DHT nodes increase or remain at zero.

[EZ64cool]

I'm also running into this issue, I tend to connect to the same VPN server after every reconnect so the external IP shouldn't be changing either.
I can pepro this 100% of the time with qbittorrent and AirVpn with the help of their open port checker. Street initial boot the port is open but once a reconnect occurs the port ends up being closed.