Peer messages causing socket error

[zalloy]

Nicotine+ version: 1.4.3 unstable
Operating System/Distribution: Kubuntu 20.04 (Focal)

Description: After doing a few searches, I get a lot of peer messages showing up in the log window, followed by "Major Socket Error: Networking terminated! [Errno 24] Too many open files". The peer messages are a variety of types (1028, 3333, 1541, 260, 1797, 14592, 10752), and are coming from a number of different users.

Expected Behavior: I don't think I should be seeing these peer messages popping up, and shouldn't be getting the socket error.

Steps to reproduce:

1. Open Nicotine+ and connect to the network.
2. Search for a few different artists (this time, I searched for Led Zeppelin, Heart, and U2).
3. After a few minutes, you should see the messages in the log window.

Additional context:
Screenshot

[mathiascode]

Has this happened before, or approximately when did it start happening?

[hboetes]

I had this too, turned out I had to increase the number of open files allowed. This can be done with ulimit and/or whatever mechanism used in your OS.

[mathiascode]

Might have to tweak the values a bit again, it shouldn't have to be necessary to manually allow more open files.

[hboetes]

On OpenBSD it looks like this in /etc/login.conf`

        :openfiles-max=1024:\
        :openfiles-cur=512:\

So without user intervention 512 would be the limit.

[mathiascode]

What is the output of ulimit -H -a (hard limits) and ulimit -S -a (soft limits)?

[hboetes]

I take it you want the default values for mortal users. Not the output for me.

$ uname -a
OpenBSD dahud 6.7 GENERIC.MP#2 amd64
dahud$ ulimit -H -a
time(cpu-seconds)    unlimited
file(blocks)         unlimited
coredump(blocks)     unlimited
data(kbytes)         786432
stack(kbytes)        32768
lockedmem(kbytes)    4038752
memory(kbytes)       4038752
nofiles(descriptors) 1024
processes            256
$ ulimit -S -a 
time(cpu-seconds)    unlimited
file(blocks)         unlimited
coredump(blocks)     unlimited
data(kbytes)         786432
stack(kbytes)        4096
lockedmem(kbytes)    1346250
memory(kbytes)       4035896
nofiles(descriptors) 512
processes            128

[mathiascode]

Damn, that's an awfully low hard limit. Oh well, perhaps a 512 Nicotine+ limit on these kind of systems will do to prevent complaints about too many open files.

[hboetes]

Yes, typical OpenBSD stuff. Perhaps you could make nicotine+ read the limit and use what's available, with an upper limit or something like that? Post a warning to the logs when it's ridiculously low?

[zalloy]

For me, the output of ulimit -H -a is:

-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             unlimited
-c: core file size (blocks)         unlimited
-m: resident set size (kbytes)      unlimited
-u: processes                       127868
-n: file descriptors                1048576
-l: locked-in-memory size (kbytes)  65536
-v: address space (kbytes)          unlimited
-x: file locks                      unlimited
-i: pending signals                 127868
-q: bytes in POSIX msg queues       819200
-e: max nice                        0
-r: max rt priority                 0
-N 15:                              unlimited

The output of ulimit -S -a is:

-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8192
-c: core file size (blocks)         0
-m: resident set size (kbytes)      unlimited
-u: processes                       127868
-n: file descriptors                1024
-l: locked-in-memory size (kbytes)  65536
-v: address space (kbytes)          unlimited
-x: file locks                      unlimited
-i: pending signals                 127868
-q: bytes in POSIX msg queues       819200
-e: max nice                        0
-r: max rt priority                 0
-N 15:                              unlimited

[mathiascode]

@zalloy and @hboetes Can you test if this branch works better (with the OS's default file limits)?

git clone -b openfileslimit https://github.com/mathiascode/nicotine-plus

[hboetes]

I just tried it, really quick, with a mortal account and, it works. :-)
I want to migrate my nicotine to another account anyway, so I'll be testing it a lot more.

[zalloy]

I tried it for a bit. The issue described in this bug report seems to be gone now. However, I noticed a couple of other issues:

1. The "Download folder(s)" option no longer works. You either have to use "Download folder(s) to" and specify a directory, or select the individual files and specify a directory to download them to.

2. After a bit, downloads seem to "die" or just stop working. Files show as queued, but trying to get the place in the queue for those files does nothing, and retrying also does nothing. The only way to get those downloads to retry is to restart Nicotine+.

[mathiascode]

1. Any error messages? Which folder from which user did you attempt to download?
2. Any error messages? Any examples of files that fail?

[zalloy]

@mathiascode No error messages were displayed for either of those issues. I neglected to note the user/folder where I experienced the first issue. Going to continue testing to see if I can reproduce it.

[zalloy]

Here's a weird issue: I just tried to download @@pjogb\-iTunes Sorted\Soft Cell - Non-Stop Erotic Cabaret from user ch3shyre. I tried "Download folder(s)" and at first it didn't work, so I tried it again, and ended up with both that directory from that user, as well as the same directory from another user Bread69!. Haven't yet been able to reproduce this, so it may be a glitch of some sort.

Also, since performing additional searches, my downloads have stopped again. I have downloads queued from users: -mtm-, Kingja, Bread69!, and ch3shyre.

I turned on the debug view to see if there were any errors not being surfaced in the default view, and I'm seeing a lot of messages like User dimkab does not respond to connect request, giving up. I'm seeing those messages both for users I have downloads queued from as well as users I don't have downloads queued from (could be attempting to connect to retrieve search results).

[mathiascode]

I can't find any obvious flaws in the code. Do you think you could try some older commits to see when it could've started happening?

[mathiascode]

I can sort-of reproduce the issue if I download some files and start spamming searches, but the downloads seem to resume once the connections have calmed down a bit.

[mathiascode]

There's some weird issue somewhere, I briefly got strange peer type messages and messages about unknown handlers.

[zalloy]

@mathiascode Yeah, I saw a bunch of that as I was testing yesterday. I plan to test further, and try some earlier commits to see if we can narrow down where the issue is. I should have time to get back to that tomorrow morning. Busy with the fam today, with the holiday.

[hboetes]

I can't help with debugging right now. The ISP of my VM is having a hard time.

[mathiascode]

The peer message warnings seem to be harmless, and are similar to this one: #323 Not sure why some clients seem to be sending invalid messages, but they seem quite rare.

I've hidden these warnings/debug messages under the "Messages" debug toggle in #365. That PR also fixes the "no handler for class" error messages.

[mathiascode]

@zalloy Can you try the latest master? I've made a lot of code changes and optimizations in the last few days, perhaps your issues were solved too. If you use the PPA, the builds will be ready in about an hour.

[zalloy]

Ok, I'll wait for the PPA to hit and then give it a try. Thanks!

[hboetes]

Right, back again.

I'm running the latest master and I get uploads in the download window and I can't remove them. How can I fix that?

[mathiascode]

Don't know, will have to investigate. :(

[mathiascode]

Does it keep happening if you remove the transfers.pickle file once? I'm pretty sure my recent changes to the transfer list may have caused incompatibilities for old transfers.pickle files (these will be reset when upgrading from 1.4.1 to 2.0.0 in the future, anyway). (edit: should not be the case)

[hboetes]

Right, now all downloads are gone, also the unfinished ones. Let's see if new uploads pop up in the downloads.

[mathiascode]

Nevermind, there's a bug where the lists mix when restoring them on startup. I'll look into it.

[hboetes]

Thanks! Awesome!

[mathiascode]

@hboetes This branch should fix the issue. Can you test it?

git clone -b transfercorrectlist https://github.com/mathiascode/nicotine-plus

[hboetes]

On my way.

[mathiascode]

I just pushed another fix to the branch.

[mathiascode]

The branch is now rebased (hopefully the last change for now, so you can test in peace :P).

[hboetes]

No worries, I'm migrating nicotine to the nicotine account, so I won't have to run it with my own user account. Yes, I'm paranoid. This will take a bit.

[hboetes]

Hmmm a user who shared 44 files of rubbish tried to download something, so I banned him. The files he tried to download I can't remove.

[mathiascode]

Thank you, there seems to be a special code case when banning users that I forgot to update.

[mathiascode]

Autoclearing of uploads and downloads also broke, I'll fix that too.

[mathiascode]

Alright, latest fixes are now pushed to the transfercorrectlist branch. This should (hopefully) fix all regressions introduced when some of the transferlist GUI code was rewritten/optimized

[hboetes]

Yes, much better. Thanks. I just tried banning and the users downloads disappeared as they should.

[mathiascode]

Nice! I pushed some more changes into the branch, fixing the direct upload to user feature in your shares list, which has been broken for very long (1.4.1).

[mathiascode]

transfercorrectlist branch changes are now in master

[zalloy]

@mathiascode Are the list issues fixed in the current PPA (nicotine 1.4.3-202007070233~ubuntu20.04.1)? Asking because for some reason I'm still getting uploads showing up in my downloads, and vice-versa.

For example, I have user clownpatrol6666 showing in my uploads with a song from a Cake album, which was something I was trying to download from them, and I also have user Tutti_Quentin showing in my downloads list, but I never selected this download, it's actually an album from my collection they're trying to get from me (they're also showing up in the uploads list with a single song from the same album).

See downloads screenshot and uploads screenshot

[mathiascode]

I'll rebuild the PPA just in case, this should not be happening anymore. Can you try running directly from the master branch, just to check if the issue still occurs there?

[mathiascode]

I checked the latest PPA build, it's outdated somehow. Hopefully the next build will be correct.

[mathiascode]

nicotine_1.4.3-202007071442~ubuntu20.04.1 includes the latest changes

[zalloy]

Thanks! Will test it out in a bit.

[zalloy]

Seems to be working great on my end. No more list issues, and downloads are smooth sailing.

[mathiascode]

Very nice, thanks for testing!