Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Peer messages causing socket error #346

Closed
zalloy opened this issue Jul 2, 2020 · 48 comments
Closed

Peer messages causing socket error #346

zalloy opened this issue Jul 2, 2020 · 48 comments
Labels

Comments

@zalloy
Copy link

zalloy commented Jul 2, 2020

Nicotine+ version: 1.4.3 unstable
Operating System/Distribution: Kubuntu 20.04 (Focal)

Description: After doing a few searches, I get a lot of peer messages showing up in the log window, followed by "Major Socket Error: Networking terminated! [Errno 24] Too many open files". The peer messages are a variety of types (1028, 3333, 1541, 260, 1797, 14592, 10752), and are coming from a number of different users.

Expected Behavior: I don't think I should be seeing these peer messages popping up, and shouldn't be getting the socket error.

Steps to reproduce:

  1. Open Nicotine+ and connect to the network.
  2. Search for a few different artists (this time, I searched for Led Zeppelin, Heart, and U2).
  3. After a few minutes, you should see the messages in the log window.

Additional context:
Screenshot

@zalloy zalloy added the bug label Jul 2, 2020
@mathiascode
Copy link
Member

Has this happened before, or approximately when did it start happening?

@hboetes
Copy link
Member

hboetes commented Jul 3, 2020

I had this too, turned out I had to increase the number of open files allowed. This can be done with ulimit and/or whatever mechanism used in your OS.

@mathiascode
Copy link
Member

Might have to tweak the values a bit again, it shouldn't have to be necessary to manually allow more open files.

@hboetes
Copy link
Member

hboetes commented Jul 3, 2020

On OpenBSD it looks like this in /etc/login.conf`

        :openfiles-max=1024:\
        :openfiles-cur=512:\

So without user intervention 512 would be the limit.

@mathiascode
Copy link
Member

mathiascode commented Jul 3, 2020

What is the output of ulimit -H -a (hard limits) and ulimit -S -a (soft limits)?

@hboetes
Copy link
Member

hboetes commented Jul 3, 2020

I take it you want the default values for mortal users. Not the output for me.

$ uname -a
OpenBSD dahud 6.7 GENERIC.MP#2 amd64
dahud$ ulimit -H -a
time(cpu-seconds)    unlimited
file(blocks)         unlimited
coredump(blocks)     unlimited
data(kbytes)         786432
stack(kbytes)        32768
lockedmem(kbytes)    4038752
memory(kbytes)       4038752
nofiles(descriptors) 1024
processes            256
$ ulimit -S -a 
time(cpu-seconds)    unlimited
file(blocks)         unlimited
coredump(blocks)     unlimited
data(kbytes)         786432
stack(kbytes)        4096
lockedmem(kbytes)    1346250
memory(kbytes)       4035896
nofiles(descriptors) 512
processes            128

@mathiascode
Copy link
Member

Damn, that's an awfully low hard limit. Oh well, perhaps a 512 Nicotine+ limit on these kind of systems will do to prevent complaints about too many open files.

@hboetes
Copy link
Member

hboetes commented Jul 3, 2020

Yes, typical OpenBSD stuff. Perhaps you could make nicotine+ read the limit and use what's available, with an upper limit or something like that? Post a warning to the logs when it's ridiculously low?

@zalloy
Copy link
Author

zalloy commented Jul 3, 2020

For me, the output of ulimit -H -a is:

-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             unlimited
-c: core file size (blocks)         unlimited
-m: resident set size (kbytes)      unlimited
-u: processes                       127868
-n: file descriptors                1048576
-l: locked-in-memory size (kbytes)  65536
-v: address space (kbytes)          unlimited
-x: file locks                      unlimited
-i: pending signals                 127868
-q: bytes in POSIX msg queues       819200
-e: max nice                        0
-r: max rt priority                 0
-N 15:                              unlimited

The output of ulimit -S -a is:

-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8192
-c: core file size (blocks)         0
-m: resident set size (kbytes)      unlimited
-u: processes                       127868
-n: file descriptors                1024
-l: locked-in-memory size (kbytes)  65536
-v: address space (kbytes)          unlimited
-x: file locks                      unlimited
-i: pending signals                 127868
-q: bytes in POSIX msg queues       819200
-e: max nice                        0
-r: max rt priority                 0
-N 15:                              unlimited

@mathiascode
Copy link
Member

@zalloy and @hboetes Can you test if this branch works better (with the OS's default file limits)?

git clone -b openfileslimit https://github.com/mathiascode/nicotine-plus

@hboetes
Copy link
Member

hboetes commented Jul 3, 2020

I just tried it, really quick, with a mortal account and, it works. :-)
I want to migrate my nicotine to another account anyway, so I'll be testing it a lot more.

@zalloy
Copy link
Author

zalloy commented Jul 3, 2020

I tried it for a bit. The issue described in this bug report seems to be gone now. However, I noticed a couple of other issues:

  1. The "Download folder(s)" option no longer works. You either have to use "Download folder(s) to" and specify a directory, or select the individual files and specify a directory to download them to.

  2. After a bit, downloads seem to "die" or just stop working. Files show as queued, but trying to get the place in the queue for those files does nothing, and retrying also does nothing. The only way to get those downloads to retry is to restart Nicotine+.

@mathiascode
Copy link
Member

  1. Any error messages? Which folder from which user did you attempt to download?
  2. Any error messages? Any examples of files that fail?

@zalloy
Copy link
Author

zalloy commented Jul 3, 2020

@mathiascode No error messages were displayed for either of those issues. I neglected to note the user/folder where I experienced the first issue. Going to continue testing to see if I can reproduce it.

@zalloy
Copy link
Author

zalloy commented Jul 3, 2020

Here's a weird issue: I just tried to download @@pjogb\-iTunes Sorted\Soft Cell - Non-Stop Erotic Cabaret from user ch3shyre. I tried "Download folder(s)" and at first it didn't work, so I tried it again, and ended up with both that directory from that user, as well as the same directory from another user Bread69!. Haven't yet been able to reproduce this, so it may be a glitch of some sort.

Also, since performing additional searches, my downloads have stopped again. I have downloads queued from users: -mtm-, Kingja, Bread69!, and ch3shyre.

I turned on the debug view to see if there were any errors not being surfaced in the default view, and I'm seeing a lot of messages like User dimkab does not respond to connect request, giving up. I'm seeing those messages both for users I have downloads queued from as well as users I don't have downloads queued from (could be attempting to connect to retrieve search results).

@mathiascode
Copy link
Member

I can't find any obvious flaws in the code. Do you think you could try some older commits to see when it could've started happening?

@mathiascode
Copy link
Member

mathiascode commented Jul 4, 2020

I can sort-of reproduce the issue if I download some files and start spamming searches, but the downloads seem to resume once the connections have calmed down a bit.

@mathiascode
Copy link
Member

There's some weird issue somewhere, I briefly got strange peer type messages and messages about unknown handlers.

@zalloy
Copy link
Author

zalloy commented Jul 4, 2020

@mathiascode Yeah, I saw a bunch of that as I was testing yesterday. I plan to test further, and try some earlier commits to see if we can narrow down where the issue is. I should have time to get back to that tomorrow morning. Busy with the fam today, with the holiday.

@hboetes
Copy link
Member

hboetes commented Jul 4, 2020

I can't help with debugging right now. The ISP of my VM is having a hard time.

@mathiascode
Copy link
Member

mathiascode commented Jul 6, 2020

The peer message warnings seem to be harmless, and are similar to this one: #323 Not sure why some clients seem to be sending invalid messages, but they seem quite rare.

I've hidden these warnings/debug messages under the "Messages" debug toggle in #365. That PR also fixes the "no handler for class" error messages.

@mathiascode
Copy link
Member

@zalloy Can you try the latest master? I've made a lot of code changes and optimizations in the last few days, perhaps your issues were solved too. If you use the PPA, the builds will be ready in about an hour.

@zalloy
Copy link
Author

zalloy commented Jul 6, 2020

Ok, I'll wait for the PPA to hit and then give it a try. Thanks!

@hboetes
Copy link
Member

hboetes commented Jul 6, 2020

Right, back again.

I'm running the latest master and I get uploads in the download window and I can't remove them. How can I fix that?

@mathiascode
Copy link
Member

Don't know, will have to investigate. :(

@mathiascode
Copy link
Member

mathiascode commented Jul 6, 2020

Does it keep happening if you remove the transfers.pickle file once? I'm pretty sure my recent changes to the transfer list may have caused incompatibilities for old transfers.pickle files (these will be reset when upgrading from 1.4.1 to 2.0.0 in the future, anyway). (edit: should not be the case)

@hboetes
Copy link
Member

hboetes commented Jul 6, 2020

Right, now all downloads are gone, also the unfinished ones. Let's see if new uploads pop up in the downloads.

@mathiascode
Copy link
Member

Nevermind, there's a bug where the lists mix when restoring them on startup. I'll look into it.

@hboetes
Copy link
Member

hboetes commented Jul 6, 2020

Thanks! Awesome!

@mathiascode
Copy link
Member

@hboetes This branch should fix the issue. Can you test it?

git clone -b transfercorrectlist https://github.com/mathiascode/nicotine-plus

@hboetes
Copy link
Member

hboetes commented Jul 6, 2020

On my way.

@mathiascode
Copy link
Member

I just pushed another fix to the branch.

@mathiascode
Copy link
Member

The branch is now rebased (hopefully the last change for now, so you can test in peace :P).

@hboetes
Copy link
Member

hboetes commented Jul 6, 2020

No worries, I'm migrating nicotine to the nicotine account, so I won't have to run it with my own user account. Yes, I'm paranoid. This will take a bit.

@hboetes
Copy link
Member

hboetes commented Jul 6, 2020

Hmmm a user who shared 44 files of rubbish tried to download something, so I banned him. The files he tried to download I can't remove.

@mathiascode
Copy link
Member

Thank you, there seems to be a special code case when banning users that I forgot to update.

@mathiascode
Copy link
Member

Autoclearing of uploads and downloads also broke, I'll fix that too.

@mathiascode
Copy link
Member

Alright, latest fixes are now pushed to the transfercorrectlist branch. This should (hopefully) fix all regressions introduced when some of the transferlist GUI code was rewritten/optimized

@hboetes
Copy link
Member

hboetes commented Jul 7, 2020

Yes, much better. Thanks. I just tried banning and the users downloads disappeared as they should.

@mathiascode
Copy link
Member

Nice! I pushed some more changes into the branch, fixing the direct upload to user feature in your shares list, which has been broken for very long (1.4.1).

@mathiascode
Copy link
Member

transfercorrectlist branch changes are now in master

@zalloy
Copy link
Author

zalloy commented Jul 7, 2020

@mathiascode Are the list issues fixed in the current PPA (nicotine 1.4.3-202007070233~ubuntu20.04.1)? Asking because for some reason I'm still getting uploads showing up in my downloads, and vice-versa.

For example, I have user clownpatrol6666 showing in my uploads with a song from a Cake album, which was something I was trying to download from them, and I also have user Tutti_Quentin showing in my downloads list, but I never selected this download, it's actually an album from my collection they're trying to get from me (they're also showing up in the uploads list with a single song from the same album).

See downloads screenshot and uploads screenshot

@mathiascode
Copy link
Member

I'll rebuild the PPA just in case, this should not be happening anymore. Can you try running directly from the master branch, just to check if the issue still occurs there?

@mathiascode
Copy link
Member

I checked the latest PPA build, it's outdated somehow. Hopefully the next build will be correct.

@mathiascode
Copy link
Member

nicotine_1.4.3-202007071442~ubuntu20.04.1 includes the latest changes

@zalloy
Copy link
Author

zalloy commented Jul 7, 2020

Thanks! Will test it out in a bit.

@zalloy
Copy link
Author

zalloy commented Jul 7, 2020

Seems to be working great on my end. No more list issues, and downloads are smooth sailing.

@mathiascode
Copy link
Member

Very nice, thanks for testing!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Development

No branches or pull requests

3 participants