OSX has a limit of 256 open files

[avahowell]

This has caused crashing issues for Mac users

[DavidVorick]

Is this a problem with siad or just the UI?

[lukechampine]

It has implications for our API design. If the design encourages opening many calls open concurrently (e.g. when downloading many files), then it's a failing of our API design, not of the UI.

[DavidVorick]

Ok. I will probably not leave this issue open for very long, because there's no action item associated with it. However we should definitely support non-blocking download calls in the api.

[lukechampine]

Agreed. That feature should dovetail nicely with our merged upload/download loop.

[daniel-lucio]

Same issue in Linux, siad rises more than 1024 file handlers

[DavidVorick]

@daniel-lucio how long were you running Sia? Did you perform a specific action to hit 1024?

Trying to figure out the best way to keep this from happening in the future.

@lukechampine would the gateway have one open per peer?

[lukechampine]

Yes, I believe the current file handler sources are:

• 1 for the API listener
• 1 for each peer connection
• 1 for each database
• 1 for each host/renter connection
• 1 for each file being uploaded/downloaded (renter side)
• 1 (?) for each file being uploaded/downloaded (host side)

However, we don't need to worry about the limit when it comes to peer connections or host operations. In the rare case that you want a huge number of peers, you can manually raise the limit. And for hosts, it's expected that they will be more advanced users, so it's sufficient to document that hosting a large amount of data may require raising the limit. (Of course, minimizing the number of open file handles is still a good idea!)

[DavidVorick]

I'm struggling to see how that would add up to 1024, though I could see it getting pretty close to 256 if you have the max number of peers. Perhaps we should drop the default maximum to 40.

[daniel-lucio]

@DavidVorick you need to know your SO. In Linux, I use the ulimit command to raise this limit. I ignore if OSX has that as well.

[avahowell]

I could see it adding up to 1024 if you include all the other file descriptors in use on the system, not just Sia.

[DavidVorick]

I had the impression that it was 256 files per process, not per machine?

[RNabel]

Same problem on Fedora 24 with a soft limit of 1024 and hard limit of 4096 open files.

siad continues to work though.

[RNabel]

Just investigated this a little closer, and it seems like siad is opening a ton of new sockets.

Reproduce

1. Start unlocking of wallet
2. find process id of siad (ps aux | grep siad or otherwise)
3. run ls -l /proc/<siad id>/fd | grep socket | wc -l

The count is about 150 to start with and then increases steadily by about 100 every few minutes. At block height 40k there are about 600 open sockets.
After the wallet is unlocked the open socket count drops to 15.

[lukechampine]

That is very interesting. I don't think the wallet itself is opening sockets, but once the wallet is unlocked, the renter can start trying to form/renew contracts, and each of those attempts requires a socket. Perhaps we are leaking those somehow.

[DavidVorick]

Seems more likely to me that bolt is not cleaning up the file handles very well, I'm guessing it opens a separate socket for each call to db.Read and then eventually cleans them all up.

[lukechampine]

AFAIK bolt doesn't use sockets, just mmap. We could (sort of) confirm this by examining socket use after deleting something like hostdb.json and restarting. If bolt is responsible, then resyncing any module should cause a bunch of sockets to be opened.

@RNabel, any chance you can use a tool like ss to find out more about the socket connections? Ideally we'd be able to capture some of the data sent. That would tell us what code is responsible for opening the sockets.

[RNabel]

@lukechampine Sure thing. I'll play around with this over the weekend and report back.

[RNabel]

Alright, I ran through the above steps again (this time with said v1.1.2) and could not replicate this behavior anymore - could it be fixed by the latest update?

[DavidVorick]

Were you using v1.1.0 before? It's possible that most of those sockets were created by the HostDB, which got a pretty big upgrade in v1.1.1.

I'm going to leave this open because I'm not convinced that the problem has been resolved, but if nobody can replicate it as of the most recent master branch, I guess we did something to fix it accidentally.

We have been in general working on improving our resource management.

[RNabel]

I'm pretty sure that I was using v1.1.1. In any case, tried again and can't reproduce the error.

[lukechampine]

Closing for now, feel free to reopen if this issue presents itself again.