Lazy file chunks store

[oleiba]

Hi, I am currently using WebTorrent for seeding a large amount of files in server-side (node.js).
One of the problems I've noticed when trying to scale-up WebTorrent is the amount of opened files.
When each torrent is created - it eventually reaches Torrent._onMetadata where it creates an instance of FSChunkStore (npm module fs-chunk-store).
This opens a file for tracking and validating pieces downloaded.
If one assumption is that most torrents will have very little and rare activity, files could be loaded on demand, rather than up-front.
I want to suggest that the file's .open will occur lazily only when a peer is interested, or alternatively closing the file and re-open it upon request.
I know this is an important feature in libtorrent's codebase, I believe this would allow WebTorrent to scale above limitations it currently has.

[feross]

@oleiba Thanks for opening this issue. It's important that we focus more on performance going forward, so I care a lot of making WebTorrent perform better for you.

Can you share more details about the kind of stress you're putting on WebTorrent.

• Which version of webtorrent are you using? Please ensure it's the latest (0.73.2).
• How many torrents are you seeding?
• How many files are in each torrent, on average?
• How large is each torrent?
• What kind of performance issues are you seeing? Error message? Slowness to add?

Please provide as much detail as you can!

[ScottBrooks]

I can add another use case to this which is almost the opposite of oleib.

From the torrent client perspective, I have a torrent with 8260 files inside of it that I'm trying to load with webtorrent.

I'm using webtorrent inside of electron, so as soon as I load the torrent in webtorrent, all the file descriptors get used up for that process, and electron stops being able to open files which makes it fail in a quite spectacular way.

To answer your list of questions

1. Which version of webtorrent are you using? 0.74.0
2. How many torrents are you seeding? 1
3. How many files are in each torrent, on average? 8,000+
4. How large is each torrent? 100k
5. What kind of performance issues are you seeing? Error message? Slowness to add? Electron stops being able to open files, load assets in the application, etc.

I'm currently using a native module that wraps libtorrent, but being able to drop the dependency on native modules would be amazing.