[Bug] DLC/update folders have no limit on open files

[shny-scott]

Describe the bug
0.2.0-rc1 adds a feature to allow loading updates and DLC from a folder without installing them. Having too many files will result in the emulator failing to load anything else with a "Too many open files" error, causing the game list to appear empty and a "corrupt profile" warning to pop up (the actual profile is unaffected). These issues disappear as soon as the offending directory is removed, often without even needing a restart.

Branch/PR/Release:
0.2.0-rc1, MinGW PGO

To Reproduce
Steps to reproduce the behaviour:

1. Add folder(s?) with many, many update/DLC files and click OK
2. See files failing to load, but base game list is still intact
3. Restart emulator, base game list blanks, system software is not detected, profiles do not load

Expected behaviour
Eden loads them normally, as it does when there are fewer files

Device:

• OS: Windows 11 25H2
• CPU: Ryzen 5 3600
• GPU: RTX 3060 Ti
• RAM: 32gb

Additional context
I have a sizable update collection as well as DLC for various games. I do have a handful of old updates and DLC for games I no longer play on PC, which probably aren't helping. Games with lots of DLC like Smash Ultimate, Groove Coaster Wai Wai Party, and Taiko no Tatsujin Drum n Fun contribute to this the most. I don't know exactly what the file limit is, but I'm sure it's hardcoded somewhere. I do not have issues if I move out subfolders for particularly large games (resulting in 51 updates/185 DLC), but I do with those subfolders in place (51 updates/505 DLC).

Screenshots and Logging
The log is a little under 3000 lines of the exact same error for various files:

[   3.170121] Common.Filesystem <Error> common/fs/file.cpp:286:Open: Failed to open the file at path=D:\Emulators\yuzu\games\Jump Rope Challenge v0 (0100B9C012706000) (BASE).nsp, ec_message=Too many open files
[   3.170131] Common.Filesystem <Error> common/fs/fs.cpp:154:FileOpen: Failed to open the file at path=D:\Emulators\yuzu\games\Jump Rope Challenge v0 (0100B9C012706000) (BASE).nsp with mode=1, type=0, flag=1
[   3.170336] Common.Filesystem <Error> common/fs/file.cpp:286:Open: Failed to open the file at path=D:\Emulators\yuzu\games\Jump Rope Challenge v0 (0100B9C012706000) (BASE).nsp, ec_message=Too many open files
[   3.170348] Common.Filesystem <Error> common/fs/fs.cpp:154:FileOpen: Failed to open the file at path=D:\Emulators\yuzu\games\Jump Rope Challenge v0 (0100B9C012706000) (BASE).nsp with mode=1, type=0, flag=1
[   3.170551] Common.Filesystem <Error> common/fs/file.cpp:286:Open: Failed to open the file at path=D:\Emulators\yuzu\games\Jump Rope Challenge v0 (0100B9C012706000) (BASE).nsp, ec_message=Too many open files
[   3.170561] Common.Filesystem <Error> common/fs/fs.cpp:154:FileOpen: Failed to open the file at path=D:\Emulators\yuzu\games\Jump Rope Challenge v0 (0100B9C012706000) (BASE).nsp with mode=1, type=0, flag=1
[   3.170762] Common.Filesystem <Error> common/fs/file.cpp:286:Open: Failed to open the file at path=D:\Emulators\yuzu\games\Jump Rope Challenge v0 (0100B9C012706000) (BASE).nsp, ec_message=Too many open files
[   3.170772] Common.Filesystem <Error> common/fs/fs.cpp:154:FileOpen: Failed to open the file at path=D:\Emulators\yuzu\games\Jump Rope Challenge v0 (0100B9C012706000) (BASE).nsp with mode=1, type=0, flag=1
[   3.170981] Common.Filesystem <Error> common/fs/file.cpp:286:Open: Failed to open the file at path=D:\Emulators\yuzu\games\Jump Rope Challenge v0 (0100B9C012706000) (BASE).nsp, ec_message=Too many open files
[   3.171020] Common.Filesystem <Error> common/fs/fs.cpp:154:FileOpen: Failed to open the file at path=D:\Emulators\yuzu\games\Jump Rope Challenge v0 (0100B9C012706000) (BASE).nsp with mode=1, type=0, flag=1
[   3.171371] Common.Filesystem <Error> common/fs/file.cpp:286:Open: Failed to open the file at path=D:\Emulators\yuzu\games\Jump Rope Challenge v0 (0100B9C012706000) (BASE).nsp, ec_message=Too many open files
[   3.171382] Common.Filesystem <Error> common/fs/fs.cpp:154:FileOpen: Failed to open the file at path=D:\Emulators\yuzu\games\Jump Rope Challenge v0 (0100B9C012706000) (BASE).nsp with mode=1, type=0, flag=1
[   3.171786] Common.Filesystem <Error> common/fs/file.cpp:286:Open: Failed to open the file at path=D:\Emulators\yuzu\games\Jump Rope Challenge v0 (0100B9C012706000) (BASE).nsp, ec_message=Too many open files
[   3.171839] Common.Filesystem <Error> common/fs/fs.cpp:154:FileOpen: Failed to open the file at path=D:\Emulators\yuzu\games\Jump Rope Challenge v0 (0100B9C012706000) (BASE).nsp with mode=1, type=0, flag=1
[   3.172045] Common.Filesystem <Error> common/fs/file.cpp:286:Open: Failed to open the file at path=D:\Emulators\yuzu\games\Jump Rope Challenge v0 (0100B9C012706000) (BASE).nsp, ec_message=Too many open files
[   3.172089] Common.Filesystem <Error> common/fs/fs.cpp:154:FileOpen: Failed to open the file at path=D:\Emulators\yuzu\games\Jump Rope Challenge v0 (0100B9C012706000) (BASE).nsp with mode=1, type=0, flag=1
[   3.172368] Loader <Warning> core/loader/loader.cpp:262:GetLoader: File Jump Rope Challenge v0 (0100B9C012706000) (BASE).nsp has a different type (unknown) than its extension.```

[toratora]

Same here. I have 518 roms, but don't have any DLC or update directories set and I get the Too many open files error once it gets to the S games.

[   1.031366] Common.Filesystem <Error> common/fs/file.cpp:286:Open: Failed to open the file at path=R:\Console ROMs\Switch\Games\Romancing.SaGa.2.Revenge.of.the.Seven\Romancing SaGa 2 Revenge of the Seven [010045301B86C000].xci, ec_message=Too many open files
[   1.031371] Common.Filesystem <Error> common/fs/fs.cpp:154:FileOpen: Failed to open the file at path=R:\Console ROMs\Switch\Games\Romancing.SaGa.2.Revenge.of.the.Seven\Romancing SaGa 2 Revenge of the Seven [010045301B86C000].xci with mode=1, type=0, flag=1

[luis128-cpu]

I get the same error. I think the limit is more or less 500 files in total. If I put 427 ROMs and 99 DLCs, it reduces the find ROMs to 413 and I can't load any game till I delete the path to those DLCs.

[macs5555]

I'm having the same problem. Game Listing stops at around 500 files. Log shows ec_message=Too many open files. It works fine with MSVC builds, but occurs with MinGW and MinGW PGO builds.

[grayhaze]

Same here. My log file is full of Too many open files for every file, settings aren't saving and profiles are getting corrupted on each load. Given this has been a problem since March, and we just hit another full release, it doesn't look like they intend on fixing it. I guess I'm ditching Eden and moving to Ryujinx.

Apologies for the inappropriate last sentence, which I wrote out of frustration.

[crueter]

Submit patches and fix it yourself instead of incessantly posting this on every single subreddit and Discord server you come across. And especially stop harassing developers in their direct messages begging for them to fix a complete non-issue.

[FelesNoctis]

@crueter the person who opened this issue is not the same as the person/people spamming the team. This isn't a non-issue, it appears to be affecting quite a few users.

[FelesNoctis]

I get the same error. I think the limit is more or less 500 files in total. If I put 427 ROMs and 99 DLCs, it reduces the find ROMs to 413 and I can't load any game till I delete the path to those DLCs.

413 + 99 = 512 open file handles, a power of two.

[FelesNoctis]

https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/setmaxstdio?view=msvc-170

C run-time I/O now supports up to 8,192 files open simultaneously at the low I/O level. This level includes files opened and accessed using the _open, _read, and _write family of I/O functions. By default, up to 512 files can be open simultaneously at the stream I/O level. This level includes files opened and accessed using the fopen, fgetc, and fputc family of functions. The limit of 512 open files at the stream I/O level can be increased to a maximum of 8,192 by use of the _setmaxstdio function.

This upper limit might be beyond what's supported by a particular Win32 platform and configuration.

I wouldn't know how to apply this to the codebase myself, but I've seen it used elsewhere with unrelated programs (game modding, etc). Are we concerned about Win32 at all at this point?

[crueter]

I still don't believe this is a real issue, but https://git.eden-emu.dev/eden-emu/eden/pulls/4011 should fix it.

[FelesNoctis]

For those wondering, we actually had already set the open file I/O handles to 8192 a couple months back, but there was a second check somewhere else that was added a long time ago for unknown reasons, and we're not about to go digging through history to find out exactly why. The above fix should solve the issue.

As I've said in other public places, but will repeat here:
Harassment is not an appropriate way to get our attention. Unfortunately, that's what happened here, and if not for the fact that this was a simple flaw to correct, we may have left it to rot as a low-priority issue that affects very few users. We're going forward with zero tolerance towards suspected harassment in the future.

[grayhaze]

Thank you for reopening the issue and providing an explanation, @FelesNoctis.

As stated elsewhere, I hit this problem at only having 80 games in my library, which includes titles with large amounts of DLC. To give a broad cross-section, such titles include "Taiko no Tatsujin - Drum 'n' Fun" (320 DLC), "Super Smash Bros. Ultimate" (99 DLC), "Pinball FX" (21 DLC), "Talisman" (20 DLC), "Theatrhythm Final Bar" (18 DLC), "Suika Game" (14 DLC), "Unmatched" (11 DLC), "Ticket to Ride" (11 DLC), and "UNO" (10 DLC). Depending upon the games an individual has in their library, it's easy to see how the 512 file limit can be hit in a short space of time.

As for the harassment, I'm sorry you had to deal with that. I realise that there are a large number of open issues on the repo currently, and as such it's not always possible to get eyes on every issue when they're raised. That's no excuse for users attacking the developers though, especially given they do this for free and are under no obligation. I appreciate the work you all do, and I apologise if I also came off as rude in my comments.

Edit: It has been brought to my attention that some developers think I was the one harassing them over private messages. I'm not sure where this came from, but I assure you it wasn't me. If you'd like to discuss this, please feel free to contact me privately.