Huge RAM usage when loading ROM from 7zip archive

[nfp0]

Description

I have a big 7zip archive (6GB) with ROMs and when I try to extract a single one through the 7zip application it extracts in 3-5 seconds, but when I open the ROM through RetroArch to play it, it takes about 30 seconds and RetroArch takes up 8GB of RAM until I close it. This RAM is never released.
This happens with multiple cores and multiple archives tested.
This is happening with small 16MB ROMs on an SSD.

Expected behavior

File gets extracted in the same 3-5 seconds as 7zip does and without taking 8GB of RAM.

Actual behavior

Steps to reproduce the bug

1. Have a big 7zip archive with ROMs;
2. Load the ROM with an appropriate core;
3. Observe the long time (20-30 seconds) required to start the game and the memory being gobbled up and never released.

Version/Commit

• RetroArch: 1.8.1 stable

Environment information

• OS: Windows 10 64-bit build 1909

[RobLoach]

Two notes on this...

1. 7zip support is limited. ZIP has the advantage as it provides the CRC hashes without having to red the whole archive.
2. Rather than having one large 7zip archive, I'd recommend having 1 archive for 1 rom.

[Sanaki]

One large 7z archive is theoretically fine, but the issue here isn't the 7z archive, it's that the archive in question is a solid archive. Essentially the whole thing has to be decompressed to get one file from it. You can make 7z in non-solid mode, but it'll be equivalent to having every file inside as its own 7z archive in terms of space consumed. Zip doesn't support solid archives, so it doesn't suffer the same issue or have the same space saving capability.

[nfp0]

@RobLoach @Sanaki The exact same ROM is being extracted from the same 7zip solid archive in 3 seconds with minimal RAM usage using the 7zip application.
I believe it's technically possible for RetroArch to do the same, specially if it used the same libraries as 7zip. That's why I opened the issue.
The alternative solutions are welcome, but do not solve this problem, as there are big advantages in using solid archives with ROM sets.

[nrdxp]

One large 7z archive is theoretically fine

This doesn't seem to be true in the current implementation. I have a 4.3 Gb archive and loading any one game seems to required decompressing the entire archive each time I try to open it, without any sort of caching. As far as I could tell, 7z the binary always extracts the entire archive even when specifying a single file to extract, so we likely would need a different archive format.

[nfp0]

As far as I could tell, 7z the binary always extracts the entire archive even when specifying a single file to extract, so we likely would need a different archive format.

@nrdxp I don't believe this to be true. Please see my previous post here: #9802 (comment)

[nrdxp]

maybe the way 7zip unpacks a single file is just inefficient, but I just tested it yesterday. If I open a game from an archive containing my entire collection, it takes more than a minute for the game to start. If I close and immediately reload the same game it takes another minute to open, but if I load the same game from an archive containing just that one game it starts almost instantly, this was with 1.12.0

[nfp0]

maybe the way 7zip unpacks a single file is just inefficient

@nrdxp That is in contradiction with my findings. 7zip extracts a 16MB rom from a 6GB solid archive in 5 seconds. That is very efficient.
RetroArch clearly is not using the same procedure to extract the ROM. It would be nice if it did, and that's the purpose of this issue.
(This was back in 2019. I haven't re-tested it recently.)

[nrdxp]

what command do you use to extract a single file from a 7z archive (if you recall). I couln't seem to figure it out, or it took long enough that I canceled the operation.

[nfp0]

@nrdxp At the time I was using the Windows GUI app and just dragging the file out of the archive.
I've tried reproducing the same scenario I had back then, but now using the 7z Linux CLI, with the 7z e command, but I can't get the same extracting speed I had back then, or maybe the files I used back then were not solid after all.