[Feature Request]: Preload disc image to RAM

[brad86m]

Description

With compression methods like CHD reducing ISO sizes (dramatically in a lot of cases), and the average memory amount for most users being 16GB, an option to preload the entire image to RAM would be very welcome.

I still use large multi-platter mechanical hard drives for game storage, and the occasional read hitching is very common and noticeable during emulation. With DuckStation and similar emulators, there is an option to preload discs, which completely eradicates this issue.

With efficient CHD compression sizes and healthy amounts of RAM, the option to so the same within PCSX2 would save having to drag and drop games over to the SSD for play.

Reason

Smoother frame times due to no HDD read hitching

Examples

.

[stenzek]

I've considered it, but honestly, users being incredibly stupid is the main blocker.

DVDs can still be upwards of 6GB, even with CHD compression, and we don't really want to have to deal with fools complaining that PCSX2 made their 8GB PC lag so hard they had to reset it, because they're following some stupid YouTube guide with "4k 60fps settings".

Plus, the fact that preloading said image would take close to a minute, even on a reasonably fast HDD. That's a long time to wait for a game to boot.

[refractionpcsx2]

Yep, I agree. Maybe in 10-15 years where the average low end user has 16-32gb of ram, it might be feasible, but right now, it's not really possible.

Some ISO's can get up to 8gb (COMPRESSED!) so that's all your RAM gone on a low end computer.

The only way this would be feasible, is if we restricted the feature to people who have 16 or maybe 32gb or more.

[brad86m]

I absolutely get that a lot of users would just complain about their memory, without looking into the implementation and what it actually does.

Still, I and a lot of users would make good use of it, and knowing how and when to use the feature appropriately. Being able to enable and disable in per-game configs and whatnot. I use 32GB in my system, and never really have anything large in the background running when playing games. It has always been a recipe for disaster.

The comment section in the UI can make it clear to users what enabling the feature will do.

Even if you consider a trial run, or even having the setting hidden in advance, would really appreciate it.

For me, it is the benefit of not having to worry about mechanical HDD seeking, rather than negligible load time improvements.

[RedDevilus]

Even with enough RAM how do you check if there isn't a bazillion things running in the background that takes up space, and you are left with let's say 4GB free. But would it even improve loading times after the initial delayed start?

[refractionpcsx2]

The comment section in the UI can make it clear to users what enabling the feature will do.

Users do not read :P You learn this when you work on a big project, they just wildly see an option they want and click it before they've finished reading what's involved.

[RedDevilus]

Amount of times I see people changing random settings, and you ask why either they answer it's a video, someone told them to, or they thought randomly flicking on things helped. Some people disable mipmapping and think it improves speeds, others change it to full because they think it's increased texture space or even renderers where they see Automatic that helps most people not think about it.

Of course you can never stop people doing bad stuff and just crippling usability because some users can't control themselves isn't a good idea either. I'm not against tinkering if you check every option and set it back if there is no visible improvement, it's how I learned to do it.

[brad86m]

Sure, but these people being idiots shouldn't mean that users who aren't so stupid can't enjoy nice things, even if they are pointless to many.

[refractionpcsx2]

I'm just wondering if we can set a threshold, so on load it checks that the system has Size of disc + 2gb free, or something, so it can disable itself on the fly if the user tries to load an image too big for memory.

[stenzek]

I use 32GB in my system, and never really have anything large in the background running when playing games.

And I have 64GB in my systems, but the average person doesn't. A good chunk of the PCSX2 userbase are playing on underpowered systems. We don't collect analytics, because it's a legal/privacy nightmare, but I'd be surprised if the average user had >8GB RAM.

Being able to enable and disable in per-game configs and whatnot.

This won't be possible. We need the disc mounted to get the serial to get the per-game config, and the preload has to happen before the mount. Classic chicken-and-egg problem. I work around that in DuckStation by having the option to transform an already-mounted image to a memory one, PCSX2's design does not allow that currently (CDVD backend is a horrible mess).

Even if you consider a trial run, or even having the setting hidden in advance, would really appreciate it.

That doesn't stop moronic users. If anything, it makes it worse, because they won't think/know to disable it, and they enabled it because of following some stupid guide.

For me, it is the benefit of not having to worry about mechanical HDD seeking, rather than negligible load time improvements.

The load times are much worse like I said. 1+ minute for preloading?

[RedDevilus]

Sure, but these people being idiots shouldn't mean that users who aren't so stupid can't enjoy nice things, even if they are pointless to many.

I'm just not sure what this would even fix in the end. Like with RPCS3 you can precompile stuff, and it wouldn't stutter when you would do it the regular way when it sees new stuff, but that's something PCSX2 didn't have problems with. There are other emulators who seems to benefit them from what I see.

[brad86m]

Tested throwing a 7GB CHD into ramdisk, and it took 9 seconds. That's not bad. I'd be more than happy with that tradeoff.

Will obviously vary greatly between systems.

[RedDevilus]

Tested throwing a 7GB CHD into ramdisk, and it took 9 seconds. That's not bad. I'd be more than happy with that tradeoff.

But what about the in-game effect, not the initial delayed start. You need to compare that as that would be your main goal no?

[refractionpcsx2]

Tested throwing a 7GB CHD into ramdisk, and it took 9 seconds. That's not bad. I'd be more than happy with that tradeoff.

from a mechanical drive? I doubt it, a mechanical drive tops out at (generously) 200mb/sec, so in 9 seconds you'll do 1.8gb. You're looking more at 35 seconds, in that scenario, which is a long time for a lot of people.

[stenzek]

Tested throwing a 7GB CHD into ramdisk, and it took 9 seconds. That's not bad. I'd be more than happy with that tradeoff.

That's not coming from the HDD. That works out to ~800MB/s, which means your OS likely cached the file already. So the ramdisk is 100% redundant.

I'm just wondering if we can set a threshold, so on load it checks that the system has Size of disc + 2gb free, or something, so it can disable itself on the fly if the user tries to load an image too big for memory.

And what happens if they open something RAM-heavy while PCSX2 is still running? Then everything starts thrashing.

[brad86m]

Tested throwing a 7GB CHD into ramdisk, and it took 9 seconds. That's not bad. I'd be more than happy with that tradeoff.

But what about the in-game effect, not the initial delayed start. You need to compare that as that would be your main goal no?

The in-game effect for me is no longer having hitching from HDD reading. On multi-plattered drives, this is unavoidable.

In RAM or on flash storage (SSD) it's not an issue.

[stenzek]

The in-game effect for me is no longer having hitching from HDD reading. On multi-platttered drives, this is unavoidable.

You do know that PS2 DVD seek times are much, much slower than HDDs, right? Fairly certain our reads are asynchronous, i.e. we start them in the background when the game kicks the read. The issue is more drives going to sleep, than the seek times.

[refractionpcsx2]

And what happens if they open something RAM-heavy while PCSX2 is still running? Then everything starts thrashing.

I assume windows/linux/whatever will be clever enough to use a swap file, but we could always be more strict with the threshold, like disc + 4gb free, or even 6gb.

If the user is gonna open loads of shit and run out of ram, they're gonna do it either way lol.

[RedDevilus]

Tested throwing a 7GB CHD into ramdisk, and it took 9 seconds. That's not bad. I'd be more than happy with that tradeoff.

But what about the in-game effect, not the initial delayed start. You need to compare that as that would be your main goal no?

The in-game effect for me is no longer having hitching from HDD reading. On multi-platttered drives, this is unavoidable.

Are you comparing to an external USB attached HDD? That makes sense. But PCSX2 isn't that slow on internal HDD and you bypass that with internal SSDs for potential hiking either way.

[brad86m]

Tested throwing a 7GB CHD into ramdisk, and it took 9 seconds. That's not bad. I'd be more than happy with that tradeoff.

from a mechanical drive? I doubt it, a mechanical drive tops out at (generously) 200mb/sec, so in 9 seconds you'll do 1.8gb. You're looking more at 35 seconds, in that scenario, which is a long time for a lot of people.

Stupidly, no. I copied from my OS SSD, basically making my test pointless lol.
I was rushing.

[brad86m]

Tested throwing a 7GB CHD into ramdisk, and it took 9 seconds. That's not bad. I'd be more than happy with that tradeoff.

But what about the in-game effect, not the initial delayed start. You need to compare that as that would be your main goal no?

The in-game effect for me is no longer having hitching from HDD reading. On multi-platttered drives, this is unavoidable.

Are you comparing to an external USB attached HDD? That makes sense. But PCSX2 isn't that slow on internal HDD and you bypass that with internal SSDs for potential hiking either way.

Attached via SATA on the mainboard. Same issues happen when using as external in USB 3.1.
It's common with 12+ TB drives due to the l;layering of platters.

[RedDevilus]

HDD ATA interface: ATA-66 [66.0 MB/s]
Ethernet interface: 100Mbit [12.5 MB/s]
DVD drive: 4x speed DVD-ROM [5.3 MB/s]
CD Drive: 24x speed CD-ROM [3.6 MB/s]
USB 1.1 interface: 12Mbit [1.5 MB/s]

Also keep in mind these are the speed results of the PS2 components, even a slow HDD is going to be much faster than any disc in mind with seek times and spinning.

And PCSX2 already uses multiple caches to reuse textures and other types so it's more dynamic and not as intensive for older computers.

[brad86m]

HDD ATA interface: ATA-66 [66.0 MB/s] Ethernet interface: 100Mbit [12.5 MB/s] DVD drive: 4x speed DVD-ROM [5.3 MB/s] CD Drive: 24x speed CD-ROM [3.6 MB/s] USB 1.1 interface: 12Mbit [1.5 MB/s]

Also keep in mind these are the speed results of the PS2 components, even a slow HDD is going to be much faster than any disc in mind with seek times and spinning.

I now. It's not the speed of the drive, but the issue with it having to read a different platter. Data is shared over different layers, resulting in slight hitching when swapping. A single platter drive would only have such a problem if the drive was sleeping.

As I said, sadly, it can't be stopped on these larger drives, and that's why I would personally make use of preloading.

[user18081972]

Data is shared over different layers, resulting in slight hitching when swapping.

Sounds like you are saying the seek times of the HDD you are talking about are higher than that of a PS2 CD/DVD, or high enough to make the game stutter anyway.

[brad86m]

Data is shared over different layers, resulting in slight hitching when swapping.

Sounds like you are saying the seek times of the HDD you are talking about are higher than that of a PS2 CD/DVD, or high enough to make the game stutter anyway.

More than fast enough to keep up with reading data. Seek speed is not the issue. Multi platter drives come with a fundamental design flaw (if you can class it as such) where they will pause for an incredibly short period of time when having to continue reading data from a different platter. This is enough to cause a noticeable short stutter in anything that is currently being read. In this case, a PS2 image. A single large file can be spread over multiple platters.

[RedDevilus]

Data is shared over different layers, resulting in slight hitching when swapping.

Sounds like you are saying the seek times of the HDD you are talking about are higher than that of a PS2 CD/DVD, or high enough to make the game stutter anyway.

More than fast enough to keep up with reading data. Seek speed is not the issue. Multi platter drives come with a fundamental design flaw (if you can class it as such) where they will pause for an incredibly short period of time when having to continue reading data from a different platter. This is enough to cause a noticeable short stutter in anything that is currently being read. In this case, a PS2 image. A single large file can be spread over multiple platters.

Well, if you can provide the necessary benchmarks and detail them for several games (and how you measured it so it can be reproduced) then it can be talked about. While there is truth to what you say, personally it shouldn't affect it that negatively without this option. As when you run a game first time is caching per renderer, also hash cache and more. In the past without them it didn't feel that bad, though I'm happy to be proven wrong if there is a real potential improvement for other users.

[brad86m]

Data is shared over different layers, resulting in slight hitching when swapping.

Sounds like you are saying the seek times of the HDD you are talking about are higher than that of a PS2 CD/DVD, or high enough to make the game stutter anyway.

More than fast enough to keep up with reading data. Seek speed is not the issue. Multi platter drives come with a fundamental design flaw (if you can class it as such) where they will pause for an incredibly short period of time when having to continue reading data from a different platter. This is enough to cause a noticeable short stutter in anything that is currently being read. In this case, a PS2 image. A single large file can be spread over multiple platters.

Well, if you can provide the necessary benchmarks and detail them for several games (and how you measured it so it can be reproduced) then it can be talked about. While there is truth to what you say, personally it shouldn't affect it that negatively without this option. As when you run a game first time is caching per renderer, also hash cache and more. In the past without them it didn't feel that bad, though I'm happy to be proven wrong if there is a real potential improvement for other users.

There isn't really anything to prove. It is what it is. Multi platter drives cause hitching when swapping between platters.

There is only one solution, and that is not reading large files from them. As moving over to an SSD gets rather tiresome, the possible implementation of preloading the entire image to memory would have been a nice thing to have for those of us that could make use from it.

[user18081972]

I've been using a Playnite script to cache the game image into RAM for years now. Only downside is that it could be purged from the cache if Windows feels like it, so its a very hacky workaround that doesnt actually work fully reliably.

Also I cant use the nice PCSX2 game library feature, i HAVE to run a game through playnite for it to work.

It might also be useful for people who run games from network storage like a NAS or a WebDAV location

[ichee]

Just to add to this, preloading the disc image can serve the purpose of not having to frequently access the drive, beyond saving and loading saves, which can, slightly, help with reduced power consumption by allowing the drive to idle longer.

As stated above, a bigger, and more likely, benefit would be when disc images are stored on a network drive, which would be much more problematic to stream data from.

Querying if system has 16GB+ before either exposing the feature or entirely ignoring the option if the requirement isn’t met could help prevent “user stupidity”.
The option could also simply be not exposed in UI and only have it in .cfg file instead to help alleviate the user complaint burden.

Granted, most, if not all, of these use cases would simply be for user convenience, even if not at all beneficial but for the most unusual cases.

[brad86m]

Just to add to this, preloading the disc image can serve the purpose of not having to frequently access the drive, beyond saving and loading saves, which can, slightly, help with reduced power consumption by allowing the drive to idle longer.

As stated above, a bigger, and more likely, benefit would be when disc images are stored on a network drive, which would be much more problematic to stream data from.

Querying if system has 16GB+ before either exposing the feature or entirely ignoring the option if the requirement isn’t met could help prevent “user stupidity”. The option could also simply be not exposed in UI and only have it in .cfg file instead to help alleviate the user complaint burden.

Granted, most, if not all, of these use cases would simply be for user convenience, even if not at all beneficial but for the most unusual cases.

Very good points here. We'll see what happens in the future, but I do hope Stenzek comes back to this with the possibility of adding it in at some point.