Fix SRAM not saving to disk [Improvement] [$5]

[jvook]

SRAM does not write to disk unless RA gracefully exits or Autosave SRAM interval is set.

Expected behavior

SRAM should be written to disk when an in-game save occurs

Actual behavior

RA hedges against games that will use SRAM as working memory and does not write to disk until the emulator closes. So if there is a non-graceful exit to RA then the in game save will be lost.

---

There is a $5 open bounty on this issue. Add to the bounty at Bountysource.

[jvook]

Suggested solution

Assumption: SRAM is held in memory during emulator running.

Test to determine whether it is safe to write SRAM to disk:

1. Detect change in SRAM
2. Note timestamp of change
3. If last SRAM change was greater than T seconds ago (e.g., 10 seconds), write SRAM change to disk. Else, disregard SRAM change and do not write to disk.

Effect of this would be that games that are using the SRAM as working memory would fail the test and never write to disk, because a game that is using SRAM as working memory would be writing more than every T seconds (and any edge case that contradicts would not risk ruining disk or performance anyway because it is only manipulating the SRAM every 10 seconds). Otherwise the game would pass the "safe to write" test and will write SRAM to disk immediately. Adjust T to be appropriate amount of seconds (it seems 1 or 2 seconds is probably fine).

In case the game is using the SRAM as working memory, this test should be placed on a cooldown so that it the test is not being run 60 times per second (if the test fails then cooldown for T2 seconds, e.g., 10 seconds).

Remaining edge case I can think of is that a game is manipulating a portion of SRAM but still using a portion as actual SRAM. These games would fail to write to disk, but would still write on graceful exit, and it is a much smaller pool of potential games that results in unexpected data loss as opposed to current implementation.

[inactive123]

@fr500 Wasn't this already fixed as of the most recent nightlies?

[andres-asm]

@twinaphex
Auto save state on exit was fixed

Saving SRAM when RA crashes or whatever is not.

[inactive123]

Saving SRAM when RA crashes is not even possible. When RA crashes, it crashes. I am not going to install a signal handler which could interfere with being able to debug stuff. We have seen how well that works out with Reicast. And buggy cores should just be patched to no longer be buggy, plain and simple. I'd wager 95% of all crashes that can potentially happen in RA is all due to the core.

[jvook]

@twinaphex Issue does not relate to saving sram when RA crashes. This is about writing sram to disk immediately. Proposed solution addresses issues caused by a few games that use sram as working memory. Ask radius about them. The downside of the way RA handles it is that a system like PSX, which never used sram as memory, suffers from losing in game save data. E.g., when RA crashes, when OS crashes, when RA is force closed, on power loss, etc.

As written in op: Expected behavior - SRAM should be written to disk when an in-game save occurs.

Nothing to do with error signaling or whatever.

These saves ought to be written immediately because in game save data should not be lost so easily.

The current solution is to set an obscure option to "Autosave SRAM" every 10 seconds or whatever, which senselessly rewrites the same save data over and over. It's not on by default so you don't learn about it until after you lose your saves. And it doesn't even avoid the issue with the sram-abusing games, merely mitigates it.

[inactive123]

Just set an sram autosave interval if you want that.

We are not going to do constant file I/O writing by default. This is bad design, it causes performance issues because disk I/O is a big bottleneck and cart based consoles dealt woth saving in microseconds or less rather than milliseconds or even seconds. And also, it wears out SSDs. It is just bad design whichever way you look at it.

So just set autosave interval to some specified amount of seconds and call it a day.

[inactive123]

And no, this is not going to be implemented regardless of any bounty. You already have a solution wi the save interval if you want this. But this is not going to be set to some set interval by default.

Take a look at some game like Shiren The Wanderer on N64. Every time you take a single step, it does an SRAM save. If you tried to write to disk every single time you walk in that game, your framerate gets slaughtered and your disk gets worn out.

so, no. Its not only this game - its the fact that non volatile memory backup on consoles cannot be a direct 1/1 mapping equivalent to saving to a harddisk, disk I/O is way too bad and too much of a bottleneck for that to scale. Nevermind with SSDs that have a finite amount of writes.

[jvook]

@twinaphex You clearly haven't read the proposed solution. It avoid all the issue you are suggesting. I'm not here to whine bout it I'm here to improve retroarch. That's why I wrote an actual solution. My solution is superior to the autosave interval in every way, and would use fewer disk writes, if that is your concern. If you want to pretend I'm suggesting constant disk io or error tracking or autosave default or whatever instead of reading 3 paragraphs that I thought hard about, that's your problem.

[jvook]

@twinaphex I will repost it since you seem to miss the suggested solution. And the bounty was at the suggestion of your devs.

Suggested solution

Assumption: SRAM is held in memory during emulator running.

Test to determine whether it is safe to write SRAM to disk:

Detect change in SRAM
Note timestamp of change
If last SRAM change was greater than T seconds ago (e.g., 10 seconds), write SRAM change to disk. Else, disregard SRAM change and do not write to disk.
Effect of this would be that games that are using the SRAM as working memory would fail the test and never write to disk, because a game that is using SRAM as working memory would be writing more than every T seconds (and any edge case that contradicts would not risk ruining disk or performance anyway because it is only manipulating the SRAM every 10 seconds). Otherwise the game would pass the "safe to write" test and will write SRAM to disk immediately. Adjust T to be appropriate amount of seconds (it seems 1 or 2 seconds is probably fine).

In case the game is using the SRAM as working memory, this test should be placed on a cooldown so that it the test is not being run 60 times per second (if the test fails then cooldown for T2 seconds, e.g., 10 seconds).

Remaining edge case I can think of is that a game is manipulating a portion of SRAM but still using a portion as actual SRAM. These games would fail to write to disk, but would still write on graceful exit, and it is a much smaller pool of potential games that results in unexpected data loss as opposed to current implementation.

[inactive123]

I read this stuff already. Don't tell me what I have and haven't read.

And I still don't understand what the point is. Just enable save interval and call it a day already. If you say you can do it better, then create the PR already if you are convinced you have some superior solution. I don't see why we need a bounty for it then if you have it all figured out.

Besides - SRAM/serialization state could change regardless of whether anything is being saved or not. I think this could backfire in a lot of different ways.

Overall, I don't see it really being worthwhile and I don't see the point. RA isn't supposed to crash like that and it rarely does on its own. And if it crashes, you have bigger problems going on regardless which are most of the time core-bound. I'd say fix the thing that is supposed to be fixed then instead of creating this elaborate workaround which seems to just add complexity for no real reason.

[Monroe88]

The current solution is to set an obscure option to "Autosave SRAM" every 10 seconds or whatever, which senselessly rewrites the same save data over and over.

It only writes SRAM to disk if it changed since the last interval. If it is not changed, then it doesn't get written.

RetroArch/tasks/task_save.c

Line 150 in 3cb5b65

differ = memcmp(save->buffer, save->retro_buffer,

It's not on by default so you don't learn about it until after you lose your saves.

There was a proposal to set it to default to 10 seconds intervals on PC and Android, but it was rejected for reasons that were mentioned earlier. Personally, I think it should at least be enabled by default with 60 second intervals because the benefits outweigh the potential downsides. The autosave function is threaded so it shouldn't affect performance when it writes, at least on PC platforms.

In any case, it probably should be made more clear that RA only saves SRAM to disk on exit by default, unless you change the Autosave Interval setting. Perhaps that should be added to a README or FAQ somewhere.

[ghost]

Since someone in IRC was asking about SIGTERM handler support, I looked into it and apparently it was already implemented last year for Linux, in a graphics context driver specific way. For example, X11, Wayland, DRM etc. all check if a SIGTERM was received, and if so, will gracefully shutdown from within their message/window loops. So for people using launchers that are having issues with SRAM not saving on exit, they must either be using Windows/OSX/etc. where we don't have this implemented (can it be?) or the launcher is using SIGKILL or something else that causes it to not be handled gracefully.

[ghost]

also related to #4837

[lachlankrautz]

Windows user here. Running roms through steam big picture, if you close game using steam it doesn't save. One of the big benefits of libretro is being able to wrap it in whatever front end. That front end will likely just force close and therefore not save.

I'm very surprised to see people don't think it's a big deal that the game can report "save successful" and the save is still in memory and not on disk. If a certain rom is abusing save mechanics isn't it that game's problem; why would we design a system that better suits the edge case than the general case?

IMO it should save to disk when it says it's saving disk, games that save too frequently should have an option to save on interval instead.

[hizzlekizzle]

If a certain rom is abusing save mechanics isn't it that game's problem; why would we design a system that better suits the edge case than the general case?

The problem is that if we write when SRAM changes, games that use SRAM for RAM will be constantly writing to disk, which would thrash performance on I/O-constrained platforms and risk damaging or reducing the lifespan of media with limited write cycles. That's pretty high stakes, and no heuristic is foolproof. There will always be some edge case that blows it up, and from our perspective, it's better to hear "your dumb policy made me lose 3 hrs of play time" than "your dumb policy destroyed my SSD".

[trapexit]

This requires per core changes but adding a function to force a save or indicate change could be a compromise.

[andres-asm]

@lachlankrautz well steam doesn't close RetroArch. Steam kills it, it's not the same and hardly a use case we should cater to.

[ChessSpider]

yeah lost 4 hours of gameplay 👎 . Wouldn't have if there was at least some default setting of autosaving, or at least asked if I wanted to enable autosaving during install or whatever.

[Sephius]

Sorry the possible ignorance, but how other emulators deals with that? There is not a way do actually detect when actual saving game is occurring? So RA save to the disk on this kind of event. Also, there is no way of RA split SRAM in a way that the portion used for save games not being affected by game using the remaining SRAM?

Look... As game dev (not so much experienced as you should be), i can say that some times, there is approaches that makes much sense for the devs, but no sense at all for the users. A emulator should work as much as possible as the real console in term of experience (or better). I don´t know a game system witch only retain save files if a game "closes graceful". User always will expect a save to actually be saved after game says it was. And yes, loosing hours of game progress is a very bad experience. Different of a real game system RA crashes a lot when you are not using a pre configured installation.

Most os the emulators, simply save the save game to disk when you actually saved the game with the game running!

Please, consider change this approach since is not user friendly.

[andres-asm]

In many cases there is no way to differentiate a read or a write you just know there was access. Also there are many cases were you can't tell if it's writing as in saving, or just using SRAM as scratch. So no it's not trivial. Also crashes are not really a thing afaict, I haven't had crashes outside debugging in months. If there are crashes you should tell us doing what so we can fix the crashes instead.

…

On Mon, Dec 3, 2018 at 9:37 PM Sephius ***@***.***> wrote: Sorry the possible ignorance, but how other emulators deals with that? There is not a way do actually detect when actual saving game is occurring? So RA save to the disk on this kind of event. Also, there is no way of RA split SRAM in a way that the portion used for save games not being affected by game using the remaining SRAM? Look... As game dev (not so much experienced as you should be), i can say that some times, there is approaches that makes much sense for the devs, but no sense at all for the users. A emulator should work as much as possible as the real console in term of experience (or better). I don´t know a game system witch only retain save files if a game "closes graceful". User always will expect a save to actually be saved after game says it was. And yes, loosing hours of game progress is a very bad experience. Different of a real game system RA crashes a lot when you are not using a pre configured installation. Most os the emulators, simply save the save game to disk when you actually saved the game with the game running! Please, consider change this approach since is not user friendly. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4901 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABpC0AALYMyJ--q309A83TDoFKGkz6z4ks5u1d_-gaJpZM4NYIiW> .

[hizzlekizzle]

I don´t know a game system witch only retain save files if a game "closes graceful".

It's a bit of a digression, but NES required you to hold Reset before powering off or else it could/would corrupt/erase save data, and it's also one of the worst offenders for using SRAM for scratch.

[andres-asm]

Yes, say dragon quest, set your Auto SRAM interval to 1 and just play, it will be flushing SRAM once a second.

Now enable rewind and see the game glitch like crazy.

In all reality all games now tell you:

"Do not turn off the console when you see this symbol"
And if the game crashes right at that point.... though luck.

As I said. A better solution is to point out the crashes.
I haven't seen crashes during regular RA usage in years literally.

The only ways I can get RA to crash is doing stuff, and even then only stuff I've been testing or debugging. For instance recently I had crashes while loading ISOs via disk control in dosbox. Just using DOSBox though... no crashes.

I reckon there could be crashes of course, maybe while applying shaders and running out of registers... or maybe even during a load content.

But really outside of user actions I haven't had a crash while just playing and then closing content for a while.

[Sephius]

Yes, say dragon quest, set your Auto SRAM interval to 1 and just play, it will be flushing SRAM once a second.

Now enable rewind and see the game glitch like crazy.

In all reality all games now tell you:

"Do not turn off the console when you see this symbol"
And if the game crashes right at that point.... though luck.

As I said. A better solution is to point out the crashes.
I haven't seen crashes during regular RA usage in years literally.

The only ways I can get RA to crash is doing stuff, and even then only stuff I've been testing or debugging. For instance recently I had crashes while loading ISOs via disk control in dosbox. Just using DOSBox though... no crashes.

I reckon there could be crashes of course, maybe while applying shaders and running out of registers... or maybe even during a load content.

But really outside of user actions I haven't had a crash while just playing and then closing content for a while.

There some issues with what you said and it´s related with what i said about dev expectation VS user expectation:

• First, a system crashing during a saving operation is not what we are talking about. What we a talking about is the system only retain the data if in the moment system turn off correctly. We are not saying there is no other ways to losing save data.

• I don´t understand the solution "point out the crashes". If users come here to point out the some crash this means they already lost their data. Also, we are talking about a ultra complex piece of software witch emulates several kind of systems, with several different pieces of modular code, with tons of configs and witch need to make work thousands of games that was coded in a infinity kind of ways. You will never got to the point to have this program to be reliable as the original systems was.

• You, particularly, not seeing RA crashing in years does not mean system is not crashing with the users out there! And what is "regular usage"? For my experience with RA, i can say it crash several times! Using last version and JUST playing some games. Note that crashes also does not happen only by RA bugs, we are talking about PCs, and SOs which also have issues, GPU issues, drivers issues and a unimaginable amount of reasons a system like RA could crash.

• This RA behavior of only retain things like configs and save games after close are actually commented by RA users, in youtube videos and forums as a negative aspect of the system. Before download RA after seeing about in the internet the common warning i saw was "don´t forget to close the RA after change configuration or it will not retain the changes".

Sorry for insist on this, but i thing you are missing the point here. For a USER, this behavior of only actually save data on disk after exit the RA is not a expected behavior and can actually bring problems. The system does some so great amazing things, it´s difficult to understand why it can´t do this. I know the decision of what feature to focus is difficult and complex, but i really think you should consider this a priority.

It says at RA site the system is created to be simple and user friendly. This is a example of user friendly feature. Save game is one of the main aspect of playing games.

[trapexit]

@Sephius While I don't disagree that functionally what you're asking for is preferable it's nearly impossible to manage across the whole of the product without a significant undertaking. For some (most, all?) systems its just not clear when something "is saved." The complexity depends on the system and emulator. I can speak to 4DO since I more or less maintain it.

The 3DO just memory maps the NVRAM. It would normally look like a random read or write to memory from the CPU side. In 4DO we explicitly catch the memory range because its connected to the bus in a nonstandard way meaning we have the hooks to see when there is access. However, that access is totally arbitrary. There is no "starting to save", "done saving" instruction to watch for (AFAIK even at the filesystem level within the OperaFS there is nothing like that). The best that could be done is waiting for some time X after a write to copy it to disk. As mentioned in the thread we want to limit disk writing because writing too often can harm performance (even in a multi threaded environment) and could put significant wear on storage devices.

The most practical API would probably be like a watchdog. Pinging the core on occasion asking if something has changed and once RA notices its not changed it can save to disk after some time. That would limit the work necessary by the core. That would be better than the existing behavior in that it'd be fewer memory copies and comparisons... but how much better? Probably not enough a this point to rewrite the save routines for ever core.

That said... defaulting the timed save to 60s or something seems reasonable.

[andres-asm]

That default timer was done this weekend actually.

And that's what it does, it checks the content of the memory mapped as save ram every interval and writes if it has changed.

@Sephius pointing the crashing instances is still far more important than sweeping the problem under the rug by fixing the side effects of it crashing.

Saving the config "on change" has been discussed.
It would be worse, because if that you just changed causes a crash then you're effectively locked out of RA.

Again, reporting the crashes is what's important here.
I asked on discord and crashes while "just playing" are not a thing for most people.

crashes I know off:

• savestate on exit on a few cores (dolphin and ppsspp I think)
• setting a shader that causes your video driver to crash or run out of registers
• load content in some cores
• trying to override video driver or menu driver
• netplay pwd crash

[trapexit]

regarding being locked out from a bad config: true but a "save N backups of config file" would mitigate that issue. Leaving the selection of the file to be manual would simplify the implementation. If you wanted to get fancy you could have it try to recognize a crash and offer to roll back the config. But that's all offtopic for this thread.

[hizzlekizzle]

While saving config on change is very risky, IMO, I think saving config whenever you navigate to the next menu could be a good alternative. That way, if a bad setting triggers a crash, you only lose a handful (at most) of your most recent modifications and the offending option (probably) won't have been written to file, so you should be able to re-launch safely.

[Kaylakaze]

I lost 5 hours of gameplay today due to a gaggle of incompetent programmers. Yay.

[hizzlekizzle]

Be sure to file a complaint with our customer satisfaction department for a full refund.

[lachlankrautz]

it's better to hear "your dumb policy made me lose 3 hrs of play time" than "your dumb policy destroyed my SSD".

Seems like this issue regularly generates some outrage. I feel like if an SSD was ruined the blame would be on that game not the emulator anyway.

If someone took a shot at implementing an alternative solution would you consider the PR?

[andres-asm]

While I do think @Kaylakaze is completely off base and ignored a whole thread of debate.

The SSD argument is stupid too and is the kind of FUD that is repeated over and over through the years for no good reason.

Why?

1. First of all most SRAM data is minuscule. None of the cores that have big saves support the SRAM interface
2. It only writes when data has actually changed
3. SSDs dying due to writes is a meme, considering the dataset here we should stop using that as an excuse. It's a bad one
   https://techreport.com/review/27909/the-ssd-endurance-experiment-theyre-all-dead

Even if the timer is set by default though.
This won't solve the problem for any core that doesn't use the SRAM interface. Furthermore. I have still not found an actual report of what caused the crashes that made these people lose their saves.

[trapexit]

@Kaylakaze How about you describe precisely what happened and what your settings were?

@fr500 It's not about modern SSDs. It would be about sdcards and other forms of less durable flash. I've certainly had sdcards shit the bed due to what appeared to be extremely regular writes. However, the latency introduced by writing to slow storage would probably be a bigger issue vs wear if a core regularly changed data and the save period was low. Not sure that's been shown to happen though. I wouldn't expect most games to use effectively use sram randomly... but a core could.

[oxr463]

Not sure how portable this solution is, but on Unix-like systems we could write to a tmpfs in RAM e.g. /dev/shm (1).

[Kaylakaze]

@fr500 No, I read the whole thread. That's how I came to my conclusion. For two years, valid solutions had been proposed. The explanations for why they were ignored make no sense. I'm happy a default value for autosave was finally set at least. As someone mentioned, no one knows about the setting until they lose their data since almost every other emulator that's ever been made doesn't have this problem, so why would someone look for such a setting? While I understand sometimes needing to do significant coding to handle rare edge cases (such as using SRAM as RAM, which seems to be core specific), many times that can be solved with a well placed boolean (bool saveSRAMonChange) in the core. If you have a message like ePSXe does ("SRAM written to ") when it writes, someone could easily see if there was a rogue game causing constant SRAM file writes, and possibly disable it via option. Of course, none of that should be necessary if using the soutions offered here.

As for what caused the crash, it was more likely Windows than anything else. I was playing, then it was time for my work meeting so I switched over to Google Meet. Then I had some work to do so I fired up my VM, worked for a few hours. Lots of task swapping, etc. By the time I made it back to the game, it just wouldn't respond anymore (the Windows "Application not responding" window). When dealing with Windows, you learn when you should worry about trying to debug a problem and when you should just say "Windows did it" and restart the program.

[ChessSpider]

Same for me; crash due to user error. I saved intermittently within those 4 hour, so wasn't happy when i lost as much progress as i did.

[zaphod77]

I can think of a fairly simple heuristic fr detectign scratch sram.

This is under the assumption that sram is memory mapped.

The scratch area will be changed often, and the actual save area will be changed rarely (games with saves on cart always make you save in game. they do not save as you go, they save when you tell them to, and give a warning to not shut off during the save, to avoid corruption

SO we can assume that if the sram area changed often (say more than 4 times a minute), that it's scratch area, and it being changed shouldn't trigger a sram write. But if an area is changed that hasn't yet been flagged as scratch, then not too long after after it should write a new save file. also we can generally assume that if an ares is written to sequentially, it's most likely a real save, and after the writes stop, it should save.

that's my idea for an approach, and i think it will work. i just pulled some numbers out of my butt, but it should be easy enough to tell if an area is used as scratch or for real save data, based on how often the location is written to.

[trapexit]

@zaphod77 As @fr500 already mentioned an auto save was already added: #4901 (comment)

The problem isn't that people don't know ways to solve this. It's doing so in a reasonable way that doesn't impact the existing API. The latter is more difficult.

[edrozenberg]

I'm pretty new to RA and also got bitten by the lack of default autosave (3 hours lost). Like many others I came here looking for issues related to this and found out there is no default flush to disk before exit. RA is supposed to never crash (may be true!). But you have scenarios like starting a game remotely on an RPi from a computer that later falls asleep :), or unplugging power to an RPi after "saving" a game.

I read and appreciate the reasons for no default autosave, but it would be nice to somehow warn (all new) users about this. Since RA doesn't come in a box :) there's no way to print a warning on one. Maybe a warning the first time RA is launched? Maybe a BIG warning on the main RA download areas, FAQ, READMEs? It is just so far from obvious that saving in-game does not translate automatically to save to disk. And it is so likely to bite even otherwise savvy users sooner or later.

[zaphod77]

by play nicely i mean such that any time you do an ingame save, the sram is saved. this is very easily detected, EXCEPT when a game uses sram as work space.

checking periodically isn't good enough, because you have to manually trigger saves in game, and then there may be up to a five minute delay.

flushing sram when closing the emulator is easy, but it's non trivial to know when an ingame save actually happens, and immediately detect it, if the game also uses sram as workspace.

sure you may auto save every 5 minutes, or even one minute, but if it crashes after your manual save and before the auto flush, you've just lost that save, and possibly many hours of work. this is an issue with NES games, which never autosave.

the basic idea is that if there's no activity in sram for a small amount of time, flush it to disk. but if a game constantly twiddles it, then that won't help. the only real fix is a database of games and sram locations to be ignored.

[hizzlekizzle]

Most non-console systems have a default 10-second flush interval now.

[zaphod77]

yeah, the NES is just a very pathological case, and it's really difficult to just Do The Right Thing.

If the only time when sram is written to is during a save, it's very easy to do the right thing. this is always the case with memory card using cores. everything written to them is intended to actually be saved.

But battery backed sram cart saves are a whole different kettle of fish, especially if the sram is used as a scratchpad as well, which is known to be the case at times.

[JukePlz]

NES is just a very pathological case

But the NES and other consoles like Genesis are also the less affected by having the current saving behaviour, since those systems rarely have game saves at all or only have highscore saving (albeit some exceptions like sonic 3)

---

After reading the whole thread my question is:

Shouldn't this be a per-core defined logic in libretro instead of trying to find an imperfect catch-all default for all systems? Obviously the systems that have manual saves like PSX or GB are the ones that bring the most gripe to users, if we separate them from the pool we can find other defaults that make more sense, be it either a more sane auto-save SRAM timer, a core forced disk-write (for systems that don't scratch VRAM randomly) or keep the current behaviour.

Other things that come to mind is that there could be a hotkey to dump SRAM to disk so users on platforms that are at-risk of data loss (other front-ends, steam, frequent power loss, Lakka or RPI, etc) can just use that button every time they save and be done. Yes, I know, I know you can dump both VRAM and SRAM with a savestate, but that may not be desirable for newer systems as it's a waste of disk space for someone that only wants to use the native save-loading experience.

[zaphod77]

Main issue is that even with an auto flush interval of ten seconds, that's up to 10 seconds the save could have been ignored. especially with the case of save and immediately close ungracefully. you are almost guaranteed to lose the save, which can still be many hours. And what if you close during a delayed write ungracefully? save lost.

This is again only an issue with battery backed sram, and again only if a game uses it as scratch space. If the game does not use it as scratch space, doing the right thing is very easy, as it will only changed when something is saved, and it can very quickly catch it up.

Now we can almost certainly reliably assume that scratch space and actual save space never overlap. so for any game, after analyzing it's code we can determine what's actual save space, and what's scratch space. then we simply ignore writes to the scratch space when determining if the save buffer is dirty and needs a flush.

As i said, how frequently the area is modified is a reasonable heuristic to determine scratch space. If it's changing values a lot, then it's most likely scratch space. if not then it's a real save, and should be flushed to disk. So the core could probably track how often a section of sram is modified. This would allow say dragon warrior 1 to ignore the scratch space usage, but catch a real save very rapidly.

ALL optical based systems can and should use the same logic. We know for a fact that flash save areas are only written during intentional saves, so they are easily detected and written to disk rapidly.

Later systems that use flash instead of battery backed sram are similar. they can just as easily do the same stunt and do the right thing. (DS, 3ds, etc.) They will not use the save area as scratch space, to avoid wear on the flash.

disk images are again the same. every write to them is genuine, and intended to be saved.

it is only battery backed sram, independent of core, that needs special heuristics, and only if it's also used as scratch space.

I hope there is no game completely insane enough to have scratch space in the middle of a "save file".

[JukePlz]

reasonable heuristic

That's a good idea in theory but how do you implement it in practice:

What if the game only uses the ram as scratch in rare situations and the user saves frequently, turning the logic backwards?

What if the user saves before the heuristic determines the area that is scratch?

How do you test it for massive catalogs of games that may have different scratch behaviours for different parts of the game? Do you start all over again if you have to change the heuristic algorithm?

I think fixing the save logic to be inmediate for non-battery based SRAM should come first, and testing of an experimental heuristic after for the remaining systems. Asking around how standalone emulators handle those systems should also be something to take into consideration, as there may be some tricks in the known to save the trouble of writing such complex watchdog just for that.

[trapexit]

@JukePlz Having it save immediately would require a new API to do efficiently which I think generally there has been push back on. You need a way to indicate the data changed and ready to be saved. Even if an API existed I'm not sure how useful it would be. It could minimize memcmp's but really there is a race condition. How do you know that the game/software isn't in the middle of writing unless you have some high level emulation? You could easily corrupt the save if the game is writing in chunks. You perhaps could make the API something like "How long has it not been written to" and then hope that it's good enough.

The better and safer thing might be to use (default enable) auto save states. They won't have that race condition.

[hizzlekizzle]

Asking around how standalone emulators handle those systems should also be something to take into consideration

I've done this. It's usually not too much of an issue for standalone emus because they either have the scratch risk (e.g., NES emus) and they flush on exit and/or interval (like we do), or they don't have the scratch risk and have a mechanism to say "saving is done in the core, write it to file" (i.e., as trapexit described).

Multi-emus like Mednafen and Bizhawk have to deal with the same issues we have and neither have a magic bullet, unfortunately. In discussing it with zeromus, they suggested flush-on-exit with a manual hotkey trigger (I'm not sure if Bizhawk actually does this or if it was just a hypothetical suggestion).

We also have the added complication of some cores not using the libretro save interface (e.g., beetle-psx when you select the memcard save method), so the flush interval doesn't take effect, either.

[zaphod77]

If a game rarely uses sram as scratch space, it's not an issue and treating it as a real save is fine. The entire issue with changing at every, well change, is when a game does it often enough to wear out the flash drive.

And, well, you try manually saving more than 5 times a minute in a nes game. go ahead. I'll wait.

It's impossible to trigger writes to the real save data area rapidly enough to fool my proposed heuristic. Even if the game continues immediately after the save, which is pretty uncommon, you still have to open the menu or talk to the npc, select the save slot, and that always takes at least a few seconds. NES games simply aren't that fast.

Assume that everything is real until the memory location is written too frequently to be real does the right thing.

[trapexit]

The entire issue with changing at every, well change, is when a game does it often enough to wear out the flash drive.

That's not the "entire issue." Without HLE you won't know when the game isn't writing. There is a race condition now... there would be a race condition then. Solving that is non-trivial*. It also doesn't fit into the existing API so a new API (not just some change of algo) would need to be created and plumbed into every core.

RA is already defaulting "SaveRAM Autosave" to 10 seconds on a number of platforms.

RetroArch/config.def.h

Line 830 in 60cca4d

#define DEFAULT_AUTOSAVE_INTERVAL 10

If you think other platforms should have it enabled by default... make a ticket and argue why it shouldn't be a problem.

• If you just use auto save states you do in fact get safe save state storage because the whole of the machine's state (if done properly) is being saved.

[zaphod77]

the idea is to mark areas as scratch pad when they are written too way too frequently to possibly be actual saves, and ignore them in determining when writes have stopped and it's safe to flush.

It's pretty easy to detect writes to an area of emulated memory. any decent debugger can do this. so we simply add a bit of code that logs what addresses are written to and how often.

saves are always written out all at once in the coding loop that writes them. so a save is rapid writes to different (usual consecutive) memory locations, while scratch space is many writes to the same locations. So it seems entirely reasonable to have a heuristic that can reliably distinguish real saves and scratch pad use.

to be fair i don't know how insane all game coding is, but i think it's a reasonable assumption that save slots go in specific areas that are only written to during saves, and that it's possible to work this out by heuristics.

[trapexit]

We understand the premise. It does not address all the concerns listed above. Regardless... it is far more complicated than alternative designs. Alternative designs that really aren't necessary but might minimize the cost a bit. As pointed out the default is to auto save every 10 seconds for a number of mainstream platforms.