Save detection

[ghost]

Save type hardcoded to 32 KiB (256k) SRAM. Unknown how save type detection will be done.

Why not use a native_firm application to select a rom and write the path (to rom and save) and the save type to "open_agb_firm.ini"? Then trigger a reboot to open_agb_firm and it loads the ini file. I figure in native_firm there's more room to work with. You might even be able to write the config to ram since it isn't cleared on a reboot.

[profi200]

That's not what i had in mind. The goal was to completely decouple this from userland (pesky GBA VC "injects" and stuff) and make it optionally configurable/bootable from userland (but it should itself have a menu for configuration/game selection). The main way of running this should be through some bootloader. Luma's chainloader or fastboot3DS for example.

Flash memory hates writes. I don't think writing to the SD card every boot is a good idea especially writing less than erase block size. And space is not a problem currently. I still have large chunks of VRAM free and AXIWRAM is basically barely used.

I have talked with endrift (mGBA dev) about save detection and i think we agreed the best way is having a game database with at least save type and hashes of known good dumps. But then the problem is building this database properly. So i will probably resort to string search-based detection in the meantime which is not accurate in some cases.

[Kirit29]

how would this game database for saves work with modified roms aka rom hacks? As some change the save type. idk would per game settings be possible? or a check by game id?

[profi200]

If the save type is the same and the game code matches the original ROM it will probably detect it fine however with a hash mismatch. When the game code is changed there will be no other way than save override.

[Kirit29]

by game code do you mean the game id? ie BPRE for fire red? if so that can be changed too match by a hex edit

[ghost]

Ah fair enough.

Flash memory hates writes. I don't think writing to the SD card every boot is a good idea especially writing less than erase block size.

That's why I also suggested writing it to an area of ram that doesn't get overwritten on reboot. That would be more ideal.

[profi200]

Yes, i mean the 4 chars ID. It would be preferable if ROM hackers don't touch the ID. That would save everyone time.

Are there really cases where the ROM hack changes save type? Do they have a good reason to do that?

[Kirit29]

there are not many that change the save type its rare that that happens ive only seen it once.but there are many that add rtc to games that never have it ie. firered hacks.

[profi200]

Ugh, which is basically a save type change. Not sure if i can get away with assuming these games use the RTC enabled save type.

[Kirit29]

wouldn't just editing the game id via hex fix that? fore example bpre to axve. as that's ruby's id which uses a save type that's rtc enabled. that works for some emulators that have automatic rtc detection.

[profi200]

I can't expect noobs to pull out a hex editor to fix this by themself. It just doesn't work.

[Kirit29]

tbh i think the people in the rom hack playing community should know how to hex edit if they go through the process of of patching a rom with a computer much less installing a bootrom exploit on their 3ds to install cfw to run open agbfirm in the first place. and if necessary I can make a guide for you,for them, if it becomes a big enough problem that will explain the hex editing process.

[profi200]

Remember that not only experienced people play ROM hacks. I don't see many issues regarding this coming yet. At least hopefully.

My motivation to do anything is kinda down right now. I'm trying to do something useful to get the project forward but it barely works :/

[Kirit29]

No problem i understand fully. lack if motivation is a pain. Good luck on this project tho i will watch it closely as the 3ds is mainly what i use for gba emu and this would be a godsend. Not to mention a first in 3ds hackinglol. Take your time and work at your own pace

[profi200]

I mean. It's already fully playable if we ignore the missing save detection and file browser. I completed Metroid Zero Mission with this a few days ago and it was flawless. But if you compile it with a RTC enabled save type beware that it doesn't set the RTC properly yet.

I'm thinking about splitting this further into the project and a "lib3ds" of sorts. I wanted to do this for a long time with the fastboot3DS source (this is based on it) since it has code for almost everything now.

[Kirit29]

Unfortunately i can't compile it myself because my pc is out of my hands for awhile. Btw are u saying you can compile it with a different hardcoded save type? If so Im genuinely curious and would like to know how so i can get someone to do it for me ftm

[ghost]

Might I recommend checking out the source code for Gericom's GBARunner2? It's a similar project to yours and he's already got save detection working. (He also has the 3DS gyro working in GBA games! That's cool.) His code runs directly on the ARM9 and ARM7 so I figured you might be able to reuse some of it.

[Kirit29]

actually gbarunner2 uses some kind of special patches in place of genuine save type support iirc.

EDIT: Yup uses special sram patches according to gericom Gericom/GBARunner2#40

[ghost]

EDIT: Yup uses special sram patches according to gericom Gericom/GBARunner2#40

Well it can't patch everything to sram. Pokémon saves fine in it. If it was sram patched the save wouldn't be big enough and it would get corrupted.

Edit: Yeah it looks like he isn't patching them to sram. He's patching them to some kind of custom save type.

Gericom/GBARunner2@d6ae472

[ghost]

if i remember correctly pokemon can run with sram saves abiet with a few issues that are not gamebreaking

No the save corrupts when you beat the elite four and it'll throw up an error about the save being damaged every time you start the game. GBARunner2 doesn't cause this behavior. See that linked commit.

[Kirit29]

yeah i saw the linked commit lol. that's why i deleted my comment.

[profi200]

GBARunner2 works in a fundamentally different way to this. GBARunner2 runs in DS(i) mode and traps register accesses + emulating missing GBA hardware. This is basically a full GBA SoC implemented inside the 3DS SoC which means it has 100% compatibility (if we exclude GBA carts with extra hardware other than RTC and 64 MiB games). Also since this leaves ARM9/11 in 3DS mode we can use the 3DS GPU for scaling and the extra resolution of the screens.

The string search based save detection approach works but it's not 100% reliable which is why i wanted to have a database.

Save type can be changed here:
https://github.com/profi200/open_agb_firm/blob/master/source/arm9/hardware/lgy.c#L51
https://github.com/profi200/open_agb_firm/blob/master/source/arm9/hardware/lgy.c#L74

[Kirit29]

ok ive figured out how to change the save size on my own but how dabout the type? @profii200

[profi200]

It's the GBA VC ROM footer values.

https://www.3dbrew.org/wiki/3DS_Virtual_Console#Config Below the table.

[Kirit29]

profii are you aware that the official register defs leaked recently? according to gericom they make the agb save registers more clear.
Or is this old news to you lol
if not i can send them to you

[profi200]

I have them since over a year (stripped down comments but they were not so important anyway). If you look closely at some code you will see it's "heavily inspired" by official headers.

[Kirit29]

i played pokemon emerald Enhanced last night on open agb_firm and it ran flawlessly and so did the rtc.

edit: nvm the rtc doesn't run after the console is turned off. just booted it up again and the time is the same as when i turned it of last night.
edit you mentioned that the rtc doesnt set right. Do you have a plan on how to get it fully working? I'd be willing to test your efforts on that when the time comes and if it's need.

[profi200]

The RTC probably does run but it's reset to 1.1.2000 00:00 every boot. The biggest problem is probably that the 3DS side RTC is completely off unless you set it correctly with GodMode9. Other than that it's a matter of copying it from the 3DS RTC to the GBA one.

[Kirit29]

Profi is there somewhere else i can contact you? I got a few questions do you have a discord?

[profi200]

No, but i'm on the #GodMode9 IRC channel on Freenode (which has a bridge to the GodMode9 Discord).

edit:
To be clear: The no was regarding Discord stuff.

[profi200]

In theory the RTC problem is solved now. Only day of week is not set but i don't know if this is used by any game.

[Kirit29]

alright reviewing that source code of dns as far as i can see it seems it uses the date set at the start of the save to calculate day and time from then on every boot it offsets the time and day based on what i can see. it doesn't actually use the days of the week.

[profi200]

I don't know what "dns" is. Never heard of it. The original games not using day of week is at least good news (however that's just Pokémon). So i assume it's working?

[Kirit29]

Dns stands for days nights and seasons basically its a tool add code to a pokemon rom to make it use rtc when its doesn't usually like fire red for example @profi200

Also i haven't been able to test it out yet as im unable to build the firm. My pc is 30 miles away at my dad's and he's quarantined with the virus. I can test if you want but you'll have to send me a build. If that's ok with you. preferably SAVE_TYPE_FLASH_1m_SNO_RTC with save size of 128 kib

[profi200]

I opened my original Emerald cart and it looks like it's using Macronix flash which is what i have been using in my test with this game. Theoretically the save type doesn't matter as long as the size and addressing match (and the game doesn't check the flash ID).
I found 1 little flaw with the current RTC handling. When starting a new game the Pokémon games reset the RTC to 1.1.2000 00:00:00 and if i save and reload the FIRM it jumps right back up to 2020. AgbBg solves this by using date/time offsets and saving them along the savegame. I can probably get away with this since mGBA does the same (just returns the current date/time).

open_agb_firm.zip

[Kirit29]

thanks btw was looking at gbatek for the rtc based functionality of the gba and nds then found this

Day of Week Register (septenary counter)
0-2 R/W Day of Week (00h..06h, custom assignment, usually 0=Monday?)
3-7 - Not used (always zero)

idk if it helps with your problem or not

[Dartz150]

So I haven't found any other place to discuss this, so in the mean time Iwill use this issue post.
I have been testing this and so far is working to an usable state (I'm sure there are still many things to be added) and I'm pretty impressed on how it works far better than stock AGB_FIRM, congratullations!
I see that save types and saving are not implemented yet but while reading this topic it seems that these are very close to be added, so I will ask only two things to continue my tests (I'm rebuilding this each time you push new commits btw).

1.- rom.gba at the root is a temporary dir right? I supose a file explorer will be added later, or will it be fronted based much like how TwilightMenu works with nds-bootstrap?

2.- It is possible to enable the BIOS intro? I have also noticed that if no rom is loaded, the bios boot animation shows up like half minute later.

3.- I see that the color display is not washed out and a bit darkened, which looks excelent! Does it use a LUT? Or some kind of shadder?

4.- Is it planned to add a customizable layout for the screen stretch and palette darkness?

Thank you in advance, and very sorry if I bottered you with these questions, this looks pretty good so far and I would love to continue testing this alongside GBARunner2 which is another amazing project.

[Kirit29]

@Dartz150 hey dartz its Dualbladedkirito from dsi mode hacking discord! You can change the save type but you have to recompile it every time for now. Change it with these https://github.com/profi200/open_agb_firm/blob/master/include/arm9/hardware/lgy.h#L12-L30
Here
https://github.com/profi200/open_agb_firm/blob/master/source/arm9/hardware/lgy.c

[Dartz150]

Hey! Nice to see you here too @TriforceOfPwnage!
Great, thanks for the tip, works as expected.
As suggested, I do agree that a database should be used to determine the save type for the rom, and for romhacks, it should be fixed on the romhack side part as these aren't supported properly in AGB_FIRM and other hardware as well (mainly because these are mostly made for android emulators), or maybe an external editable database could be used for that, so depending on the romhack, a new entry should be writen in plain text, to tell which save type is used with said romhack, specially on the RTC fire red romhacks.

[Kirit29]

Hey rtc function properly in fire red rom hacks with this. Unlike regular agb_firm

[Dartz150]

Great to know! Another point for this over AGB_FIRM, then I see that the main holdback now is how to standardize a proper way to detect the save type chip right? See my comment above.

[Kirit29]

Pokemon adventure red doesn't boot tho. Green bottom screen and black top screen @profi200 do you know what would be causing this? I know that this game works on a gba flashcart on real hardware.

[Kirit29]

@Dartz150 also hes planning to make it have it's own rom select menu and be configurable within the firm iirc

[profi200]

@Dartz150

1. A file browser is planned.
2. Already implemented but hardcoded to off currently. Set this to true: https://github.com/profi200/open_agb_firm/blob/master/source/arm11/hardware/lgy.c#L39
3. It technically does use a shader but the PICA200 supports no custom shaders. It's just this as used by mGBA: https://gist.github.com/profi200/e31df7d31b8c2cdccc0608aaa71681fc#file-main-c-L75-L78
4. Maybe later. Without a proper citro3d port this is not easy to make configurable at runtime. Color lookup tables like on original GBA VC can be added easily though.

@TriforceOfPwnage
ROM too big/SD card corruption? This is the ARM9 calling panic(). ROM size is limited to 32 MiB (hardware limitation). I should probably implement proper errors.

[Kirit29]

is there any way to trim the rom down abit via hex? its 1131 bytes over 32mib.

[profi200]

Unless that data is not needed i'm afraid no. You could ask the autor of the ROM hack to try and cut the size down.

Games >32 MiB use a special mapper that is not supported by the cart emu: https://mgba.io/2015/10/20/dumping-the-undumped/

[profi200]

The BIOS animation and save type settings have been moved here:
https://github.com/profi200/open_agb_firm/blob/master/source/arm11/main.c#L40

Save size is now automatically calculated based on save type.

[Dartz150]

Mmm.. I'm trying to build the latest comit but I'm getting this error, it didn't happen before:

make[2]: dma330as: No such file or directory make[2]: *** [/c/open_agb_firm/arm11/Makefile:178: lgyfb_dma330.h] Error 127 make[1]: *** [Makefile:117: all] Error 2 make: *** [Makefile:60: release] Error 2

[profi200]

I didn't change anything related to makefiles. This tool must be somewhere in your path. Recommended way is to create some folder where you throw in all unofficial tools and add that to your path environment variable.

[profi200]

Filesystem access has been reimplemented for ARM11 which i wanted to get done before i implement save type detection.

I will see if i can get the later working in the next days (it will be the inaccurate save type detection for now).

[Kirit29]

Do you mind sending me a build @profi200 with bios enabled to test

[profi200]

Thought you can build this yourself now? Anyway.

[removed build]

[profi200]

I need testers. The EEPROM save types are still mostly untested. Everything without the "Confirmed" comment needs testing. Unfortunately i don't know which games use these types:

https://github.com/profi200/open_agb_firm/blob/master/source/arm11/hardware/lgy.c#L104-L110

A test build from latest commit without BIOS animation to speed up testing:
[removed build]

[Dartz150]

Great, I'll update this post with the tested save types, addind a "Working" comment at the end and the games tested with them, I will make a new post when finished.

{"EEPROM_V111", SAVE_TYPE_EEPROM_8k}, // Working
// -Super Mario Advance (Japan)

{"EEPROM_V120", SAVE_TYPE_EEPROM_64k}, // Not Working
// -Super Mario Advance / "The saved data is corrupt" message
// -Doom / Game can't save
// -Doom II / Game can't save

{"EEPROM_V121", SAVE_TYPE_EEPROM_64k}, // Working
// -Street Fighter II Revival (Europe)
// -Bomberman Max 2 - Blue Advance

{"EEPROM_V122", SAVE_TYPE_EEPROM_64k}, // Working?
// -Dexter's Laboratory - Deesaster Strikes! / Can't save
// -Medabots AX - Metabee Version / Can't save
// -Super Mario Advance 2 : Super Mario World + Mario Brothers (Japan) / Works

{"EEPROM_V124", SAVE_TYPE_EEPROM_64k}, // Working
// -Famicom Collection - Super Mario Bros 2 (JAPAN)
// -Classic Nes Series - Castlevania (Europe)

{"EEPROM_V125", SAVE_TYPE_EEPROM_64k}, // Not working
// -2 Games in 1 - Spider-Man + Spider-Man 2 / Detected correctly, white screen 
//  at startup. Note: This is the only single game in the entire GBA library that uses this save type ID.

{"EEPROM_V126", SAVE_TYPE_EEPROM_64k}, // Not Working
// - Harry Potter and the Goblet of Fire (USA) / Detected correctly, black screen.
// - Yggdra Union : We'll Never Fight Alone (USA) / Detected correctly, Half of the screen and 
// some scenes are black, crashes with screech sounds after some intro music.

// FLASH
// Assume they all have RTC.
{"FLASH_V120",    SAVE_TYPE_FLASH_512k_PSC_RTC}, // Working
// -Yu-Gi-Oh! Dungeon Dice Monsters (Japan)

{"FLASH_V121",    SAVE_TYPE_FLASH_512k_PSC_RTC}, // Working
// - Advance Wars (USA)
// - Chu Chu Rocket! (USA)
// - Dragon Quest Characters (Japan)

{"FLASH_V123",    SAVE_TYPE_FLASH_512k_PSC_RTC}, // Working
// - Golden Sun (USA)
// - Golden Sun: The Lost Age (USA)

{"FLASH_V124",    SAVE_TYPE_FLASH_512k_PSC_RTC}, // Working
// - Mario Kart: Super Circuit (USA)
// - Guilty Gear X - Advance Edition (Japan)

{"FLASH_V125",    SAVE_TYPE_FLASH_512k_PSC_RTC}, // Working
// - Super Robot Taisen A (Japan)
// - Nobunaga no Yabou (Japan)

{"FLASH_V126",    SAVE_TYPE_FLASH_512k_PSC_RTC}, // Working
// - Magical Vacation (Japan)
// - Sonic Advance (Japan)

{"FLASH512_V130", SAVE_TYPE_FLASH_512k_PSC_RTC}, // Working
// - Sonic Advance 2 (USA)
// - Shining Soul (Europe)
// - Final Fantasy Tactics Advance (USA)

{"FLASH512_V131", SAVE_TYPE_FLASH_512k_PSC_RTC}, // Working
// - Sword of Mana
// - Shining Soul (USA)
// - Sonic Pinball Party (USA)

{"FLASH512_V133", SAVE_TYPE_FLASH_512k_PSC_RTC}, // Working
// - 2 Games in 1 : Sonic Advance + Sonic Pinball Party (Europe)

{"FLASH1M_V102",  SAVE_TYPE_FLASH_1m_MRX_RTC}, // Working
// - Super Mario Advance 4 - Super Mario Bros. 3
// - Pokemon Mystery Dungeon - Red Rescue Team

{"FLASH1M_V103",  SAVE_TYPE_FLASH_1m_MRX_RTC}, // Working
// - Pokemon - Edicion Esmeralda (Spanish)
// - Pokemon - Edicion Zafiro (Spanish)
// - Pokemon - Edicion Ruby (Spanish)
// - Pokemon - Rojo Fuego (Spanish)
// - Pokemon - Emerald Version (USA)
// - Pokemon - Sapphire Version (USA)
// - Pokemon - Ruby Version (USA)
// - Pokemon - Fire Red Version (USA)

// FRAM & SRAM
{"SRAM_F_V100", SAVE_TYPE_SRAM_256k}, // Working
// - Mr. Driller 2 (Japan)
// - Tanbi Musou - Meine Liebe (Japan)

{"SRAM_F_V102", SAVE_TYPE_SRAM_256k}, // Working
// - Castlevania - Aria of Sorrow (Europe)
// - Castlevania - Harmony of Dissonance (Europe)

{"SRAM_F_V103", SAVE_TYPE_SRAM_256k}, // Working
// - Hamtaro : Ham-Ham Heartbreak (USA)
// - Shin Megami Tensei Devil Children - Koori no Sho (Japan)

{"SRAM_V110",   SAVE_TYPE_SRAM_256k}, // Working
// - F-Zero (Japan)
// - Akumajou Dracula - Circle of the Moon (Japan)
// - Castlevania - Circle of the Moon (Europe)
// - Jisu F-Zero Weilai Saiche (iQUE)(CHN)

{"SRAM_V111",   SAVE_TYPE_SRAM_256k}, // Working
// - F-Zero - Maximum Velocity (USA)
// - Kurukuru Kururin (Europe)
// - Breath of Fire (Europe)

{"SRAM_V112",   SAVE_TYPE_SRAM_256k}, // Working
// - Megaman Zero (USA)
// - Metroid Fusion (USA)
// - Kirby - Nightmare in Dream Land (USA)
// - WarioWare, Inc. - Mega Microgame$! (USA)

{"SRAM_V113",   SAVE_TYPE_SRAM_256k} // Working
// - Metroid Zero Mission (USA)
// - Megaman Zero 2 (USA)
// - Yu-Gi-Oh! World Championship Tournament 2004 (USA)
// - Megaman - Battle Chip Challenge (USA)

[Dartz150]

@profi200 I just have completed several tests with every save type, and so far it seems that the only conflicting save types are:

{"EEPROM_V120", SAVE_TYPE_EEPROM_64k}, // Not Working
// -Super Mario Advance / "The saved data is corrupt" message
// -Doom / Game can't save
// -Doom II / Game can't save

{"EEPROM_V122", SAVE_TYPE_EEPROM_64k}, // Working?
// -Dexter's Laboratory - Deesaster Strikes! / Can't save
// -Medabots AX - Metabee Version / Can't save
// -Super Mario Advance 2 : Super Mario World + Mario Brothers (Japan) / Works

{"EEPROM_V125", SAVE_TYPE_EEPROM_64k}, // Not working
// -2 Games in 1 - Spider-Man + Spider-Man 2 / Detected correctly, white screen 
//  at startup. Note: This is the only single game in the entire GBA library that uses this save type ID.

{"EEPROM_V126", SAVE_TYPE_EEPROM_64k}, // Not Working
// - Harry Potter and the Goblet of Fire (USA) / Detected correctly, black screen.
// - Yggdra Union : We'll Never Fight Alone (USA) / Detected correctly, Half of the screen and 
// some scenes are black, crashes with screech sounds after some intro music.

"EEPROM_V122" needs to be tested further with other roms just to confirm, I only have the ones I tested currently available.
With the last two types, it seems to be a rom problem rather than a save problem, as their size is the biggest found in the GBA library (32 MB)

The source I used to confirm and track the save types is: http://offlinelistgba.free.fr/index.php, I recommend to set the search parameters to "From the rom: 1, display: 2900 roms." so the entire library can be searched with Ctrl+F.

Of course a second testing view should be made to cross-confirm that nearly every save type is working correctly, let me know if you have questions about my tests.

[profi200]

What about latest commit?

[removed build]

[profi200]

Closing due to a lack of feedback. If there is still a game that doesn't save correctly (and i'm sure there is) open a new issue for collecting all the problematic games in one place.

Next is probably a very basic file browser.

[Dartz150]

Hello! Sorry for not reponding before, I was somewhat busy. I have tested with the build you comented above and can conmfirm that EEPROM_V120 and EEPROM_V122 are now working fine. The other two save types with issues remain still unfunctional.

[profi200]

Can't reproduce. V125 and V126 are working here. Did you delete the old (and likely broken) save file?

New test build with the file browser i added (yes, it's a pretty crappy file browser but good enough for now):
open_agb_firm.zip

[subad31155]

add support for epprom v122 save type????????