<10.4 FIRMs can't insert loader sysmodule (it grew by 0x200 bytes)

[phijor]

I compiled corbenik from source an noticed that everytime I enabled the custom loader, the CFW would get stuck in a loop warning me that the loader.cxi is to big (L41 here) and that it would need to grow 1 unit.

It probably hangs because the following continue; just jumps back to the while-condition, which hasn't changed, and thus executes the same code again ad infinitum. I don't know if that's intended this way.

Too find out why it complained that it'd need to grow, I did a git bisect. Starting with 19cc0eb the issue appeared for me the first time and also did the size of loader.cxi change from previously 12288 bytes to 12800 bytes after.

I understand that the 'shift all the modules'-code is not implemented right now, but can you confirm that the custom loader indeed exceeds the size of the original?

Anyway, cool project! If I could get any ressources on how develop patches/what patches are needed, I'd be on board writing some!

[chaoskagami]

That was meant as temporary, but yeah, unfortunately, looks like post-cleanup of @Wolfvak's code, the size is now fucked. It didn't start until the latest commit here, rather than 19cc0eb so I can only assume either A) my ctrulib is smaller or B) my compiler (gcc 5.3.0) is optimizing better. Nope not hitting it.

I'd suggest waiting on the patches for now; I'm working on a decidedly less crap assembler for them. Current ones are still useful to understand the bytecode format, but the syntax will likely change. Patches pre-assemble are located in (expectedly) the patch folder. Good examples are memexec.pco or the agb/twl patches.

If you want any help with the patches, feel free to PM or mail me.

[ghost]

This is odd, both dumping and loading worked perfectly for me.
And yes, this is with the 12800b loader.cxi, on O3DS.

[chaoskagami]

Hm. Actually, @Wolfvak is correct, it doesn't trigger here now that I actually copy the commit and test it.

What version of devkitPro are you using? More specifically, the following:

arm-none-eabi-gcc -v
arm-none-eabi-as -v
arm-none-eabi-ld -v
ctrulib version

[phijor]

$ arm-none-eabi-gcc -v
Using built-in specs.
COLLECT_GCC=arm-none-eabi-gcc
COLLECT_LTO_WRAPPER=/opt/devkitpro/devkitARM/bin/../libexec/gcc/arm-none-eabi/5.3.0/lto-wrapper
Target: arm-none-eabi
Configured with: ../../gcc-5.3.0/configure --enable-languages=c,c++,objc,obj-c++ --with-gnu-as --with-gnu-ld --with-gcc --with-march=armv4t --enable-cxx-flags=-ffunction-sections --disable-libstdcxx-verbose --enable-poison-system-directories --enable-interwork --enable-multilib --disable-dependency-tracking --enable-threads --disable-win32-registry --disable-nls --disable-debug --disable-libmudflap --disable-libssp --disable-libgomp --disable-libstdcxx-pch --target=arm-none-eabi --with-newlib --with-headers=../../newlib-2.2.0/newlib/libc/include --prefix=[[redacted]] --enable-lto --with-bugurl=http://wiki.devkitpro.org/index.php/Bug_Reports --with-pkgversion='devkitARM release 45'
Thread model: single
gcc version 5.3.0 (devkitARM release 45)

$ arm-none-eabi-as -v
GNU assembler version 2.25.1 (arm-none-eabi) using BFD version (GNU Binutils) 2.25.1

$ arm-none-eabi-ld -v
GNU ld (GNU Binutils) 2.25.1

ctrulib version (latest commit as of 2016-06-09):

r1195.6d65d53

[chaoskagami]

I have absolutely no clue, since your output matches mine. The only thing I can think of is that your version of GCC is miscompiling code for some unknown reason. Can you upload the arm9loaderhax.bin somewhere? I'm going to compare the output of your compiler.

EDIT: Also, the entire build directory would be helpful, as well.

It should only fail once it grows past 12800 - right now, we're the same size as nintendo's loader, and the CXI still has about 101 bytes of padding at the end. (f0 0f f0 0f ...)

[phijor]

I guess I figured something out:

I am still running 10.7 on my N3DS (don't ask me why). I dumped the loader.ncch from the decrypted firmware (10.2 NATIVE_FIRM that is, I believe). Then I used ctrtool -i to get to know about it:

$ ctrtool -t ncch -i 4013000001302.ncch | grep 'Content size'
Content size:           0x00003000

Well, do the same thing for the custom loader.cxi and you get:

$ ctrtool -t ncch -i out/corbenik/module/loader.cxi | grep 'Content size'
Content size:           0x00003200

Now, I downloaded the 11.0 firmware, let corbenik decrypt it and extracted the loader:

$ ctrtool -t ncch -i 4013000001302.ncch | grep 'Content size'
Content size:           0x00003200

See a pattern? The 11.0 loader is one unit larger than on 10.7 (10.2) NATIVE_FIRM. Nintendo actually provided us with 0x200b more to work with when they shipped 11.0. TIL.

So a possible solution would be for me to finally upgrade and for you to state that with <11.0, the custom loader won't work.

PS: I really should've been doing homework instead of fiddling with bytes. Back to Algebra.

[ghost]

Well, he does recommend using FIRM 0x52/0x21, and that's what I've been testing with. There's currently no reason not to use it (except NTR maybe)

So, moving the loader NCCH around the FIRM section 0 might be the solution.

[chaoskagami]

I'll leave it open and change the title. I need to fix the expansion anyways, so until then I'll leave it open. It'll also work well as a notice to anyone running into this.

[phijor]

Do you perhaps have any ressources where one can find out about how things should be loaded into memory? If I know what makes expansion more difficult than contraction I might be able to come up with fix for it.

[chaoskagami]

It's not all that hard to fix, IMO, but because we do things cakes style we need to do it in-place. It isn't something I had run into yet, so I just didn't bother.

Every offset/every size after the sysmodule section has to be changed to account for the number of mediaunits it was grown, and also adjust headers, etc. This also involves memmove to actually make space to inject. We need to keep the firmware contiguous in memory, unlike Luma (it reloads and repatches on reboot, we do not.)

As an aside, once this is fixed, it should probably also be submitted to Cakes in source/patch.c, since that's where the sysmodule injection code originates from.

[chaoskagami]

Assigning myself, since I'm fixing this now. Better to just get it done.

[chaoskagami]

It looks like older NATIVE_FIRM has been removed from the CDN, by the way.

[phijor]

Well, 0000001B and 0000001F still seem to be available.

[chaoskagami]

Interestingly, when I tried to download 0000001F a minute ago, it failed. I must have typo'd.

Anyways, I have a few things to squash first (master definitely doesn't work right at this moment.)

[chaoskagami]

It looks like the size of loader was actually updated in 10.4, so I'm changing the title. 1F is fine. It's actually anything older.

[chaoskagami]

I've pushed a test (which works during regular boot) but I have no clue whether it remains valid during a firmlaunch. Native firmlaunch should only happen on o3ds, anyways, so care to test this @Wolfvak ?

You'll want to use 10.2 FIRM.

ctrtool finds the cache firmware (cache/p_native) valid, so I assume all is well, but it needs testing.

[chaoskagami]

I've tested using the cache firmware as native (with patches off) and it seems to work, so for now I'll be closing this issue unless anyone finds anything severely wrong with the expansion code.