Indiana Jones/Battle of Naboo microcode Implementation

[Frank-74]

Any chance of work being done to HLE this game?

I've made a fixed branch of Project64 that has an RDB option hack for Indiana Jones and Battle for Naboo here: https://github.com/Frank-74/project64/tree/ForceFpuLoc

This build has fixes for savestates as well, so both old and new types work, compressed or uncompressed.
Compiled build with VS2008 SP1 here: https://dl.dropboxusercontent.com/u/9498358/n64/Project64FixFpuLoc.zip

GLideN64 just hangs with a black screen, whereas Glide64 responds with:
Error: Unsupported uCode!
crc: d5c4dc96.

Jabo 1.7 just crashes the emulator, but 1.6.1 gives this:
Unable to detect microcode settings
Microcode ID: $8A4F88F1.

LLE is fine, but it's slow.

[gonetz]

Only if somebody will decode that microcode.

[Papermanzero]

Same applies fro the factor 5 games like rogue squadron.

[Frank-74]

Maybe microcode for, Battle for Naboo, Rogue Squadron, Indiana Jones will be quite similar if not the same.
Indiana Jones is quite a good adventure when you get into it. Even with the odd bugs.

[gonetz]

I looked at Rogue Squadron implementation in Lemmy plugin. Very specific and unique thingy.

[olivieryuyu]

@Frank-74
Can you rename this ticket Indiana Jones/Battle of Naboo microcode Implementation please?

For your information, I started some prelimilary work on those games. They are significant different than SWRG but Indy and Battle of Naboo are quite the same (some differences exists).

For the time being, I am hopeful it can be done with a lot of efforts.

[StenApp]

I'd like to help here, but except for the n64 Manuals (http://ultra64.ca/resouorces/documentation/)
i have no infos what to do.
How can i dump ucode? Nemu doesn't help here...

[olivieryuyu]

This ucode is really complex. For those interested, there is an old article about the game here:

http://www.ign.com/articles/2000/11/10/bringing-indy-to-n64

A good part of the ucode was rewritten but it was not totally "rewritten from scratch".

Few features in this ucode are very unique. Interesting but challenging.

[StenApp]

I dumped some ucode from battle for naboo/indy and compared it with rogue squadron.
One part was the same for all, and one is the same like in BfN and Indy.
RSP code indy.txt
RSP code indy2.txt
RSP codebfn2.txt
RSP codebfn11.txt
RSP coders1.txt

[StenApp]

@olivieryuyu Can you provide any useful information on how i can help with decoding?
How to start out?

[olivieryuyu]

@StenApp

I already deciphered a big portion of the microcode.

Thanks for proposal though!

[StenApp]

@olivieryuyu sounds great, thank you. Can you give a short status? Blog entry maybe?

[olivieryuyu]

Update will be shared where possible. Stay tuned.

[weinerschnitzel]

@olivieryuyu, i would love to read a bit more about your adventures with reverse engineering. I understand as far as dumping the microcode and translating MIPS assembly to x86 or C, but im sure you've learned some tricks to identify certain patterns. Would be cool to see a video or guide that tells someone what they need to get the job done

[olivieryuyu]

a really crazy ucode...

Do you believe that the menu are simple sprites (texture rectangles)???

Nope!!! They are triangles!!!!!!

Screenshot from Angrylion plugin

[AmbientMalice]

I assume that was in order to achieve the "flipping page" transitions without resorting to framebuffer tricks.

[gonetz]

We started new crowdfunding campaign for Indi and Naboo on Idiegogo:

https://igg.me/at/indihle

Visit it to see what is already done and what is planned to be done when the campaign will reach the goal. Any kind of support is welcome.

[StenApp]

Supported of course! Like to see "unknown ucode" message vanished.

[StenApp]

How about to post a reddit entry?
I'm to new there so my post got deleted.

[olivieryuyu]

I have no reddit.

The campaing does not seem to be successful, that is a bit unfortunate...

It is a hard work to do the reverse engineering and the microcode of Factor 5 are really huge and complex compared to the others.

For instance Indiana Jones/Battle of Naboo is double the size of F3DEX2, which was the biggest Nintendo microcode.

[oddMLan]

@StenApp

It probably got deleted because there's already one for the campaign: https://www.reddit.com/r/emulation/comments/7ogzuc/gliden64_indiana_jones_and_the_infernal_machine/?utm_source=amp&utm_medium=comment_header

[StenApp]

@oddMLan
I think it was my first post on reddit, I would have needed to post or comment things before.

@olivieryuyu
I'm not sure why so less people support your decoding, maybe many think they get it anyway or they don't care, because of the age of N64.
It's sad because Factor5 games are the last games to get native.
I'd like to see in the end of decoding how you did that in form of technical blog or documentation.
None the less: respect for your work.

[weinerschnitzel]

I am sure there are others that will donate more before the campaign is over. And the campaign is very modest, it will be filled.

Although, yes, i do think this should drive more attention.

[tony971]

It might be useful to make sure that people realize that the dev is not Gonetz, who has a running patreon for GLideN64.

[olivieryuyu]

It is a campaing of both Sergey and me, the work on the HLE project being done together.

After Indy, there is still many things in HLE to do:

Batte of Naboo specific commands
HVQ microcode (if really need)
Fog issue in Zelda OOT
S2DEX issues in various games
Dark Rift issues

Of course this campain for Indy would be the last one for supporting HLE reverse engineering in GlideN64. But seeing the little attention it brings, it is a bit demotivating :(

[weinerschnitzel]

@olivieryuyu what is your input for the 64DD games? Certainly those will need some work, too. Campaign is funded so get it done please! :)

[StenApp]

mhhh one spend anonymously the rest to fill the 600 Dollar, it's the second time one did.

[olivieryuyu]

@weinerschnitzel. Thanks! 64DD games uses standard ucodes, I already checked out.

@StenApp : at the end of my HLE adventures (many months), I will do a small introduction on how I do the reverse engineering. Yet the first step is to understand MIPS (https://chortle.ccsu.edu/AssemblyTutorial/index.html)

[StenApp]

@olivieryuyu
When I look at my dumps of Indy and BFN I see what you mean with double the size as SDEX2 code.
One started with 0x000 J 0x064 and one with 0x000 ORI V1, R0, 0x0000.

[olivieryuyu]

It should be checked on real hardware first. The game is weird on this aspect.

[gonetz]

This glitch does not appear with angrylion's plugin. Most likely it is a problem with my mip-mapping shader.

[olivieryuyu]

Ok then i guess it should be a new ticket for this issue.

[gonetz]

Done: #1833

[GamingTVYT]

Hello everyone, I found another glitch in (Star Wars BFN) now in the screen selection menu, I recorded a video showing the glitch. https://www.youtube.com/watch?v=fFKR-Im8cNo
SaveState if you want to check.
Star Wars Episode I - Battle for Naboo (U) [!].zip

[fzurita]

It's a core issue. Set counter factor/Count per op to 1 fix the menus.

[GamingTVYT]

@fzurita Corrected, thank you. 👍

[gonetz]

@Diduz Could you give me savestate for Indiana from the same place but for Project64?

[Diduz]

@gonetz I'm afraid I'm not able to get the game working under Project64. I get a black screen with the NTSC version. The unreleased PAL version starts, but it gets garbled after some screens. I'm using Project64 2.3.2.202.

[oddMLan]

@Diduz

Indy/Naboo do not work under PJ64 2.3.x. Use Project 64 2.4.x.

You can get nightly builds at pj64-emu.com or emucr.

[olivieryuyu]

here the savestate

Indiana Jones and the Infernal Machine (E) (Unreleased).pj.zip

It seems incorrect in both LLE and HLE.

[gonetz]

@olivieryuyu Thanks!

[gonetz]

I fixed the issue with mip-mapping. I noticed, that Project64 still have issues with this game: there are problems with shading, both in HLE and LLE. mupen64plus works just fine.

New beta build:
https://drive.google.com/file/d/10FoCieiX68XCij5Uw1pOx9J_P4Tbf91u/view?usp=sharing

[Diduz]

@gonetz Works like a charm! :-) Indy seems perfect to me now. Thanks for all your work!

[nterry]

@gonetz @olivieryuyu That's too bad... I guess my ignorance on this topic is pretty clear. Id be fine to dump the microcode myself, but I have no clue how/where to start... I DO own a 3.2 gameshark and an ultrasave cart tool from http://64drive.retroactive.be/features.php#ultrasave, so I could, in theory, do it myself. I am more than willing to learn and am well aware of the cliff-face level of an uphill battle I face...

That said, I am willing to go the distance. Can you point me in the right direction on some instructions, docs, etc to do this? Also, does loading a custom microcode change the ABI for the RCP? Or do I still use libultra? If it changes, do you have the ABI that I could code against?

Finally, is it possible to simply inline "compiled" microcode (as in "straight from the rom" in some kind of ascii-like encoded form) in c++ using the correct syscall? Or would I have to reverse engineer and reproduce the assembly and inline it that way?

Hell, Id still be willing to pay for a nice set of documentation/instructions to explain details about all of this if you'd be willing to create it. You clearly have a LOT more knowledge in this particular field of CS than I do. Id be willing to even upfront some of it as a good-faith gesture. Let me know if that would be worthwhile with respect to your time, etc.

[nterry]

If you are really interested to do that, I guess your best chance would be indeed to dump the ucode and check the microcode implementation of Sergei

@olivieryuyu Also, where can i find this Sergei and/or his ucode implementation?

[AmbientMalice]

@nterry here are the commits for the HLE code https://github.com/gonetz/GLideN64/commits/indi_naboo

[gonetz]

@nterry I have no skills in microcode decoding and can't help you.
As I know, the main tool olivieryuyu used is Project64 built-in debugger. Open Settings, select Advanced, check Enable debugger. You will get Debugger menu with plenty of tools. Debugger allows you to debug/dump both CPU and RSP commands.
I don't know, are there any instructions how to work with these tools. May be olivieryuyu will write some manual for ucode reverse engineering, but it will not be soon.

[olivieryuyu]

@nterry

I am afraid that using a microcode would be much more complex than deciphering it (I intend in the future to provide a short tutorial in this respect).

You would need to dump properly the ucode (both Nemu and PJ64 are unable to do it without bugs), then you would need to compile it with the N64 SDK (I would guess, I am not sure, that it does support the RSP asm, which is not at all standard) to get potentially a library usable for compiling a homebrew rom with this ucode. Of course you would need to change the gbi.h to support the related commands (see and understand in depth Sergey implementation in this respect, it is not needed to decipher the ucode as such). Also as written by the devs, the OS of N64 had been modified, most likely to manage differently the segmentation of the memory and potentially to use COP1 for some operations (as the MP matrix calculated on the CPU side).

As you may understand, this is far from being a trivial exercise to do.

[LegendOfDragoon]

You would need to dump properly the ucode (both Nemu and PJ64 are unable to do it without bugs)

What is your method of dumping?

[olivieryuyu]

Mix of both and manual corrections where necessary.

[StenApp]

#1259 (comment)

@olivieryuyu i made some dumps. But you meant they were partly ones. How can i be sure that i got the whole ucode?

[olivieryuyu]

You will need to go through the code and catch all overlays with PJ64. Then compare them with Nemu ucode extraction to get the full ucode dump. Then correct incorrect address (mostly offset) of some asm commands.

Gillou has potentially a way to do at once but I dunno how he does that.

[StenApp]

@Gillou68310 Can you please help us here? Do you have a hint how to get the ucode of a n64 game?
I capture a state with the debugger of nemu or pj64. In an earlier comment you see my dumps:
#1259 (comment)

[Jj0YzL5nvJ]

You will need to go through the code and catch all overlays with PJ64. Then compare them with Nemu ucode extraction to get the full ucode dump. Then correct incorrect address (mostly offset) of some asm commands.

That's more similar to recovering the DNA of a fossil, but without worrying about DNA contamination from third parties and bacteria.

[Gillou68310]

@Gillou68310 Can you please help us here? Do you have a hint how to get the ucode of a n64 game?

Yes I wrote a bit of code to dump the ucode text and data sections directly from rdram provided that TASK_UCODE_SIZE and TASK_UCODE_DATA_SIZE have correct values, otherwise I just dump 4096 bytes.

But you won't be able to build a ROM with just the text and data sections. The "makerom" utility from the SDK requires ucodes in ELF format. A tool like "rsp2elf" could potentially do the job but it's not included in the SDK. Also since the application code can reference the RSP text and data sections, a symbol file will be required.

[gonetz]

Merged to master.