Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build Guide #17

Open
Axis4s opened this issue May 11, 2024 · 46 comments
Open

Build Guide #17

Axis4s opened this issue May 11, 2024 · 46 comments

Comments

@Axis4s
Copy link

Axis4s commented May 11, 2024

Would be helpful!

Trying to build it from the vague guide that is available

@agilly1989
Copy link

I got it to build on Windows using the Visual Studio (community) C++ tools. Get this
image
and then open the developer command prompt
image
and you should be ok from there.

cmake .
cmake --build .

The issue is, you need the game in an "elf" format, and I haven't been able to find/convert anything into it

@ExtCan
Copy link

ExtCan commented May 11, 2024

The Wii uses elf files, right? Could a game converted to be a Wiiware retro console title and then used?

@Mode8fx
Copy link

Mode8fx commented May 11, 2024

You can also use MSYS2 on Windows. If I remember correctly, clone the repo with git clone --recurse-submodules as specified (that part can be done in a normal command line), then in MSYS2 run:

  • cd "N64Recomp directory"
  • cmake "N64Recomp directory"
  • cmake --build .

The issue is, you need the game in an "elf" format, and I haven't been able to find/convert anything into it

I would also like to know how to do this.

The Wii uses elf files, right? Could a game converted to be a Wiiware retro console title and then used?

ELF is basically just a standard executable extension that many systems use; they're not actually the same, similar to how Wii WADs and Doom WADs aren't the same.

@CaveHermit64
Copy link

As far as I can tell, you need to decompile an N64 ROM in order to get an ELF file for it.

I was able to generate the ELF file for Duke Nukem Zero Hour by using this Decompilation project for the game: https://github.com/Gillou68310/DukeNukemZeroHour

You also need to figure out the memory address for a given game's program "entrypoint" however, and include that in the TOML file (This page might be useful: https://n64brew.dev/wiki/CIC-NUS#Variants). In the case of Duke Nukem Zero Hour, it seemed to use a nonstandard entrypoint, but I put in 0x80003940 as a guess (I chose the memory address referenced in the main function at the bottom of main.c in the decompiled source), and that seemed to work.

@Poyozito
Copy link

As far as I can tell, you need to decompile an N64 ROM in order to get an ELF file for it.

I was able to generate the ELF file for Duke Nukem Zero Hour by using this Decompilation project for the game: https://github.com/Gillou68310/DukeNukemZeroHour

You also need to figure out the memory address for a given game's program "entrypoint" however, and include that in the TOML file (This page might be useful: https://n64brew.dev/wiki/CIC-NUS#Variants). In the case of Duke Nukem Zero Hour, it seemed to use a nonstandard entrypoint, but I put in 0x80003940 as a guess (I chose the memory address referenced in the main function at the bottom of main.c in the decompiled source), and that seemed to work.

a little question, after you found the entrypoint, how exactly did you generated the elf file?

@CaveHermit64
Copy link

a little question, after you found the entrypoint, how exactly did you generated the elf file?

I followed the instructions in that Duke Nukem decomp repo under "BUILDING", and the ELF file was generated along the other files in the build directory.

Truth be told, I'm still not entirely sure what these ELF files are in the context of N64 games, but the readme for this Recomp repo said that decompilation was the easiest way to get an ELF file, and I happened to know of the decomp project for Zero Hour, so I gave it a shot.

For the record I couldn't figure out how to actually use the output of the Recomp binary. After supplying it with the ELF and entrypoint address it generated a bunch of c files, and I was able to generate object/o files from these by including the "include" directory from the Zelda64Recomp repo with GCC, but I couldn't figure out how to actually link the object files to an executable. I figure it might have something to do with the lib directory submodules from the Zelda64Recomp repo, but I didn't have any luck. It could be that there was an earlier entrypoint address than the one I supplied, but in any case I'm giving up for now.

@Mode8fx
Copy link

Mode8fx commented May 11, 2024

As far as I can tell, you need to decompile an N64 ROM in order to get an ELF file for it.
I was able to generate the ELF file for Duke Nukem Zero Hour by using this Decompilation project for the game: https://github.com/Gillou68310/DukeNukemZeroHour
You also need to figure out the memory address for a given game's program "entrypoint" however, and include that in the TOML file (This page might be useful: https://n64brew.dev/wiki/CIC-NUS#Variants). In the case of Duke Nukem Zero Hour, it seemed to use a nonstandard entrypoint, but I put in 0x80003940 as a guess (I chose the memory address referenced in the main function at the bottom of main.c in the decompiled source), and that seemed to work.

a little question, after you found the entrypoint, how exactly did you generated the elf file?

Compiling an N64 rom with this method also compiles an ELF alongside it.

CaveHermit [edit, I put the wrong name], what did you use for the TOML? Here's what I tried:

[input]
entrypoint = 0x80003940
elf_path = "dukenukemzerohour.elf"
output_func_path = "RecompiledFuncs"

... but I'm getting a bunch of Potential jal resolution ambiguity warnings and an error indicating No function found for jal target: 0x80000000.

@CaveHermit64
Copy link

CaveHermit [edit, I put the wrong name], what did you use for the TOML? Here's what I tried:

[input]
entrypoint = 0x80003940
elf_path = "dukenukemzerohour.elf"
output_func_path = "RecompiledFuncs"

... but I'm getting a bunch of Potential jal resolution ambiguity warnings and an error indicating No function found for jal target: 0x80000000.

That looks about the same as what I wrote for my TOML file, although make sure that you have the correct relative paths for elf_path and output_func_path (The way you have it written, it'll only work if the elf file is right next to the Recomp binary).

Did you get any c files in your output_func_path directory? I think I remember my output having "jal resolution ambiguities". but I'm not entirely sure.

@Fondant
Copy link

Fondant commented May 11, 2024

just tried this (with paper mario 64), and yes it did have the resolution ambiguity and produce a recompiledFuncs full of c files

@NelsonL19
Copy link

Just to add onto the discussion here, I ran across a bash script here that is said to convert .z64 to .elf here. While using the .elf file generated from the script doesn't work with the built executable for recompilation, by opening the .elf in a text editor, you get the same entry point for the ROM as you would by opening it in Ghidra (as I did with Pilotwings 64).

@Mode8fx
Copy link

Mode8fx commented May 11, 2024

That looks about the same as what I wrote for my TOML file, although make sure that you have the correct relative paths for elf_path and output_func_path (The way you have it written, it'll only work if the elf file is right next to the Recomp binary).

Did you get any c files in your output_func_path directory? I think I remember my output having "jal resolution ambiguities". but I'm not entirely sure.

My TOML is indeed next to the Recomp binary, so it's fine. I have 1,362 C files plus three other files in my output_func_path, but still running into those warnings and that error.

Just to add onto the discussion here, I ran across a bash script here that is said to convert .z64 to .elf here. While using the .elf file generated from the script doesn't work with the built executable for recompilation, by opening the .elf in a text editor, you get the same entry point for the ROM as you would by opening it in Ghidra (as I did with Pilotwings 64).

I found that same script earlier but couldn't get it to work (I probably just don't have the right mips toolchain or something). Could you just use something like Ghidra to generate an ELF for any rom?

@andrewCohn
Copy link

andrewCohn commented May 11, 2024

Could you just use something like Ghidra to generate an ELF for any rom?

Funny you should mention this, I'm experimenting with exactly this right now. Here is a library for using n64 roms in Ghidra: https://github.com/zeroKilo/N64LoaderWV

I'm poking around, seeing if I can find a way to reverse engineer the entry point for some arbitrary rom. Ill report back with any updates if I make any progress.

@agilly1989
Copy link

I did find this https://gist.github.com/JayFoxRox/5eaa45b374a44ce27143388bab1753b7 in my searching.... but I can't get it to work. A better option will turn up sooner or later so I am putting this on the back-burner for now.

@Axis4s
Copy link
Author

Axis4s commented May 11, 2024

ooh, interesting
im trying to recompile mario tennis to see what would happen

and i see, i need this .elf from a game in order to have the recompile program to work?

@Fondant
Copy link

Fondant commented May 11, 2024

incase anyone needs it here is what i could find for a base toml config file:

#Config file for

[input]
entrypoint = 0x
# Paths are relative to the location of this config file.
elf_path = ""
output_func_path = "RecompiledFuncs"

manual_funcs = [
]

[patches]
stubs = [
]

ignored = [
]

# Hooks

# Single-instruction patches

@Axis4s
Copy link
Author

Axis4s commented May 11, 2024

incase anyone needs it here is what i could find for a base toml config file:

#Config file for

[input]
entrypoint = 0x
# Paths are relative to the location of this config file.
elf_path = ""
output_func_path = "RecompiledFuncs"

manual_funcs = [
]

[patches]
stubs = [
]

ignored = [
]

# Hooks

# Single-instruction patches

interesting so i just need a entry point value and the elf path then the tool will generate the recompiled game?

@NelsonL19
Copy link

NelsonL19 commented May 11, 2024

interesting so i just need a entry point value and the elf path then the tool will generate the recompiled game?

That's consistent with what was shown in this video as well the example toml file in the Majora's Mask repo, so yeah that's all we should need

I do honestly believe that Ghidra and similar tools will probably be our best bet for figuring out a way to generate an elf file for games that don't have decompilation projects yet

@Fondant
Copy link

Fondant commented May 11, 2024

That's consistent with what was shown in this video, so yeah that's all we should need

at 2:27 if anyone is interested, shows Rocket: Robot on Wheels being converted

@agilly1989
Copy link

https://n64decomp.com/basics/rom/rom_image_specification.html might help somewhere

@Fondant
Copy link

Fondant commented May 12, 2024

Depending on the used CIC variant (CIC Specification) certain modifications to Entrypoint will be done before giving control flow to the ROM’s BOOT segment.

click on CIC Specification get 'to do' lol

@agilly1989
Copy link

You know what might work.... the Ares emulator https://github.com/ares-emulator/ares/tree/master/ares/n64/cartridge

An extra function to dump an elf file from that could work.

I honestly don't know, just throwing ideas in the air.

@Mode8fx
Copy link

Mode8fx commented May 12, 2024

You know what might work.... the Ares emulator https://github.com/ares-emulator/ares/tree/master/ares/n64/cartridge

An extra function to dump an elf file from that could work.

I honestly don't know, just throwing ideas in the air.

Is that an actual feature of Ares, or are you speaking hypothetically?

@agilly1989
Copy link

You know what might work.... the Ares emulator https://github.com/ares-emulator/ares/tree/master/ares/n64/cartridge
An extra function to dump an elf file from that could work.
I honestly don't know, just throwing ideas in the air.

Is that an actual feature of Ares, or are you speaking hypothetically?

hypothetically, I am just compiling information for people who are smarter than I to be able to do something... Currently looking at https://github.com/HerbieHSSO/PyN64 but it makes little sense to me.

@Chrs2324
Copy link

I mentioned this in my issues thread but gonna send it here too for everyone else because it is important:

I was able to generate an elf file with this bash script https://gist.github.com/JayFoxRox/5eaa45b374a44ce27143388bab1753b7
What you have to do in order to make this bash script work is you have to install n64chain in order for it to correctly generate the elf file https://github.com/tj90241/n64chain. However, I have hit a roadblock. While the entry point address needed to generate the elf file is correct, the N64Recomp program still spits out an error when feeding in the toml and elf files. The game that I am using to test this is Conker's Bad Fur Day. Hopefully this helps people get further along with figuring this out, but until now, this issue is still not resolved.

@Kenshin9977
Copy link

If you ever need to convert to any of the 3 existing format of N64 roms you can use Tool 64 https://www.zophar.net/utilities/n64aud/tool-n64.html
There is this short explaination about N64 rom formats, tldr it's all about endianess https://www.reddit.com/r/emulation/comments/7hrvzp/the_three_different_n64_rom_formats_explained_for/

@Mr-Wiseguy
Copy link
Member

Apologies for not having this out yet, but I will be creating a minimal recomp example in the near future that should help clear up how to use the tool. The main requirement is that you have an elf file (which can be generated from a disassembly or decompilation) or a symbol file and the ROM.

@agentduckman12
Copy link

i did find this to convert a z64 file to elf https://gist.github.com/JayFoxRox/5eaa45b374a44ce27143388bab1753b7

@Mr-Wiseguy
Copy link
Member

That elf won't work here, it just packages the ROM up into an elf without providing any symbols. The recompiler needs to know where the code is in the ROM, which is what the symbols in elf are used for.

@agentduckman12
Copy link

oh ok

@Kenshin9977
Copy link

Apologies for not having this out yet, but I will be creating a minimal recomp example in the near future that should help clear up how to use the tool. The main requirement is that you have an elf file (which can be generated from a disassembly or decompilation) or a symbol file and the ROM.

I guess that a lot of people (myself included) come from this video which gives the impression that the tool only needs a ROM. It is my understanding, and it seems to be what you're saying, that this tool won't bypass the decompilation step which still has to be done manually as it need an elf file that you get through a manual decompilation.
Or the symbol file (supported in the future) simplifies a lot the reverse engineering needed to the point where we can say it's not neeeded anymore to have a decompiled version of an N64 game?

@Mr-Wiseguy
Copy link
Member

The work needed to produce a viable elf file for a recomp is only days worth of effort with modern N64 reverse engineering tools, such as splat. You don't need to decompile any code; a disassembly works fine. If you know how to use the reverse engineering tools then it's practically automated nowadays with how good all the tooling is. That's why Nerrel mentioned that it took me a couple of days each to get MM, Rocket and Superman running initially. With that said, it still requires knowledge of reverse engineering as they (and this recompiler itself) are very technical tools.

The symbol file mechanism I'm adding will let you throw a ROM directly at the program without an elf, but the symbol file will still require a similar amount of reverse engineering to get running. I wouldn't be surprised if somebody makes a tool in the future to generate a symbol file mostly automatically from a ROM, which would reduce that already small amount of work required even further.

@thatguy4194
Copy link

Sorry if this is redundant or completely wrong, I'm new to N64 modding, but I really enjoy emulation and game preservation, and this project absolutely fascinated me. Does this this create a sufficient structure of symbols to create an adequate ELF file, or would there still be more work to be done? If not, can you link to a resource explaining how in a future README? I ask because I think a lot of people are attracted to this idea, and it would be a shame to have them be turned off by this roadblock so quickly. Thank you for making such a great advancement in the space.

@Mr-Wiseguy
Copy link
Member

It looks like that only searches for specific symbols, rather than generating a list for all the code in the ROM, so no. Depending on how well it works it would be a good start, but you'd still need to find where all the code in the ROM is besides what this tool looks for.

I'm not sure I could write instructions on how to learn reverse engineering in general (and for N64 specifically), given the scope of what that'd have to cover. It took me a while to learn what I know about it, which I got from working on ROMhacks and helping with decompilations (among other things), so I wouldn't really know how to encapsulate all that knowledge into a guide. This tool is mostly focused on reducing the effort and time needed, but like I mentioned it's still a very technical process.

There are communities where people do N64 reverse engineering that do exist, but given how popular this project has gotten (a fact I couldn't be happier about), I'd be worried about overwhelming them with how many people would end up joining if I linked them directly. As I mentioned, I could imagine that people may make other tooling to make the recompilation process easier in the future. If that happens, I'll be sure to link them in the description of this tool.

@justinsnes
Copy link

For those building in Windows, I was having this error when attempting to run cmake . via the command prompt:
Running 'nmake' '-?' failed with: no such file or directory
To be completely verbose, running the following commands after cloning the repository got it working for me:

  • git submodule update --init --recursive
  • cmake -S . -B Build
  • cd Build
  • cmake --build .

@C2PO-Production
Copy link

C2PO-Production commented May 16, 2024

Personally, the command cmake --build . gives me an error in my Windows command prompt,
but with the command cmake --build . --config Release, it works correctly.

EDIT: ----------------------------------------------------------------------------

It seems that I have successfully retrieved the ELF file of Conker’s Bad Fur Day using the “convert.sh” tool. It’s important to note that in order to make the tool work, you need to get "n64chain" and the header.bin file (to be placed in the libn64 folder of n64chain). You can find the header.bin file on libdragon (I won’t say more as it’s apparently a sensitive file).

I had to switch to Ubuntu on VMware and get help from Copilot to succeed in doing this.
Also, I had to slightly modify the convert.sh file for replacing

mips-elf-objcopy by mips64-elf-objcopy
and
mips-elf-objdump by mips64-elf-objdump

It is also necessary to put these files (mips64-elf-objcopy and mips64-elf-objdump) in the PATH for them to work properly with the command export PATH=$PATH:/home/YOURPROFIL/n64chain/tools/bin/

I hope I have helped you, I will continue my research on my side.

C2PO-elf

You will notice that I have the same problem as some others, the SYMBOL TABLE start, end, and size are all numbered at 00000000, yet the ELF file is exactly the size of my ROM. Consequently… it is impossible for me to recompile it.

@JayroJones
Copy link

I got it to build on Windows using the Visual Studio (community) C++ tools.
and you should be ok from there.

cmake . cmake --build .

The issue is, you need the game in an "elf" format, and I haven't been able to find/convert anything into it

I get 4 error popups after installing it with that C++ toolset. I'm on Win10 x64 Pro, if that matters. Msys2 and Cygwin won't compile this thing either. Can someone just share the compiled binary file already? This is annoying as shit.

@MonkeSmart
Copy link

Oh, man, reading all this I have a question - is it theoretically possible to recompile a PS4 game like this?

@Mode8fx
Copy link

Mode8fx commented May 20, 2024

Oh, man, reading all this I have a question - is it theoretically possible to recompile a PS4 game like this?

The only way you’re getting a PC port of Bloodborne is if Sony makes it themselves or someone leaks the source code.

@MonkeSmart
Copy link

The only way you’re getting a PC port of Bloodborne is if Sony makes it themselves or someone leaks the source code.

Not true. There is a potential emulator in development (though it seems like there is a possibility that the dude who develops it was already contacted by Sony and paid to not develop it LMAO), and, theoretically, somebody could actually reverse-engineer the source code, like modders do. I just asked the person who created the "Tool to statically recompile N64 games into native executables" if this is possible to do with PS4 as it was with N64.

@CCIGAMES
Copy link

i need a makefile, it wont work without one?

@knardhocks
Copy link

Please make a video or something going through the process.
Right now I've deep dived into a lot over the past week and hit the ELF roadblock that everyone else seems to be hitting.

These are the steps I've come to understand needs to happen from scratch:

Git Cloned N64Recomp
Cmake the git clone
Built via Visual Studio
Working EXE awaits input.

*Digging around for methods to generate the ELF
*Used splat on z64 for yaml + asm / assets / etc - This gets some decent data to work with from the z64

Step 2 -> ????
Step 3 -> Generate ELF
Now also reading some methods people are using is not properly compiling the right symbols / types / etc, so the ELF becomes useless and there's more involvement in that.
Step 4 -> Correctly generate an accurate ELF ?
Step 5 -> Plug toml and ELF into EXE
Step 6 -> Feel smart for great success

@ethteck
Copy link

ethteck commented May 22, 2024

This project isn't at the point where a checklist like that is going to work. Like wiseguy said earlier, this process requires knowledge of several tools and how n64 games are structured. Hopefully at some point, various components of the pipeline can be streamlined. But right now, you have to really understand a lot of the inner workings of things

@knardhocks
Copy link

This project isn't at the point where a checklist like that is going to work. Like wiseguy said earlier, this process requires knowledge of several tools and how n64 games are structured. Hopefully at some point, various components of the pipeline can be streamlined. But right now, you have to really understand a lot of the inner workings of things

Guess I'll have to deep dive into reverse engineering resources / documentation then.

@Makaque
Copy link

Makaque commented May 24, 2024

Just throwing in my experience so far. I was able to get my ROM in an elf format using the existing WIP Pokemon Snap decomp project https://github.com/ethteck/pokemonsnap.
Found the entry point and wrote a toml settings file.
Ran N64Recomp and generated C code files for the game.
Now I'm in the process of trying to figure out how to compile this into an executable. I'm digging through the Zelda64Recomp project and trying to harvest the necessary bits of code from there. Unfortunately I haven't touched C in a long time and have no experience with CMake or MSVS. I'm a bit confused at this point though because it looks like Zelda64Recomp doesn't include source files for Zelda, it recompiles the source on build when you supply an elf, is that right? So is it more like a specific use case version of the N64Recomp, but with UI and patches and such? If that's the case, what do I do with my generated Pokemon Snap C code? I think I'm misunderstanding something here.

@ImanCol
Copy link

ImanCol commented Sep 14, 2024

This project is fascinating, but there are many doubts and a lack of documentation. Let's hope there are more concrete answers. What I want to know is how to link the graphical API, or how this process is done. I kind of understand the .elf thing, and it won't always work because there are games that use overlays.

@ImanCol
Copy link

ImanCol commented Sep 14, 2024

I mentioned this in my issues thread but gonna send it here too for everyone else because it is important:

I was able to generate an elf file with this bash script https://gist.github.com/JayFoxRox/5eaa45b374a44ce27143388bab1753b7 What you have to do in order to make this bash script work is you have to install n64chain in order for it to correctly generate the elf file https://github.com/tj90241/n64chain. However, I have hit a roadblock. While the entry point address needed to generate the elf file is correct, the N64Recomp program still spits out an error when feeding in the toml and elf files. The game that I am using to test this is Conker's Bad Fur Day. Hopefully this helps people get further along with figuring this out, but until now, this issue is still not resolved.

If you are still playing this game (Conker BFD) I would like to know if I can contact you, I am currently making some mods

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests