Refactor n64tool, add ed64romconfig binary

[meeq]

Enhanced n64tool with support for long command-line flags, better error messages, and support for 1M ROM size. Removed support for byte-swapped V64 ROM format.

Usage: n64tool [-t <title>] -l <size>B/K/M -h <file> -o <file> <file> [[-s <offset>B/K/M] <file>]*

This program creates an N64 ROM from a header and a list of files,
the first being an Nintendo64 binary and the rest arbitrary data.

Command-line flags:
        -t, --title <title>    Title of ROM (max 20 characters).
        -l, --size <size>      Force ROM output file size to <size> (min 1 megabyte).
        -h, --header <file>    Use <file> as IPL3 header.
        -o, --output <file>    Save output ROM to <file>.
        -s, --offset <offset>  Next file starts at <offset> from top of memory. Offset must be 32-bit aligned.

Size/offset suffix notation:
        B for bytes.
        K for kilobytes.
        M for megabytes.

Created a separate ed64romconfig binary that can be run after n64tool (but before chksum64) that sets the EverDrive64 ROM Header Configuration:

Usage: ed64romconfig [-r] [-c] [-w <savetype>] <file>

This program takes a big-endian N64 ROM and sets the header so that
EverDrive64 will respect the declared save type, RTC, and region-free
settings without needing to create a save_db.txt entry for it.
See: https://github.com/krikzz/ED64/blob/master/docs/rom_config_database.md#developer-override

Command-line flags:
        -w, --savetype <type>  Declare cartridge save type.
        -c, --rtc              Declare real-time clock support.
        -r, --regionfree       Declare region-free ROM.

Supported cartridge save types:
        none        Game does not save or uses Controller Pak.
        eeprom4k    Game saves to 4 kilobit EEPROM.
        eeprom16k   Game saves to 16 kilobit EEPROM.
        sram256k    Game saves to 256 kilobit SRAM
        sram768k    Game saves to 768 kilobit SRAM
        sram1m      Game saves to 1 megabit SRAM
        flashram    Game saves to 1 megabit FlashRAM

Refactored all of the examples Makefiles to follow standard conventions, add comments for special-cases, and use consistent patterns.

Regarding byte-swapping

Premise

Big-endian (.z64) byte ordering is the canonical Nintendo 64 ROM format. Little-endian (.n64) and byte-swapped (.v64) formats are confusing and only exist for historical reasons that are no longer relevant for the overwhelming majority of use-cases. If there is any legitimate reason to prefer publishing a byte-swapped ROM instead of big-endian, I could not find it.

Argument

libdragon has a responsibility to create authentic artifacts that target real Nintendo 64 hardware. Nintendo 64 ROM files are interpreted in a big-endian environment. There should be no question by homebrew developers as to which ROM format to use.

Developers should not be distributing ROMs that require "unmangling" in order to memory-map the data properly. Flash carts and emulators should not have to make assumptions about how to interpret a Nintendo 64 ROM file based on hard-coded magic numbers of the first 4 bytes of the ROM header.

The most developer-friendly approach is to deprecate the .n64 and .v64 formats and strongly encourage everyone to use .z64 going forward.

Counter-Arguments

• "The physical Mask ROM byte order is not big-endian"
  • Response: The intended representation of the ROM data is clearly big-endian:
    • The N64's MIPS CPU is big-endian.
    • The console's memory map and DMA read the cartridge ROM in big-endian.
    • The game title in the ROM header is only readable in big-endian.
    • The official Nintendo SDK emits ROM files in big-endian.
• "What about the theoretical homebrew developer still using a Doctor V64?"
  • Response: If such person exists, they can convert the .z64 to .v64 as part of their build process. When they go to share their work to others, they should publish the .z64 ROM file.
• "Byte-swapped ROM support already existed; keeping it around costs us nothing"
  • Response: The maintenance burden is not zero: n64tool's byte-swap flag was responsible for two subtle easy-to-miss bugs while modifying the code. n64tool does not need the additional complexity of byte-swapping. The feature is completely orthogonal to the other functionality of n64tool. There are plenty of other tools that solve the generalized problem of byte-swapping a file.
• "This could cause breakage for existing workflows"
  • Response: This is the strongest argument against my proposal, and my only hesitation to removing this feature. I do not want to break anyone's build. However, I do want to encourage anyone currently building a byte-swapped ROM to start publishing big-endian (.z64) instead, and I don't think the solution to continue generating V64 is difficult, onerous, or unreasonable. n64tool will suggest the fix if the -b flag is used, so affected developers will know exactly how to embrace this change.

Compromise

The process of building a ROM file should always be in big-endian byte-order. Byte-swapping can be done as a post-processing step after the big-endian ROM file is finalized.

The refactored examples Makefiles still support the N64_BYTE_SWAP flag, but the implementation has changed so that it is now a distinct target that depends on the .z64 build:

ifeq ($(N64_BYTE_SWAP),true)
$(PROG_NAME).v64: $(PROG_NAME).z64
	dd conv=swab if=$^ of=$@
endif

If a developer really wants a .v64 ROM, they still can easily create one, but it will no longer be handled by n64tool.

Backlash-anticipation

I understand that libdragon has its own legacy of using the V64 format, and that some individuals wished to continue supporting the byte-swapped format to prevent churn. I welcome anyone to provide opinions on how to improve this proposal.

[meeq]

The magic rom configuration has been verified on EverDrive hardware. A warning has been added for the case of EEPROM + RTC which is not supported on real hardware (but works on most emulators).

[jago85]

Why do you think the RTC and EEPROM can not work together in real hardware? Did someone test this already? I believe it could be possible. The chips are connected to the same PIF channel. But this is a single wire bi-directional data line. If I remember correctly it is using an open-drain configuration (each chip can only drive the line low). As the chips are using different commands (EEPROM: 0x00, 0x04, 0x05 / RTC: 0x06, 0x07, 0x08), they may work perfectly without any interference. But I never testet it with real chips.

[meeq]

@networkfusion tested this on real hardware with an EverDrive X7 and was only able to get the RTC to work if EEPROM wasn’t used. If both EEPROM and RTC were enabled, EEPROM would be detected but RTC would not.

[jago85]

Ah ok. So, at least the ED64 X7 doesn't support it. And real PCBs with both chips don't exist (to my knowledge). Would be a nice experiment to build one :)

[meeq]

The test ROM in #152 will happily detect both EEPROM and RTC together on emulators. As you said, there's no reason from a software-side that they can't co-exist.

I don't know enough about the hardware to say if there's any sort of conflict in actually building an EEPROM+RTC cart.

I'm curious as to why the EverDrive X7 is implemented this way and if the 64drive hw2 has the same limitation. I'll ask Marshall and see if he has any insights.

[buu342]

I'm curious as to why the EverDrive X7 is implemented this way and if the 64drive hw2 has the same limitation.

If I remember correctly from my testing, it should support both...

[meeq]

Here is the test ROM that uses the header configuration for EEPROM16Kbit + RTC: https://cdn.discordapp.com/attachments/763114214688555039/862140409907380244/ctest_RTCEEPROM16k.zip

And here is that test ROM running on @networkfusion 's EverDrive X7:

[meeq]

Update: EEPROM + RTC works on 64drive hw2

[rasky]

Thanks for this PR, I especially like that command line parsing is streamlined in this PR. My comments also contain suggestions for further cleanups.

[rasky]

Great work! I think this is good to go

[rasky]

This is an unrelated change, I think it should not be part of this PR

[meeq]

It's a minor change; I included it because the example was broken, and I needed to test each example's Makefile. The size parameter of of eepfs_read and eepfs_write was a last-second addition to the PR that introduced this example.

[rasky]

Seems like a convoluted way of writing save_type = SAVETYPE_DISABLED

[meeq]

Oh, that's a relic from before I introduced the #define constants -- you're right that it's a sub-optimal way to set that constant.

[rasky]

It looks like this could get executed multiple times in a loop. Maybe it should instead be detected as a command line syntax error

[meeq]

The if block directly above aborts the command if there are any more arguments that would cause the write file to be opened more than once, but I could make this more obvious

[rasky]

I think in general on UNIX it is more customary to use positive error values for processes. I'd change this and the others to 1.

[rasky]

Do we really need this warning? By looking at the command line syntax, I would say it's pretty clear the intended behavior. If I asked for just RTC, I probably meant it.

[anacierdem]

@meeq feel free to open a new pr for any remaining changes you have.