AI5WINArcTool not working properly

[WoodyTheWoodsman]

I'm trying to unpack the MES.arc (which contains the scripts files) from the game Doukyuusei 2 (1997) by elf, and your tool sends an error "cp932 codec can't decode byte 0x9e in position 72: illegal multibyte sequence". Could you take a look? Intersting enough, it works normally with another game by elf, Doukyuusei, which is the first game in the series. However, when i extract the files from MES.arc from Doukyuusei, then convert .mes files to .txt, edit the scripts inside of them, then pack everything back, the edited text appears inside of a game either as dots or gibberish. I don't know if just sending you the MES.arc files from both games would be enough, here's the links where i found them.
https://archive.org/details/elf_doukyuusei
https://archive.org/details/doukyuusei-2
Some person have already patched them to run on modern systems. The download speed is slow however. If you don't want to dowload them, i can send you MES.arc files later.

[TesterTesterov]

Sorry I took too much time, but I finally gotten to look at the archives. Well... Seems it was an unknown schema. These ones have some exotic features too. For example, some sort of garbage(?) (well, changing it does not seem to break the game, at least) in the archive names, which makes it harder for tool to deal with them even if it known the schema. AI5WIN's archives are really nasty with this. Why could not the developers make the normal archives?..

I'll see what I could do.

[TesterTesterov]

@WoodyTheWoodsman, okay, I did some magic and got that to work. Still, there are problems with the engine, you see...
I have extracted the scripts, edited them via AI5WINScriptTool (0 version of the script) and repacked them. Yet the engine (only this old version, you see) seems to have problem with half-width symbols...

Well, it does work with full-width ones, though, and it does not look too bad, so no problem, I guess.

In any way, @WoodyTheWoodsman, check the new version (AIN5WINArcTool 2.2) and write about the results.

[WoodyTheWoodsman]

@TesterTesterov, Sorry for late response, i almost did'nt hope you will answer. I will get to testing as soon as possible.

[WoodyTheWoodsman]

@TesterTesterov, i have used your tools to unpack scripts, then tried to edit them, then pack everything back. The game seems to work normally when i do not edit any text or edit it without using full width symbols, but when i convert the line of text to full width symbols using this - https://dencode.com/string/character-width - then replace the text in the script with the converted line, the game shows an error message window containing only START.MES. When i edit the text using english symbols normally, i get the issue as you described on the first picture.

[TesterTesterov]

@WoodyTheWoodsman, and this is the problem. Basically, as I heard, you need to "edit 21 system variable to 72 or so", but without extensive tests I don't sure how to do that in that particular version. And I do not have time to do that kind of testing now. Still, you may try to try some commands in the script, there are not too many of them. If I may guess, there should be command with first int and the following expression as arguments or something like that.

[WoodyTheWoodsman]

@TesterTesterov, i see. If you do not have the time, can i help you with testing? I'm not a coder though, and the list at the end of AI5WINScriptTool does not cover all of them.

[TesterTesterov]

@WoodyTheWoodsman, indeed, it would be beneficial. If you'll manage to track the command, I'll update data accordingly.

In order to find all the commands, you may do several things...

• Look through that issue.
• Look through that code. You need to find "0" section of "command_library". I wrote there known opcodes of the version. Alas, the names of commands in 0 may be messed up, but still.
• Disassemble all Doukyuusei 2's scripts, look which commands are used and try to insert/edit them before the first message. Perhaps, if you find a correct command and edit the parameters to 21 and 72, you may get the result you desire.

[WoodyTheWoodsman]

@TesterTesterov, alright, i see what i can do.

1. I have looked through the issue and i have noticed that the files were case sensitive even before reading it. I did not use message breaks yet, just replaced a line of text to see if it works.
2. Still looking though the code.
3. Unpack all of them using your tools and look through them? Got it. I'll look through DATA.ARC first, BG, BGM, GIRL and BOY arcs are unlikely to contain the code we need, but who knows.
   Perhaps i know why i have'nt found any attempt to translate Doukyuusei 1 and 2 to english. Probably because of these issues. You are literally the only one who developed some tools to extract scripts from AI5WIN games, except the one instance when a whole team of coders and translators translated PC-98 version of YU-NO and developed specific tools for it (somewhere in 2007, i think?). I do know there is a successfull attempt to translate Doukyuusei 2 to chinese. I have found the chinese version while looking for the one i've sent you, and i still have it. It probably uses half width symbols like the original. Should i try to use your tools on it too and see if it works?

[WoodyTheWoodsman]

@TesterTesterov, alright, i don't understand. I successfully used your tools on chinese version. While inside the game everything is chinese, the text in the scripts is japanese. And i can tell where is japanese, and where is chinese, even though i don't understand chinese. This is weird.

[WoodyTheWoodsman]

@TesterTesterov, interesting. I have found 2 arcs in the chinese version that the original one does not have - CHS (means chinese?) and HDATA. Their contents are seems to be identcal to MES and DATA arcs (CHS to MES and HDATA to DATA), however, while AI5WINARC tool unpacked them properly, AI5WIN script tool does not properly process .mes files found in CHS and sends an error Object of type bytes is not JSON serialiable. However, it still gives partially decompiled .txt script.

[TesterTesterov]

@WoodyTheWoodsman, you need only to extract and disassemble all the MES. You won't find any relevant code in different places, as I presume.

As for Chinese version... Perhaps, Chinese technicians did not actually hack the code, but made something to replace the strings. Or maybe they have rebuilt the code based on the original scripts and made it like a patch. Anyway, they made much more technical work. And I do not think their version is relevant, as Chinese use full-width characters.

Still, probably my tool could work with Chinese files. Just change "cp932" on "cp936" within the Python code of it.

[WoodyTheWoodsman]

@TesterTesterov, i see. All arcs except DATA and MES contain only sound (.wav) files. What i found in DATA is a DEFMAIN.MES file, and it's the only .mes in there. I've noticed that START.MES mentions it. I'll start looking in there, then all .mes that does not have any character names in them.
Have you tried disassembling AI5WIN.exe?
As for Chinese version, i'll do it later. You may be right, as the Chinese version contain few more files that the original one does not have.

[TesterTesterov]

Disassembling AI5WIN.exe? Well, technically I could do it with IDA Pro, but it would de facto a reverse engineering... It would be far from MES scripts for sure, as it is an executable file. If I to reverse engineer and test it long enough, I would be able find the necessary command, but it would be quite long. As I said earlier, currently I do not have enough time at my hands.

[WoodyTheWoodsman]

@TesterTesterov, i understand. I could try it myself, but i've never reverse-engineered anything. Can it be done with Ghidra?
In fact, i'm not sure what i'm looking for. Are you sertain that "system variable 21" is in .mes scripts? I've looked at your code for the list of commands and searching SYS in decompiled scripts gives nothing yet.

[TesterTesterov]

@WoodyTheWoodsman, Ghidra? Perhaps... I haven't work with it, so I cannot be sure. As for system variable 21, this is not my data. I would've just analyze text output functions first.

Oh, and if you are novice, you may, perhaps, find HexRays on IDA Pro very helpful.

[WoodyTheWoodsman]

@TesterTesterov, looking through it. I wonder if tomyun is still available, he could probably help. Trying to look for the name of array of system variables. Other than various #1-0b, #1-02 nothing comes to mind yet. I could look inside .mes with names comparable to these numbers.

[TesterTesterov]

@WoodyTheWoodsman, the commands' names in 1 and 2 versions are more or less correct, but they are too different from the 0 version, which names are more or less unknown to me. Also, all "names" are only self-defined, so to say, as I do not know the original script behind MES byte-code. This only serve for users convenience, as the byte-code does not contain any signs for the original command names. It contains bytes.

Yes, perhaps he could help. Still, it is possible that only 1 and 2 versions does work that way. It seems like the Doukyuusei 2 just "interprets" only full-width characters, not half-width, unlike other versions. Oh, there is also the heretic "-1 version", which is pain to hack, but I did not work on it, as all games on -1 version has implementations on later versions.

[WoodyTheWoodsman]

@TesterTesterov, i have tried to use full width characters on Doukyuusei 1 - the issue is exactly the same. Tomyun's stuff describes PC-98 versions of these games (which i have - including Kakyuusei and DOS version of Doukyuusei 1). No idea if it can help us with anything, but still.
I don't know how to contact tomyun directly through Github, i guess one of us have to ask around PC-9800 Series Central Discord or his Twitter (which i do not use). Otherwise, i have to wait when you get more free time. I would really like to help, but don't know how. Might look into .exe as i do not know where to look for system variables in .mes scripts. All i noticed is that some (or all of them?) look the same at the beginning of the scripts.

[TesterTesterov]

@WoodyTheWoodsman, well, I do have neither Discord, nor Twitter. But you may write there or just write an Issue in his tool page in Github.

[zsolszesz]

@WoodyTheWoodsman Hey, not sure if you still have problems with this, but I just saw this so I took a quick glance. Unlike the PC-98 version, the setting for this is in the exe instead of the mes file (it was the same with Isaku Renewal).
It looks like this with HexRays Decompiler (in sub_40E070):

You can skip the if check by modifying the following two bytes in a hex editor:
Offset D4A3: 72 -> EB
Offset D4A4: 04 -> 0C

Half-width characters will work fine afterwards:

Keep in mind that the game can skip the string terminators like this, because it still only checks every second byte for that. You would either need to put one more byte in the text where it's necessary, or patch the exe so that it checks every byte, but that will break the full-width Japanese characters.

[JackOfBadTrades]

Hello. I'm having the same issue, but with Kakyuusei 2. It seems that all elf games have this same problem.

Downloaded the game from H-suki and it works well on my machine without the need of an emulator, so it might be the Windows 10 Support edition.
mes.arc file, if you need it.

I would be extremely grateful if manage to get your tool working on this game.