Translate feature is in both unrpyc branches error prone implemented

[madeddy]

Noting this down to be complete. A task for a bad weather day.
Just out of curiosity i try for some time to figure out how this feature works. What i worked out is, you need apparently two runs:

Basis commands as known: python unrpyc.py /path/to/your/app and now you may add:

1. With arg -T file_name to write a translation file and if you don't want the default english you pass additional arg -l language.
2. Then with arg -t filename the file we did write in 1)

Problems in order:

• py3: I forgot with the pathlib move to cast the translation inputs to pathlike but access them like it.(error) Also confused a filename between both. Dang. I make a pull for this. (edit:Done!)
• (After repairing pathlike in py3) With run 1) we get the translation file with a short string, but its binary/unknown encoding gibberish.
• Feeding this file to run 2) does nothing I'm aware of.

I tested with multiple games and always the same. Examples:
From a v8.1.3 app: py3_tl.txt
From a v7.5.3 app: py2_tl.txt

Oh and i tested even with unrpyc v1.1.8 and got the same result. Maybe i use this wrong?

[CensoredUsername]

With run 1) we get the translation file with a short string, but its binary/unknown encoding gibberish.

You get a file just containing a pickle, so I'm not particularly surprised by that.

I think how the feature works is that it allows you to decompile a game that already has translations, in a different language.

When you first run it with -T it'll gather the contents of any TranslateString and Translate nodes in the specified language and save it in the specified file.

Then when you run it again with -t, it'll decompile the thing, using the stored translations to change the unrpyc output to be in the wanted language.

[madeddy]

Ok. I figured this basically.

with -T it'll gather the contents of any TranslateString and Translate nodes in the specified language and save it in the specified file.

But as can be seen in the attached files, from use of option -T, there are not much in there in terms of strings. I printed the return values of unrpyc.py#L172 out and they where basically empty.

terminal output

olli@tty2*blue $ python unrpyc.py -T /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game//tl/tl.txt -l german /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/ -c
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/room/neus/neus_room_quest.rpyc...
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/story/level3.rpyc...
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/room/milf_rooms/milf_rooms_events.rpyc...
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/room/two_bodies_rooms/two_bodies_mc_room_quest.rpyc...
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/room/mc/mc_room_quest.rpyc...
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/story/level1.rpyc...
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/room/bathroom/bath_room_quest.rpyc...
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/story/extra.rpyc...
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/image.rpyc...
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/room/two_bodies_rooms/two_bodies_rooms.rpyc...
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/room/nym_rooms/nym_rooms.rpyc...
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/gui.rpyc...
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/room/kitchen/kitchen_spell.rpyc...
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/audio.rpyc...
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/ui_game/gamescreens.rpyc...
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/room/neus/neus_spell.rpyc...
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/tl/None/common.rpymc...
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/room/living/living_spell.rpyc...
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/script.rpyc...
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/room/attic/attic_room.rpyc...
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/ui_game/tree_stats_quests/tree_skill_stats_quests_screens.rpyc...
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/room/photo.rpyc...
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/mini_game/fifteen_game.rpyc...
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/init_object.rpyc...
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/mini_game/spanking.rpyc...
extract_translations: .dialogue: {} .strings: {}
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/mini_game/spanking_image.rpyc...
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/style.rpyc...
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/mini_game/puzzle.rpyc...
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/room/rooms.rpyc...
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/story/level0.rpyc...
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/class/kinetic_text_tags.rpyc...
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/room/bathroom/bath_hallway.rpyc...
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/variables.rpyc...
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/options.rpyc...
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/room/neus/neus_hallway.rpyc...
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/story/introduction.rpyc...
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/fx.rpyc...
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/save_compatibility.rpyc...
extract_translations: .dialogue: {} .strings: {}
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/ui_game/tree_stats_quests/skill.rpyc...
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/room/milf_rooms/milf_rooms.rpyc...
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/ui_game/gallery/gallery.rpyc...
extract_translations: .dialogue: {} .strings: {}
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/ui_game/icons.rpyc...
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/class/glitch_ren.rpyc...
extract_translations: .dialogue: {} .strings: {}
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/ui_game/inventory/inventory.rpyc...
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/ui_game/tree_stats_quests/quests.rpyc...
extract_translations: .dialogue: {} .strings: {}
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/room/bathroom/bath_spell.rpyc...
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/ui_game/ctc.rpyc...
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/class/util.rpyc...
extract_translations: .dialogue: {} .strings: {}
extract_translations: .dialogue: {} .strings: {}
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/class/DynamicAnimation.rpyc...
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/class/tree.rpyc...
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/class/room.rpyc...
extract_translations: .dialogue: {} .strings: {}
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/class/glitch_tag.rpyc...
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/character.rpyc...
extract_translations: .dialogue: {} .strings: {}
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/class/inventory.rpyc...
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/class/quest.rpyc...
extract_translations: .dialogue: {} .strings: {}
extract_translations: .dialogue: {} .strings: {}
extract_translations: .dialogue: {} .strings: {}
extract_translations: .dialogue: {} .strings: {}
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/ui_game/notify_personalized.rpyc...
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/story/endings.rpyc...
extract_translations: .dialogue: {} .strings: {}
extract_translations: .dialogue: {} .strings: {}
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/story/level2.rpyc...
extract_translations: .dialogue: {} .strings: {}
extract_translations: .dialogue: {} .strings: {}
extract_translations: .dialogue: {} .strings: {}
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/room/living/living_room_quest.rpyc...
extract_translations: .dialogue: {} .strings: {}
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/screens.rpyc...
extract_translations: .dialogue: {} .strings: {}
Extracting translations from /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/scripts/room/kitchen/kitchen_room_quest.rpyc...
extract_translations: .dialogue: {} .strings: {}
extract_translations: .dialogue: {} .strings: {}
extract_translations: .dialogue: {} .strings: {}
Writing translations to /home/olli/.xlib/RPG/_test/MWNeus-0.9-pc/game/tl/tl.txt...
Decompilation of 61 script files successful

(i put in there just
print(f"extract_translations: .dialogue: {translator.dialogue} .strings: {translator.strings}") )

[CensoredUsername]

Huh. Does this game have translate nodes in it?

[madeddy]

So, your question made me think in a different direction about this whole translation feature and i realized i misunderstood it.
I thought you need to give it with opt -l language a name which isn't already in .../tl/ present to get all the strings from the files which you can translate then. I did this every time in my tests with 7 different games.
However it's reverse. You must choose a language which is already present in tl. In the second run it puts these languages strings in the rpy files. So if i am right this time, this feature just switches the basic language of the game with another. But why do such thing? I can just run the game and switch the l10n in the menu. Makes no sense to me.

To my excuse i oriented myself in my expectation how it works on Ren'Py and his tl feature. Also, this here comes without a manual. 😁

Anyway with one game in py3 i have this error:

olli@tty2*blue $ python /home/olli/Code/rpy_tools/unrpyc/unrpyc-2.0.0/unrpyc.py -T /home/olli/.xlib/RPG/_test/OurBrightDays-0.1.3-pc/game/tl/en_tl.txt /home/olli/.xlib/RPG/_test/OurBrightDays-0.1.3-pc/game/ -c
...
Error while decompiling /home/olli/.xlib/RPG/_test/OurBrightDays-0.1.3-pc/game/tl/english/scripts/00parts/01arrival/01script/script.rpyc:
Traceback (most recent call last):
File "/home/olli/Code/rpy_tools/unrpyc/unrpyc-2.0.0/unrpyc.py", line 253, in worker
return extract_translations(filename, args.language)
File "/home/olli/Code/rpy_tools/unrpyc/unrpyc-2.0.0/unrpyc.py", line 215, in extract_translations
return magic.safe_dumps(translator.dialogue), translator.strings
File "/home/olli/Code/rpy_tools/unrpyc/unrpyc-2.0.0/decompiler/magic.py", line 612, in safe_dumps
SafePickler(file, protocol).dump(obj)
File "/usr/lib/python3.10/pickle.py", line 487, in dump
self.save(obj)
File "/usr/lib/python3.10/pickle.py", line 560, in save
f(self, obj)  # Call unbound method with explicit self
File "/usr/lib/python3.10/pickle.py", line 972, in save_dict
self._batch_setitems(obj.items())
File "/usr/lib/python3.10/pickle.py", line 998, in _batch_setitems
save(v)
File "/usr/lib/python3.10/pickle.py", line 560, in save
f(self, obj)  # Call unbound method with explicit self
File "/usr/lib/python3.10/pickle.py", line 932, in save_list
self._batch_appends(obj)
File "/usr/lib/python3.10/pickle.py", line 959, in _batch_appends
save(tmp[0])
File "/usr/lib/python3.10/pickle.py", line 603, in save
self.save_reduce(obj=obj, *rv)
File "/usr/lib/python3.10/pickle.py", line 687, in save_reduce
save(cls)
File "/usr/lib/python3.10/pickle.py", line 572, in save
self.save_global(obj)
File "/home/olli/Code/rpy_tools/unrpyc/unrpyc-2.0.0/decompiler/magic.py", line 525, in save_global
self.write(pickle.GLOBAL + obj.__module__ + '\n' + obj.__name__ + '\n')
TypeError: can't concat str to bytes
Writing translations to /home/olli/.xlib/RPG/_test/OurBrightDays-0.1.3-pc/game/tl/en_tl.txt...
Decompilation of 468 files successful, but decompilation of 26 files failed

I think this should be a py2->3 residue.
script.zip

[CensoredUsername]

in the rpy files. So if i am right this time, this feature just switches the basic language of the game with another. But why do such thing? I can just run the game and switch the l10n in the menu. Makes no sense to me.

Yep, that's what it does, that's what I meant in the last post. I believe Jackmcbarn's intention was so you can read through the decompiled code in a different language, preferably the one you can read.

Ah, seems like magic.py messes up generating the save_global code. pickle.GLOBAL is a bytes object, and everything there should be getting properly encoded. Easy fix.

[madeddy]

Yeah. I still needed to find pickle.GLOBAL to look what the hell this is. Not in the py docs... how nice.

self.write(pickle.GLOBAL + obj.module + '\n' + obj.name + '\n'

needs to be changed to

self.write(pickle.GLOBAL
           + bytes(obj.__module__, 'utf-8') + b'\n'
           + bytes(obj.__name__, 'utf-8') + b'\n')

and it works from the first look. If in the outfile everything right is do i not know.

[CensoredUsername]

Yeah. I still needed to find pickle.GLOBAL to look what the hell this is. Not in the py docs... how nice.

The internal format of the pickle protocol is only documented in the source code ;). Luckily the pickle docs page links to that at the top. I'm very familiar with it due to all the shenanigans I've done with it over the years.

Yup. I came up with the complete:

    def save_global(self, obj, name=None, pack=struct.pack):
        if isinstance(obj, FakeClassType):
            if PY2:
                self.write(pickle.GLOBAL + obj.__module__ + '\n' + obj.__name__ + '\n')
            elif self.proto >= 4:
                self.save(obj.__module__)
                self.save(obj.__name__)
                self.write(STACK_GLOBAL)
            else:
                self.write(pickle.GLOBAL + (
                    obj.__module__ + '\n' + obj.__name__ + '\n').decode("utf-8")
                )
            self.memoize(obj)
            return

(magic.py is part of the picklemagic library used by un.rpyc, or at least, I wrote it at first as a separate library. So I'll make the change there as well).

[madeddy]

(laughs)
I expected already i miss something or that it is needed also to cater to py2.

so you can read through the decompiled code in a different language, preferably the one you can read.

Yeah, i misunderstood this really. Seems not often used by people, but at least it makes sense now. If nothing else at least i found another bug to fix for you. 😛

I think we can close this then in a few...

[CensoredUsername]

I expected already i miss something or that it is needed also to cater to py2.

I wrote magic/picklemagic as a lib that supported both py2/3. To prevent them from diverging and making my life more painful it's easier to just keep them in sync.

The other fix is just doing the right thing regarding the protocol. Py2 only goes up to pickle protocol 2, but in py3 a bunch of things have been added. And ren'py might at one point just up the protocol so it's good to just keep up to date.

I think we can close this then in a few...

Yeah I'll close it after actually committing the code.

[madeddy]

it's easier to just keep them in sync.

This came to mind as i noticed you let the py2 code in it even in py3 branch. Makes sense from this POV.

[madeddy]

Not this important imho, for a bad weather day, but problems should be always noted. I add this here instead of making another issue and adjusted for it the title.

There are multiple problems with the cli args for the TL feat in itself and in combination with other decompiling cli args:

• -l language arg without the needed -T arg (or any other!) is silently without effect
• --try-harder in combination with TL args is part-silently without effect. Only by failed decompile output noticeable.
• -T tranlation-file and -t tranlation-file can be given at the same time and -t is silently without effect.

Not this hard to fix with some condition checks after the parser and/or argparse itself has some very useful features for stuff like this, like subparser, mutualy-exclusive.
What i don't understand is, why J.McBarn made a second run with -t if its anyway mandatory? Why not run this automatically after the TL-file from -T is written?

[CensoredUsername]

What i don't understand is, why J.McBarn made a second run with -t if its anyway mandatory? Why not run this automatically after the TL-file from -T is written?

I'd advice not to look for logic where there likely is none. It probably made it easier to implement as the tool is currently structured for a single pass. Doing a double pass for translations (and not having to save them to a file) would indeed be significantly cleaner. Then it could also only use one command line flag.

[madeddy]

I'd advice not to look for logic where there likely is none.

Yeah, that's right. I tend to search in everything the sense and logic.

Then it could also only use one command line flag.

Exactly my thought! I believe whatever we ever write in a possible doc for this, will still confuse people.

Also, to hijack this issue just because its related to translation was a fault. There are other collisions in the cli args... well, something for another time.

[madeddy]

Just a update:
I looked at this a bit and think the TL feat can be relatively less invasive changed. This will then not write the TL file out after step 1 and goes direct to step 2 to switch the main language.

Means, it uses then just one argparse option instead of three and some guards for bad argparse arg combinations would be unneeded.
--translate {language}
This would be the new arg without a default language(english is mostly the None lang), so users need to look the available up in the TL directory instead running this with default and wondering why nothing happens like now.

So far are my view. As said, this feature is IMO not often used and not really broken, so its at the end of my TODO(read: what-i-like-to-get-done-before-next-release).

[CensoredUsername]

That'd indeed be a nice way of handling it. Regarding TODO's before releases, for stuff that doesn't break any previous behaviour I wouldn't worry too much. I've in fact been working to just a new release today to get the new compatibility rules out of the door. With the new testing framework preparing releases seems to be even easier as there's a lot less risk of un.rpyc bugs (I still tested it with actual ren'py for the record).

I'll just do that now. No worries about it, when dev's finished on other action items we can just make another one.