Invalid unicode strings in various wld files

[bryab]

I am trying to figure out how to deal with the fact that many strings in these files are either invalid utf-8 (and encoded in some other format?) or are simply not strings at all.

Take feerrott for example. Its main string hash has invalid utf8; it will fail to load altogether.
For feerrott_obj, the animated fire texture's TextureFragmentEntry's "file_name" parameter is an invalid string.

I'd love to try to figure this out - what tools did you use for examining these files?

[bryab]

OK, I got this working using the encoding_rs create and decoding use WINDOWS_1252. Will post up this up at some point.

[cjab]

Nice! I think in my branch currently I'm just swallowing those utf-8 errors. I had no idea the encoding was windows 1252, good find! :D

As for tools I've been using kind of hodge podge of things. hexyl for viewing fragment data, and more recently I've used kaitai (http://kaitai.io) to build some quick and dirty parsers to test things out. This has been a nice little calculator also :P https://github.com/alt-romes/programmer-calculator

I do have a little TUI I've written for stepping through fragments and viewing their fields plus a hex view of their contents (via hexyl). I'll see if I can get that up here as well. Last weekend didn't work out the way I had hoped 😆.

[bryab]

I'm going to reopen this because honestly I don't think the windows 1252 encoding is correct. Something else is going on here. For example in the feerrott zone, using windows_1252, one of the materials says its texture name is:

•:å*•z•j•:;õj…wšpzuaœ;å¨â{å*•z•j•:å*•z<ß×xóp±¸à4žøhï

I'm wondering if there's some "magic", possibly specifically in texture names that doesn't represent an actual string but instead some kind of other instruction.