UNERF: Encrypted ERF (v2.2 blowfish) not working

[hemidemisemiquaver]

It seems that the unerf tool is testing the password's md5 instead of the decimal string corresponding to the password, which prevents it from extracting even when using a correct password.
ERF 2.2 Format
ERF Blowfish decryption / encryption steps
Attaching an example.
Created in Dragon Age: Origins Toolset - ERF Editor

ERF version: v2.2
Encryption: Blowfish
Module Id (decimal, as expected by DA:O Toolset): 123
Password (decimal, as expected by DA:O Toolset): 1234567890

DA:O Toolset ERF Editor opens the file, shows its content and extracts just fine (hello_world.txt)
Running unerf results in error:

unerf --pass 499602D2 e HelloWorld.erf
ERROR: Failed reading ERF file
    Because: Password digest does not match

HelloWorld.zip

[DrMcCoy]

Hmm, that used to work, so I guess we broke it. I need to look at that later today,
Thanks for reporting! :)

Which version of xoreos-tools are you using? I.e. a release package, a recent git checkout, ... unerf --version should tell you more if you're not certain.

[hemidemisemiquaver]

Ah, sorry, I should have included that! Here you go:
xoreos-tools 0.0.5+0.gaed64a2 [0.0.5+0.gaed64a2] (2018-07-03T12:39:22Z)

[DrMcCoy]

Ah, yeah, I see the problem.

The password, as a decimal number, is 1234567890. However, due to how the digest is calculated and how both the Neverwinter Nights premium modules and Dragon Age 2 handle their ERF encryption and digests, my thought of unifying that was that our unerf tool should expect the hexadecimal representation of the string "1234567890".

So the password that our unerf tool wants is 31323334353637383930, which is the string in hexadecimal form.

I.e.

$ echo -n 1234567890 | hexdump -C
00000000  31 32 33 34 35 36 37 38  39 30                    |1234567890|
0000000a

Yes, that is quite confusing, I know. I guess we should at least document that better, in the --help text and in the man page?

Or possibly, handle this a bit better and less confusingly. --password should probably be more context sensitive and do the unsurprising thing depending on the version of the ERF file given. And then add --password-number, --password-text and --password-hex so that you can explictly override the format if necessary.

Does that sound sensible?

[hemidemisemiquaver]

Ah, that makes sense!
I think any approach would work (documentation - with an example / determining how to handle the password dynamically based on the version / or option to specify the password format).

Personally, as a user, I would probably prefer it to take password the same way as DA:O Toolset, so I wouldn't need to worry about the dec-str-hex gymnastics every time.

As a programmer, I would probably want something very consistent across games / versions. A question here, is the content actually encrypted with 0x31323334353637383930 key? Or is it encrypted with 0x499602D2 ? If it's the former, then using 31323334353637383930 is perfectly reasonable. If it is the latter - like the toolset wiki seems to describe it, then there is no way to really avoid confusion (as md5 hash is of a decimal string, while the password is really a bigint).

• Generate random number (represented as string, base 10), such as "12345678".
• Generate digest of the string (without null-terminating character) using MD5 and store it in the header.
• Convert those digits to 64-bit integer (0x0000000000BC614E).
• Store it in byte array in little-endian encoding (0x4E, 0x61, 0xBC, 0x00, 0x00, 0x00, 0x00, 0x00).
• Use that array as key to initialize Blowfish.

In any event, at least now that I know what it expects, it is not a blocker anymore. Thanks for the explanation!

[DrMcCoy]

A question here, is the content actually encrypted with 0x31323334353637383930 key? Or is it encrypted with 0x499602D2 ?

It's a bit more complicated.

The MD5 digest is calculated using the string "1234567890", i.e. 0x31323334353637383930.

But for the actual compression, the number 1234567890 is treated as a little-endian uint64_t, and that is the key. I.e. the decompression key is the 8-byte array containing { 0xD2, 0x02, 0x96, 0x49, 0x00, 0x00, 0x00, 0x00 }.

Again, that's just for Dragon Age: Origins. Dragon Age 2 is less weird there.

[DrMcCoy]

For Dragon Age 2, the key is just plain old bytes. The NWN premium key is also plain old bytes, but part of the key is the MD5 hash of the main MOD file.

Basically, that's why I thought the unified password parameter would be plain old bytes as well, with the string being the "natural" form, since that's used for the digest. And it's probably the easiest specifying a plain old byte array as hex on the comment line. That was my thought-process, at least.

[hemidemisemiquaver]

I think the simplest solution would be to just keep the current behavior and treat this as a documentation improvement. I’d say a DA:O specific example in the —help and man would suffice. Conversion is easy, just adding 30 to each digit is simple enough as long as the user remembers to.