LM and LM-opencl fails with --enc:iso8859-1

[magnumripper]

#5884 (comment)

I realize now that is because ISO-8859-1 simply isn't compatible with the LM format, which uses DOS codepages such as CP437.

I assume 96c9f62 is what caused this, and we should add some code to skip that test vector or handle it some other way.

[magnumripper]

This is complicated, as usual when it comes to codepages or encodings. I should have written this back in #5702:

It "happens" to works fine (see below) with any DOS code page (three-digit), at least those we currently support. Even those where \x9a corresponds to something other than "Ü". And they should, because we never claimed it to be Ü, just the binary character value \x9a in the string that is hashed.

It fails on nearly all "ANSI" codepages (four-digit) and all ISO-8859 ones. Actually one could say it happens to fail - and it just happens to work fine with CP1255 and CP1256.

The reason is casing. The format uppercases any lowercase characters in the password candidate before hashing. This is codepage-aware: If \x9a is a lowercase character in the selected codepage, it will be uppercased to something else before hashing so we get a different result and our self-test fails.

• In DOS codepages, \x9a is Ü or some other uppercase character, so it's not changed. Works fine.

• I think for CP720 (arabic), the characters doesn't even have a "case" so they too are unchanged and work fine.

• But in CP1250 and most other non-DOS code pages (including all ISO-8859) we support, \x9a happens to correspond to some lowercase character. It will be uppercased -> the hash ends up wrong -> self-test fails.

• In CP1255 and CP1256, I believe casing isn't even a thing (again arabic), so it wont be changed and passes self-test. Either that, or \x9a happens to represent some non-letter such as a math symbol or degree sign. Whatever - casing will not change it so it works.

• We don't even try to uppercase UTF-8 within the LM format (would affect performance and makes no sense anyway). So self-test works fine even for that.

• For --encoding=raw we are dumbed down to 7-bit ASCII with transparent support for 8-bit. That is, we uppercase nothing 8-bit so self-test will pass.

Oh and for the record, when running LM (including NETLM and some others) we default to CP850 if no encoding options are given. And UTF-8 input. For western countries it will magically just work. This default is configurable in john.conf and it's all described in doc/ENCODINGS.

So what are our options?

1. If we can find any byte value among ALL codepages that is never uppercased, we could use that instead. I now recall some struggle with finding one - I believe there simply is none!
2. We could chug along as we always did, but disable that very test vector if an offending codepage is selected. The docs warns about running with a non-DOS codepage, I'm pretty sure.
3. We could refuse to even run LM formats with a non-DOS codepage (or UTF-8 target encoding), with some description pointing to the docs. Doing so makes no sense anyway - you may semi-randomly crack some 8-bit strings but they will be misrepresented in the output and log/pot files.

I think we should go for option 2. @solardiz ?

[solardiz]

If we seem to have failed at option 1, then it's option 2 or 3 or a mix of these two - "disable that very test vector if an offending codepage is selected", but also print a warning.

When you say that running "LM formats with a non-DOS codepage" "makes no sense", is this because of our limitations ("misrepresented in the output and log/pot files") or also because it doesn't correspond to what would happen on a target system using LM hashes and such codepage?

[magnumripper]

is this because of our limitations ("misrepresented in the output and log/pot files") or also because it doesn't correspond to what would happen on a target system using LM hashes and such codepage?

The latter. No Windows system would use anything but a "DOS codepage" for LM hashes, so like I said "you may semi-randomly crack some 8-bit strings. Like having "парол" in your wordlist but it happens to crack "∩Α≡ΕΔ".

[magnumripper]

I see it was changed again in d7a4b6d to mitigate the problem a little (that was the struggle I recalled having), After that we at least have it working in any DOS codepage. Just not in ANSI or ISO-8859 ones.