Implement Cisco type 8 (sha256) and 9 (scrypt)

[magnumripper]

Cisco type 9 is scrypt. I know nothing more but here's a sample (from CMIYC):
$9$cvWdfQlRRDKq/U$VFTPha5VHTCbSgSUAo.nPoh50ZiXOw1zmljEjXkaq1g
Plaintext is 123456

Base64 charset is likely same as Cisco type 4.

[jfoug]

from http://www.cisco.com/c/en/us/td/docs/ios-xml/ios/security/d1/sec-d1-cr-book/sec-cr-e1.html

Examples

The following example shows how to generate a type 8 (PBKDF2 with SHA-256) or a type 9 (SCRYPT) password:

Device# configure terminal
Device(config)# username demo8 algorithm-type sha256 secret cisco
Device(config)# username demo9 algorithm-type scrypt secret cisco
Device(config)# end
Device# show running-config | inc username

username demo8 secret 8 $8$dsYGNam3K1SIJO$7nv/35M/qr6t.dVc7UY9zrJDWRVqncHub1PE9UlMQFs
username demo9 secret 9 $9$nhEmQVczB7dqsO$X.HsgL6x1il0RxkOSSvyQYwucySCt7qFm4v7pqCxkKM

so $9$nhEmQVczB7dqsO$X.HsgL6x1il0RxkOSSvyQYwucySCt7qFm4v7pqCxkKM is another example with plaintext cisco (NOTE, prior plaintext of secret was WRONG, it is cisco)

[jfoug]

hash cat has type 8 and 9 cisco hashes now: http://hashcat.net/forum/thread-3710.html

[jfoug]

I think type 8 is 80 bit salt, 1000 iteration pbkdf2-hmac-sha256. I have been trying to get it to work in pass_gen, but have not gotten it yet. I was easily able to get cisco4 working, so I was hoping for ease on the type 8. No go so far. The salt is 14 bytes, which 'could' give 80 bits if the 14th byte was always one of the first 4 base-64 values. It is not. So it looks like you get 84 bits of salt? I am not quite sure what to do with that. I have tried 3 different things for salts. full raw 14 bytes, truncated to 80 bits (after base-64 decode), and 88 bits (padding with 0 bits). I have tried 64 to 50k (step 64) and 100 to 50k step 100 iterations for all of them. Nothing lines up yet. I sent an email to Atom, asking if he would point me in the right direction.

I also thought the last 4 bits might be 1<<bits. The value I am seeing is 0xA (10), so 2<<10 is 2048. That sounds like a normal iteration count, BUT that was one of them I tried (and did not get a match).

[jfoug]

More examples: hashcat plaintext

$8$TnGX/fE4KGHOVU$pEhnEvxrvaynpi8j4f.EMHr6M.FzU8xnZnBr/tJdFWk
$9$2MJBozw/9R3UsU$2lFhcKvpghcyw8deP25GOfyZaagyUOGBymkryvOdfo6

[jfoug]

This reply came from atom:

Hey Jim,

I did discuss this topic with Phil and we agreed to publish the detailed informations about both hashes on hashcat forum. Hope that's ok for you.

--
Jens

So hashcat will publish at some time. I have tried about all I can (withing pass_gen). I even fell back and tried the interleaved bytes, like cryptmd5. That did not help either. I am sure it is encoding problems, I just do not know how to do it yet.

[frank-dittrich]

https://hashcat.net/forum/thread-3803-post-21661.html#pid21661

[jfoug]

Finally got things working. Problem I was having was the wrong plaintext (cisco and not secret)

[jfoug]

here is pass_gen on cisco8

sub cisco8 {
    if (defined $argsalt && length($argsalt)==14) { $salt = $argsalt; } else { $salt = randstr(14,\@i64); }
    my $wpaH = pp_pbkdf2($_[1],$salt,20000,"sha256",32);
    my $s = base64_wpa($wpaH);
    print "u-cisco8:\$8\$$salt\$$s:$u:0:$_[0]::\n";
}

[jfoug]

Got cisco9 working also

use Crypt::ScryptKDF qw (scrypt_raw);
# ...
sub cisco9 {
    if (defined $argsalt && length($argsalt)==14) { $salt = $argsalt; } else { $salt = randstr(14,\@i64); }
    my $h = scrypt_raw($_[1],$salt,16384,1,1,32);
    my $s = base64_wpa($h);
    print "u-cisco9:\$9\$$salt\$$s:$u:0:$_[0]::\n";
}

[jfoug]

I have made changes to pbkdf2_hmac_sha256_fmt.c (and base64,[ch]) to allow cisco8 to work. Pretty trivial. Just a small change to valid, a prepare (to convert from $8$ to cannonical), a change to get_salt, and 2 conversion functions in base-64.

46869ac Adds cisco8 to CPU side. GPU should be similar trivial to add.
e1908ad Adds cisco8 to GPU (fixes a static missed on prepare on CPU side)

[magnumripper]

Good stuff. Makes me want #816 again. Do you have any idea how to fix that one in a KISS matter?

[jfoug]

I do not follow how #816 relates to this change.

Also, I am not 100% sure that cisco9 would fit into any of our existing scrypt formats, due to too many nuanced differences. I am not 100% sure that the way I did it is absolutely the best. I make an assumption, that a 20k iteration, 14 char salt is a cisco8 that has been prepared.

[magnumripper]

I do not follow how #816 relates to this change.

For benchmarking the new test vectors, you'd run eg.
./john -test -format:pbkdf2-hmac-sha256 -cost=20000

[magnumripper]

Build bots alerted me of a segfault:

Testing: PBKDF2-HMAC-SHA256, rounds=12000 [PBKDF2-SHA256 128/128 XOP 4x]... (4xOMP) make: *** [check] Segmentation fault

https://travis-ci.org/magnumripper/JohnTheRipper/builds

[jfoug]

$ ../run/john -test -form=pbkdf2-hmac-sha256
Will run 4 OpenMP threads
Benchmarking: PBKDF2-HMAC-SHA256, rounds=12000 [PBKDF2-SHA256 128/128 XOP 4x]... (4xOMP) DONE
Raw: 814 c/s real, 206 c/s virtual

I also ran a make check and it processed with zero problems.

[jfoug]

I have just tested with linux64, and same result (works fine, pass). does this also do a make clean or start in a clean dir ?

The first time it failed, the clang build was fine, only the gcc failed. From that point on ,both fail. I am seeing NO failures, so can not replicate teh problem.

[frank-dittrich]

@magnumripper

Good stuff. Makes me want #816 again. Do you have any idea how to fix that one in a KISS matter?

I wanted to have a look, but so far I just didn't have enough time.

[jfoug]

@magnumripper can we do a make clean on travis ?

[magnumripper]

They are virtual machines always starting from scratch. They even install libssl-dev and other things, each and every time.

[magnumripper]

I too have yet to reproduce the Travis problem. I have tried to understand what difference there is. I see they use XOP, I'm not. Other than that, I run the same SSE_PARA, OMP_NUM_THREADS and so on.

[magnumripper]

Oh, and I see you run XOP. So what could it be? Theoretically it could be a bug in the bot or the virtual hardware but I do not think so.

[magnumripper]

I did add a make clean just to be 100.0% sure, but it did not help.

[magnumripper]

Also, as you probably did yourself, I have audited your changes and I really can't see any problem.

[frank-dittrich]

Which was the last commit that was successfully built by Travis? Apparently b309863.
Jim could revert all recent changes in a local repository and create a dummy pull request, just to check whether Travis builds that successfully?
(I'm too tired right now, so I'll probably fail even with such a simple task.)

If Travis can't build a version anymore that is the same as b309863, then we know it is not our fault and can submit a bug report to Travis...

[magnumripper]

I can reproduce now. Fixed soon.

[jfoug]

What did you do to reproduce? I am curious since I have run this on 4 OS's, 32 and 64 bit, and it is happy to run. I have also done tests (not just -test), finding passwords. It is happy to work for me, everywehere I tried it EXCEPT travis.

[magnumripper]

Compiler version dependant. I get it with clang, or eg. gcc 4.6.3. In the latter case it vanishes when I compile with -O0 :-(

[magnumripper]

Unfortunately I do not have Valgrind available atm. Trying to understand what happens.

This will be totally arch & compiler dependant but the first time I hit these lines in prepare():

    if (!Buf)
        Buf = mem_alloc_tiny(120, 1);

Buf is already 0x04 (the pointer was overwritten prior to this).

[frank-dittrich]

Can you make gcc --version and clang --version part of the Travis builds?

[jfoug]

I am hoping this may fix it 9a204b6

If so, then I will get this into opencl version

[jfoug]

Nope, no fix. I can see NO reason that var gets overwritten.

[jfoug]

If my last change does not work, we might want to simply drop the static pointer and replace it with a static buffer BUT I would like to find out WHY this is failing. THERE HAS TO BE a overwrite somewhere. I simply can not see it.

[jfoug]

I am done poking the hornet nest. @magnumripper you might try this prepare:

static char *prepare(char *fields[10], struct fmt_main *self)
{
    static char Buf[120];
    char tmp[50];

    if (strncmp(fields[1], FMT_CISCO8, 3) != 0)
        return fields[1];
    if (strlen(fields[1]) != 4+14+43)
        return fields[1];
    sprintf (Buf, "%s20000$%14.14s$%s", FMT_PREFIX, &(fields[1][3]), crypt64_to_mime64(&(fields[1][3+14+1]), tmp, 43));
    return Buf;
}

NOTE, this still does not show us where the overwrite is happening at.

[magnumripper]

I have nailed it and I'm testing a canonical fix.

[jfoug]

I am really interested in this fix. It escapes me. I just do not see the bug.

[magnumripper]

I thought I had it but it still escapes me. I set a hardware breakpoint on that static variable so I know the overwrite happens during base64_decode() but I am not sure why yet.

For a while I was sure the problem was that crypt64_to_mime64() does not add MIME "=" padding but that was not it. It's taken care of elsewhere.

Actually the bug might well be prior to your Cisco changes (and just triggered by them): If I drop all test vectors but the Cisco ones, problem goes away!

[magnumripper]

Please do not commit anything unless you are SURE you have found a specific bug. You might end up just hiding the bug and we do NOT want that!!

[magnumripper]

Aw man, this was a lot easier than we thought:

@@ -257,7 +257,7 @@ static void *get_salt(char *ciphertext)
 static void *get_binary(char *ciphertext)
 {
        static union {
-               char c[BINARY_SIZE];
+               char c[BINARY_SIZE + 1];
                ARCH_WORD dummy;
        } buf;
        char *ret = buf.c;

base64_decode() added a null termination beyond BINARY_SIZE

[magnumripper]

And yes, this is an OLD bug. This overwrite happened with the very first test vector!

[magnumripper]

Travis is good

[magnumripper]

Reopen for cisco9

[kholia]

Why does this looks very familiar?

This is why my buffers have weird sizes when I do base64 decoding ;)

[magnumripper]

I overhauled every format that included base64.h in c4574d0. I might have added some redundant extra but many of them had this bug for sure. Also, many formats included base64.h although they did not need to.

[frank-dittrich]

Wouldn't it be better to adjust the definition of BINARY_SIZE instead of using BINARY_SIZE + 1 and define a BINARY_SIZE for formats that don't have thist definition?

[magnumripper]

No, because BINARY_SIZE is reported to core. Core will copy that many bytes to an aligned buffer in the db.

Imagine loading a million hashes with BINARY_SIZE of 33 and BINARY_ALIGN of 4. Thats a lot of wasted memory and cache.

[jfoug]

Got cisco9. It is in scrypt_fmt.c Simple EXCEPT for that DAMN CRAPPY base-64 stuff.

$ cat cisco9.in
cisco9_1:$9$nhEmQVczB7dqsO$X.HsgL6x1il0RxkOSSvyQYwucySCt7qFm4v7pqCxkKM::cisco
cisco9_2:$9$cvWdfQlRRDKq/U$VFTPha5VHTCbSgSUAo.nPoh50ZiXOw1zmljEjXkaq1g::123456
cisco9_3:$9$X9fA8mypebLFVj$Klp6X9hxNhkns0kwUIinvLRSIgWOvCwDhVTZqjsycyU::JtR


$ cat pw
123456
cisco
JtR

$ OMP_NUM_THREADS=4 ../run/john cisco9.in -w=pw
Loaded 3 password hashes with 3 different salts (scrypt [Salsa20/8 128/128 AVX])
Will run 4 OpenMP threads
Press 'q' or Ctrl-C to abort, almost any other key for status
cisco            (cisco9_1)
JtR              (cisco9_3)
123456           (cisco9_2)
3g 0:00:00:00 DONE (2014-11-06 13:17) 214.2g/s 200.0p/s 600.0c/s 600.0C/s 123456..JtR
Use the "--show" option to display all of the cracked passwords reliably
Session completed

$ cat ../run/john.pot
$7$C/..../....nhEmQVczB7dqsO$AG.yl8LDCkiErlh4ttizmxYCXSiXYrNY6vKmLDKj/P4:cisco
$7$C/..../....X9fA8mypebLFVj$Pp/GAiPTalxAUn0D/ioghTbLGVgagzy1qSFtOvjDXWC:JtR
$7$C/..../....cvWdfQlRRDKq/U$3Slqq4KsBBzduRAcn.oAjhoF8gpcf/wz9j/oymX7Mj1:123456

$ ../run/john cisco9.in -show
cisco9_1:cisco::cisco
cisco9_2:123456::123456
cisco9_3:JtR::JtR

3 password hashes cracked, 0 left

[jfoug]

Cisco9: 7216d2c

We could probably also handle django in prepare() also, since it really is only the way N/r/p are in the salt, and the base-64 encoding. Yes, all 3 scrypts use DIFFERENT base-64, hence why I DESPISE that encoding method

[videgro]

I would like to inform you that I used the discussion in this issue to make a Java implementation of the Type 8 and Type 9 hash calculations. Thanks!
See: https://github.com/videgro/cisco-password-hashes