very bad (and decreasing) performance on mode "Single" in certain circumstances #2425

Open
p4tpr0 opened this Issue Feb 12, 2017 · 2 comments

Comments

Projects
None yet
3 participants
@p4tpr0

p4tpr0 commented Feb 12, 2017

I have a hash file where every line looks like this:

candidate:$dynamic_25$hash$salt

"candidate" is a very serious password candidate to test, with 50-60% success rate without even using mangling rules.
When using mode "single" on this big hash file with "SingleRetestGuessed = N" and no rules, I see a bad performance that decrease along the cracking session. Sample session:

$ time ./john --single=None --nolog --verbosity=1 pw-1M-real  --pot=pw-1M-real.pot
Using default input encoding: UTF-8
Loaded 1000000 password hashes with 1000000 different salts (dynamic_25 [sha1($s.$p) 128/128 AVX 4x1])
Press 'q' or Ctrl-C to abort, almost any other key for status
0g 0:00:00:02 50.00% (ETA: 16:24:20) 0g/s 0p/s 0c/s 0C/s
30769g 0:00:00:08 50.00% (ETA: 16:24:33) 3445g/s 5563p/s 5563c/s 5563C/s aoygm
48159g 0:00:00:18 50.00% (ETA: 16:24:53) 2544g/s 4108p/s 4108c/s 4108C/s agmys
61712g 0:00:00:29 50.00% (ETA: 16:25:15) 2062g/s 3341p/s 3341c/s 3341C/s aprzjx
82423g 0:00:00:52 50.00% (ETA: 16:26:01) 1557g/s 2523p/s 2523c/s 2523C/s cxox
93556g 0:00:01:08 50.00% (ETA: 16:26:33) 1357g/s 2201p/s 2201c/s 2201C/s yvoq
105281g 0:00:01:29 50.00% (ETA: 16:27:15) 1170g/s 1897p/s 1897c/s 1897C/s pfkwx
117217g 0:00:01:54 50.00% (ETA: 16:28:04) 1028g/s 1665p/s 1665c/s 1665C/s vpqu
132177g 0:00:02:29 50.00% (ETA: 16:29:15) 881.5g/s 1428p/s 1428c/s 1428C/s acwzhx
145131g 0:00:03:06 50.00% (ETA: 16:30:29) 776.4g/s 1257p/s 1257c/s 1257C/s airzx
193370g 0:00:06:35 50.00% (ETA: 16:37:27) 488.4g/s 791.9p/s 791.9c/s 791.9C/s amoxr
...

The performance is good if none of the candidates matches its hash, or if each candidate matches its hash. The performance is very bad if the file is a mix of both cases.

How to reproduce:

  • create a 1M hash file with this command:
perl -e 'use Digest::SHA qw(sha1_hex); $p="a"; for ($i=0;$i<1000000;$i++) {$s=substr(sha1_hex($p), 16); $h=sha1_hex($s.$p); print "$p:\$dynamic_25\$$h\$$s\n"; $p++;}' > pw-1M
  • create a more realistic file, where ~38% candidates do not match their hash:
sed 's,\([iwnfjzahlc]\):,\1x:,' pw-1M >pw-1M-real
  • use john on each file, with "SingleRetestGuessed = N" in john.conf, and following run time options:
./john --single=None --nolog --verbosity=1 pw-1M  --pot=blank-pot
./john --single=None --nolog --verbosity=1 pw-1M-real  --pot=other-blank-pot

Memory consumption is quite high also, but performance is my main concern here.

John build-info:

Version: 1.8.0.9-jumbo-1-bleeding
Build: freebsd10.3 64-bit AVX-ac
SIMD: AVX, interleaving: MD4:3 MD5:3 SHA1:1 SHA256:1 SHA512:1
$JOHN is ./
Format interface version: 14
Max. number of reported tunable costs: 3
Rec file version: REC4
Charset file version: CHR3
CHARSET_MIN: 1 (0x01)
CHARSET_MAX: 255 (0xff)
CHARSET_LENGTH: 24
SALT_HASH_SIZE: 1048576
Max. Markov mode level: 400
Max. Markov mode password length: 30
gcc version: 5.4.0
Crypto library: OpenSSL
OpenSSL library version: 020000000 (loaded: 01000113f)
LibreSSL 2.4.4 (loaded: OpenSSL 1.0.1s-freebsd 1 Mar 2016)
GMP library version: 5.1.3
File locking: fcntl()
fseek(): fseek
ftell(): ftell
fopen(): fopen
memmem(): System's

Full discussion on the mailing-list:
http://www.openwall.com/lists/john-users/2017/02/02/1

@magnumripper magnumripper added the bug label Feb 13, 2017

@magnumripper magnumripper added this to the 1.8.0-jumbo-2 milestone Feb 13, 2017

@jfoug

This comment has been minimized.

Show comment
Hide comment
@jfoug

jfoug Feb 14, 2017

Collaborator

Note, this bug has nothing to do with the min-cracks == 64 in dynamic. But I am going to see if that can be reduced to 1, since that DOES impact memory greatly, but I think that reduction will also greatly impact single throughput, when normal single rules are used. Dyna has a LOT of overhead. Reducing it to 1 (or SIMD_COEF) will make other runs of single run at probably 10-20% of what they should.

I have done a bit of single stepping, and I have not found where this problem is located at. I will look more when I have more time.

Collaborator

jfoug commented Feb 14, 2017

Note, this bug has nothing to do with the min-cracks == 64 in dynamic. But I am going to see if that can be reduced to 1, since that DOES impact memory greatly, but I think that reduction will also greatly impact single throughput, when normal single rules are used. Dyna has a LOT of overhead. Reducing it to 1 (or SIMD_COEF) will make other runs of single run at probably 10-20% of what they should.

I have done a bit of single stepping, and I have not found where this problem is located at. I will look more when I have more time.

@magnumripper

This comment has been minimized.

Show comment
Hide comment
@magnumripper

magnumripper Jan 8, 2018

Owner

The min. keys per crypt should always be (SIMD_COEF * SIMD_PARA * THREADS) as far as I know. Bumping it to 64 doesn't make any sense to me.

Owner

magnumripper commented Jan 8, 2018

The min. keys per crypt should always be (SIMD_COEF * SIMD_PARA * THREADS) as far as I know. Bumping it to 64 doesn't make any sense to me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment