Added optional parallelization of the MD5-based crypt(3) code with Op…

…enMP.

doc/CHANGES

@@ -1,5 +1,6 @@
 The following changes have been made between John 1.7.8 and 1.7.9:
 
+* Added optional parallelization of the MD5-based crypt(3) code with OpenMP.
 * Added optional parallelization of the bitslice DES code with OpenMP.
 * Replaced the bitslice DES key setup algorithm with a faster one, which
 significantly improves performance at LM hashes, as well as at DES-based
@@ -268,4 +269,4 @@ Mac OS X (PowerPC and x86), SCO, BeOS.
 * Bug and portability fixes, and new bugs.
 * Bonus: "Strip" cracker included in the default john.conf (john.ini).
 
-$Owl: Owl/packages/john/john/doc/CHANGES,v 1.69 2011/11/20 05:20:33 solar Exp $
+$Owl: Owl/packages/john/john/doc/CHANGES,v 1.70 2011/11/21 02:36:55 solar Exp $

doc/FAQ

@@ -190,34 +190,36 @@ and refuses to work if an error occurs.  If you need to test all of the
 low-level routines at once, use "--test".
 
 Q: Does John support multi-processing or distributed processing?
-A: There's currently built-in parallel processing support (to make use
-of multiple CPUs and/or CPU cores on a single system) for DES-based
-crypt(3) hashes (traditional, bigcrypt, and BSDI-style), OpenBSD-style
-Blowfish-based crypt(3) (bcrypt) hashes (with John's own optimized code)
-and for the underlying system's thread-safe password hashing function
-(crypt_r(3) on Linux or crypt(3C) on Solaris).  The latter is only
-reasonable to use for crypt(3) hash types not yet supported by John
-natively (that is, for glibc 2.7+ SHA-crypt hashes as used by recent
-versions of Fedora and Ubuntu, and for SunMD5 hashes).  To use this
-limited OpenMP support, you need to make an OpenMP-enabled build of John
-by uncommenting one of the OMPFLAGS lines near the beginning of the
-Makefile.  This requires GCC 4.2+ or another OpenMP-capable C compiler.
-For other hash types and/or to distribute the workload between multiple
-machines, other approaches need to be used.  For a small number of nodes
-(CPUs, CPU cores, and/or machines), it is reasonable to use a manual
-approach.  One of those approaches is to have your nodes try different
-password lengths.  This is easily accomplished with "incremental" mode's
-"MinLen" and "MaxLen" settings (see CONFIG).  Typically, you would not
-really need to split the workload for "single crack" and wordlist modes
-since these are relatively quick, although you may dedicate one node to
-those initially.  You may safely run multiple instances of John in the
-same working directory, all writing to the same "pot file" (this is a
-feature).  You do, however, need to assign each of them a unique session
-name, with "--session".  Other approaches, such as splitting password
-files naively (without regard to salts), are typically less efficient
-(in some cases to the extent where there's no speedup from using
-multiple nodes at all).  Some advanced and automated approaches are
-listed on the wiki at:
+A: There's currently built-in parallel processing support using OpenMP
+(to make use of multiple CPUs and/or CPU cores in a single system) for
+all crypt(3) hash flavors (DES-, MD5-, and Blowfish-based) supported by
+John natively, as well as for LM hashes and, when running on Linux or
+Solaris, also for the underlying system's thread-safe password hashing
+function.  The latter is only reasonable to use for crypt(3) hash types
+not yet supported by John natively (that is, for glibc 2.7+ SHA-crypt
+hashes as used by recent versions of Fedora and Ubuntu, and for SunMD5
+hashes, which may optionally be enabled on Solaris).  In "community
+enhanced" -jumbo versions, parallelization with OpenMP is also supported
+for many (but not all) of the hash types added in those versions.  To
+use John's OpenMP support, you need to make an OpenMP-enabled build by
+uncommenting one of the OMPFLAGS lines near the beginning of the
+Makefile.  This requires GCC 4.2 or newer, or another OpenMP-capable C
+compiler.  For other hash types and/or to distribute the workload
+between multiple machines, other approaches need to be used.  For a
+small number of nodes (CPUs, CPU cores, and/or machines), it is
+reasonable to use a manual approach.  One of those approaches is to have
+your nodes try different password lengths.  This is easily accomplished
+with "incremental" mode's "MinLen" and "MaxLen" settings (see CONFIG).
+Typically, you would not really need to split the workload for "single
+crack" and wordlist modes since these are relatively quick, although you
+may dedicate one node to those initially.  You may safely run multiple
+instances of John in the same working directory, all writing to the same
+"pot file" (this is a feature).  You do, however, need to assign each of
+them a unique session name, with "--session".  Other approaches, such as
+splitting password files naively (without regard to salts), are
+typically less efficient (in some cases to the extent where there's no
+speedup from using multiple nodes at all).  Some advanced and automated
+approaches are listed on the wiki at:
 http://openwall.info/wiki/john/parallelization
 
 Q: What is the format of the crash recovery files ("john.rec", other
@@ -232,4 +234,4 @@ trivial to explain, it is not possible to reasonably describe some
 others without going into great detail on John internals.  If you really
 need to know, read the source code.
 
-$Owl: Owl/packages/john/john/doc/FAQ,v 1.26 2011/10/24 01:44:46 solar Exp $
+$Owl: Owl/packages/john/john/doc/FAQ,v 1.27 2011/11/21 02:36:55 solar Exp $

src/MD5_fmt.c

@@ -42,7 +42,23 @@ static struct fmt_tests tests[] = {
 	{NULL}
 };
 
-static char saved_key[MD5_N][PLAINTEXT_LENGTH + 1];
+static char (*saved_key)[PLAINTEXT_LENGTH + 1];
+
+struct fmt_main fmt_MD5;
+
+static void init(void)
+{
+	MD5_std_init();
+
+#if MD5_std_mt
+	fmt_MD5.params.min_keys_per_crypt = MD5_std_min_kpc;
+	fmt_MD5.params.max_keys_per_crypt = MD5_std_max_kpc;
+#endif
+
+	saved_key = mem_alloc_tiny(
+	    sizeof(*saved_key) * fmt_MD5.params.max_keys_per_crypt,
+	    MEM_ALIGN_CACHE);
+}
 
 static int valid(char *ciphertext)
 {
@@ -103,36 +119,43 @@ static int binary_hash_6(void *binary)
 
 static int get_hash_0(int index)
 {
+	init_t();
 	return MD5_out[index][0] & 0xF;
 }
 
 static int get_hash_1(int index)
 {
+	init_t();
 	return MD5_out[index][0] & 0xFF;
 }
 
 static int get_hash_2(int index)
 {
+	init_t();
 	return MD5_out[index][0] & 0xFFF;
 }
 
 static int get_hash_3(int index)
 {
+	init_t();
 	return MD5_out[index][0] & 0xFFFF;
 }
 
 static int get_hash_4(int index)
 {
+	init_t();
 	return MD5_out[index][0] & 0xFFFFF;
 }
 
 static int get_hash_5(int index)
 {
+	init_t();
 	return MD5_out[index][0] & 0xFFFFFF;
 }
 
 static int get_hash_6(int index)
 {
+	init_t();
 	return MD5_out[index][0] & 0x7FFFFFF;
 }
 
@@ -172,21 +195,31 @@ static char *get_key(int index)
 
 static int cmp_all(void *binary, int count)
 {
+#if MD5_std_mt
+	int t, n = (count + (MD5_N - 1)) / MD5_N;
+#endif
+	for_each_t(n) {
 #if MD5_X2
-	return *(MD5_word *)binary == MD5_out[0][0] ||
-		*(MD5_word *)binary == MD5_out[1][0];
+		if (*(MD5_word *)binary == MD5_out[0][0] ||
+		    *(MD5_word *)binary == MD5_out[1][0])
+			return 1;
 #else
-	return *(MD5_word *)binary == MD5_out[0][0];
+		if (*(MD5_word *)binary == MD5_out[0][0])
+			return 1;
 #endif
+	}
+	return 0;
 }
 
 static int cmp_one(void *binary, int index)
 {
+	init_t();
 	return *(MD5_word *)binary == MD5_out[index][0];
 }
 
 static int cmp_exact(char *source, int index)
 {
+	init_t();
 	return !memcmp(MD5_std_get_binary(source), MD5_out[index],
 	    sizeof(MD5_binary));
 }
@@ -203,10 +236,13 @@ struct fmt_main fmt_MD5 = {
 		SALT_SIZE,
 		MIN_KEYS_PER_CRYPT,
 		MAX_KEYS_PER_CRYPT,
+#if MD5_std_mt
+		FMT_OMP |
+#endif
 		FMT_CASE | FMT_8_BIT,
 		tests
 	}, {
-		MD5_std_init,
+		init,
 		valid,
 		fmt_default_split,
 		(void *(*)(char *))MD5_std_get_binary,
@@ -225,7 +261,7 @@ struct fmt_main fmt_MD5 = {
 		set_key,
 		get_key,
 		fmt_default_clear_keys,
-		(void (*)(int))MD5_std_crypt,
+		MD5_std_crypt,
 		{
 			get_hash_0,
 			get_hash_1,