--regen-lost-salts option: Bug fixes, closes openwall#4517

Since every salt has a copy of the initial salt->list pointer, a special
fixup is needed after removing the first binary in the list in
crk_process_guess().  This was the actual openwall#4517 issue.

Other fixes:
The generated salts written to the salt database lacked some info and the
total number of salts wasn't updated.  From now we copy the whole initial
database struct, avoiding several current and at least some possible future
problems.
Improved the "Remaining xx hashes with yy different salts" output to be
more accurate/informative when this feature is used.
Error messages referred to a non-existing file in a non-existing directory.
Several comments were bogus, eg. referring to code that was long gone.
General code-style fixes (including a couple of barely related, made while
trying to understand the code paths involved).
Add bodges for --show to work with hashes cracked using this feature. For
this to work you need to supply the --regen-lost-salts option with --show
too.
Since we unfortunately give the exact format as one of the parameters to
--regen-lost-salts (a fairly stupid decision made 8 years ago), add code to
automatically pick that format wheneven a --format parameter isn't given,
or complain if it is and they don't match.
Minor updates to doc/Regen-Lost-Salts, eg. dropping references to things
that were deprecated eight years ago.

doc/Regen-Lost-Salts.txt

@@ -1,42 +1,43 @@
-***************************************************************************
-* JtR --regen-lost-salts code, written by JimF, 2012-2015, for use
-* within the salted dynamic formats.
-*
-* No copyright is claimed, and the software is hereby
-* placed in the public domain. In case this attempt to disclaim
-* copyright and place the software in the public domain is deemed
-* null and void, then the software is Copyright (c) 2012-2015 JimF
-* and it is hereby released to the general public under the following
-* terms:
-*
-* This software may be modified, redistributed, and used for any
-* purpose, in source and binary forms, with or without modification.
-***************************************************************************
-
-
---regen-lost-salts=type:hash_sz:mask
-
-This option will allow certain types (with short salts), to be found, when
-they are present in a set of raw hashes, but where we have lost the salt value.
-Normally, without the salt, JtR would not be able to do anything with the hash,
-and could not find the password.  In this mode, JtR will load the hashes, giving
-each a 'fake' salt (but all get the same salt).  JtR then builds all of the
-possible salt values for this format, and associates every hash with each one
-of these salts.  So, JtR will now run, and find passwords using all possible
-salts for each of the hashes, thus recreating the salt (and finding the passwords).
-
-This function has recently been re-written.  The prior way of usage is still supported
-(limited), and should be considered depricated.  See the end of this document for
-the *** NOTE section.
+Salt brute-force feature:  --regen-lost-salts
+
+This option will allow hashes to be cracked even where the salt, normally
+stored together with the hash, was somehow lost.  Normally a cracker would not
+be able to attack such hashes.  Using this feature, only supported for dynamic
+formats (including "dynamic compiler" ones), JtR will load the hashes and
+brute-force all possible salts for each.
+
+Obviously this comes at a price - but not always:  If you attack a single hash
+with 1000 possible salts, it will end up with same same p/s as an attack on
+1000 uniquly salted hashes.  If on the other hand the number of hashes is
+larger than 1000, you'll end up with no penalty (well, until you've cracked so
+many that there are less than 1000 left to crack) and for very short salt
+sizes and/or large number of hashes, normal "same-salt boosts" may even apply.
+
+The feature should only be used with bare hashes input, meaning the ciphertext
+should be, for example, raw hex such as d3d9446802a44259755d38e6d163e820 without
+any tag such as the "$dynamic_xx$" prefix and obviously without the "$babe" salt
+suffix.  Other fields such as login/gecos are optionally supported though, just
+like they always are:
+
+	user:d3d9446802a44259755d38e6d163e820:101:100:Joe Random Luser
+
+All salts should be missing in the input file; Mixing missing-salt hashes with
+correctly salted ones in one same session is not supported.
 
 Usage:
+--regen-lost-salts=<type>:<hash_sz>:<mask>
 
---regen-lost-salts=dynamic_9:32:?d?d?d-   --format=dynamic_9
+Showing cracked hashes:
+Use --show as usual, but also include the full --regen-lost-salts parameters
+used while cracking them.  This is required in order to match the pot file
+entries (which have a format tag and the correct salt appended) to the bare
+hashes input.
 
-Note, regen has been updated to work with dynamic generic expression compiler also.
-Usage for same media-wiki as above is:
 
---regen-lost-salts='@dynamic=md5($s.md5($p))@:32:?d?d?d-'   --format='dynamic=md5($s.md5($p))'
+Detailed example:
+--regen-lost-salts=dynamic_9:32:?d?d?d-
+or, using dynamic compiler mode:
+--regen-lost-salts='@dynamic=md5($s.md5($p))@:32:?d?d?d-'
 
 The above command will run on a set of 32 byte 'raw' hashes, but which are known
 to contain media-wiki hashes.  Media-wiki is of the format md5(salt.-.md5(pass))
@@ -70,16 +71,19 @@ So that 'parts' of the --regen-lost-salts are:
             salt code will NOT work for salts this large.
 
 Here are more details about the mask value:
+- Note that these mask placeholders often differ from what is used by mask mode
+  and other parts of John.  For backward compatibility reasons, we are reluctant
+  to change this.
 - mask can contain static bytes. These will simply always be output. The example
   above did this with the '-' character in the last spot.
 - the mask can use different 'character class' bytes.  These are similar to the
   character types within JtR rules.  To set a 'class' for a specific byte location,
   simply use 2 characters, a question mark, and the class character.
   Here are the classes.
-    ??  - becomes a single ? char (2 question marks mean a literal question mark) 
+    ??  - becomes a single ? char (2 question marks mean a literal question mark)
     ?d  - decimal digits [0-9]  ** very common salt **
     ?l  - lower case letters [a-z]  (only ANSI lower case, does not take encodings
-          into account) 
+          into account)
     ?u  - upper case letters [A-Z]
     ?a  - upper / lower case letters [a-zA-Z]
     ?h  - hex lower case [0-9a-f]  ** common salt **
@@ -111,21 +115,11 @@ now, use this command line for john:
 --regen-lost-salts=dynamic_9:32:?1?d?d-   --format=dynamic_9
 
 This will use the user class-1 for the first byte, then digits for the other 2.
-This means our salt will start from 100- and go to 999- which skips 10% of the
-salts, and speeds things up by 10%.  This is due to the user-class-1 not having
-the '0' byte in it.
 
 
-At this time, only a fixed length salt is handled.  Doing a variable sized salt
-regen, simply adds too much complexity, and does slow things down, just a touch.
-This is different than the older --regen-lost-salts=3    This 'used to do 0- to 9-
-then 00- to 99- then 000- to 999-   The 'depricated' --regen-lost-salts=3 now only
-does 000- to 999- and would require running 2 more regen-salt runs to cover the
-entire salt range.
+An ancient version of the code used a single 'type' digit to specify what format
+to attack.  This still works but is deprecated:
 
-******************************************************************************
-
-*** NOTE, depricated method (still works, but should not be used)
 There are only a few types supported.  To properly run, you MUST use one of the
 proper types in the -format=type command, AND use the -regen-lost-salts=N with
 N being set properly.  The valid N's are:
@@ -147,7 +141,3 @@ find them using 'normal' methods.  This new functionality was added to handle
 leaked hashes where the original salt has been lost.  It is VERY slow to run
 in this manner, due to JtR having to check every possible salt, but it does
 allow these hashes to be cracked 'properly'.
-
-NOTE, the dyna-generic expression compiler is now supported, so dummy dynamic
-formats do not have to be hand coded simply to use the regen-salts logic on
-hashes where the salts were lost.

src/cracker.c

@@ -374,8 +374,7 @@ static void crk_remove_hash(struct db_salt *salt, struct db_password *pw)
 }
 
 /* Negative index is not counted/reported (got it from pot sync) */
-static int crk_process_guess(struct db_salt *salt, struct db_password *pw,
-	int index)
+static int crk_process_guess(struct db_salt *salt, struct db_password *pw, int index)
 {
 	char utf8buf_key[PLAINTEXT_BUFFER_SIZE + 1];
 	char utf8login[PLAINTEXT_BUFFER_SIZE + 1];
@@ -478,6 +477,23 @@ static int crk_process_guess(struct db_salt *salt, struct db_password *pw,
 	if (!(crk_params->flags & FMT_NOT_EXACT))
 		crk_remove_hash(salt, pw);
 
+	if (options.regen_lost_salts) {
+		/*
+		 * salt->list pointer was copied to all salts so if the first
+		 * entry was removed, we need to fixup all other salts.  If OTOH
+		 * the last hash was removed, we need to drop all salts.
+		 */
+		struct db_salt *next, *s = crk_db->salts;
+
+		do {
+			next = s->next;
+			if (!crk_db->password_count)
+				crk_remove_salt(s);
+			else if (s->list && s->list->binary == NULL)
+				s->list = s->list->next;
+		} while ((s = next));
+	}
+
 	if (!crk_db->salts)
 		return 1;
 
@@ -891,8 +907,7 @@ static int crk_password_loop(struct db_salt *salt)
 			if (crk_methods.cmp_all(pw->binary, match))
 			for (index = 0; index < match; index++)
 			if (crk_methods.cmp_one(pw->binary, index))
-			if (crk_methods.cmp_exact(crk_methods.source(
-			    pw->source, pw->binary), index)) {
+			if (crk_methods.cmp_exact(crk_methods.source(pw->source, pw->binary), index)) {
 				if (crk_process_guess(salt, pw, index))
 					return 1;
 				else {

src/dynamic_fmt.c

@@ -1011,16 +1011,16 @@ static char *prepare(char *split_fields[10], struct fmt_main *pFmt)
 
 	// at this point max length is still < 512.  491 + strlen($dynamic_xxxxx$) is 506
 
-	if (strncmp(cpBuilding, "$dynamic_", 9)) {
-		// ok, here we add the 'generic' regen salt code
-		if (options.regen_lost_salts && !strchr(cpBuilding, '$')) {
-			char *cp = load_regen_lost_salt_Prepare(cpBuilding);
-			if (cp)
-				return cp;
-		}
-		return split_fields[1];
+	// If --regen-lost-salts and salt is missing, add the first possible salt
+	if (options.regen_lost_salts && !strchr(cpBuilding + strlen(pFmt->params.label) + 2, '$')) {
+		char *cp = load_regen_lost_salt_Prepare(cpBuilding);
+		if (cp)
+			return cp;
 	}
 
+	if (strncmp(cpBuilding, "$dynamic_", 9))
+		return split_fields[1];
+
 	if ( (pPriv->pSetup->flags&MGF_SALTED) == 0)
 		return cpBuilding;
 
@@ -1101,22 +1101,20 @@ static char *prepare(char *split_fields[10], struct fmt_main *pFmt)
 static char *split(char *ciphertext, int index, struct fmt_main *pFmt)
 {
 	static char out[1024];
-	char search_char = '$';
 	private_subformat_data *pPriv = pFmt->private.data;
 
 	if (strnlen(ciphertext, 951) > 950)
 		return ciphertext;
 
-	if (!strncmp(ciphertext, "@dynamic=", 9))
-		search_char = '@';
-
 	// mime. We want to strip off ALL trailing '=' characters to 'normalize' them
 	if (pPriv->dynamic_base64_inout == 3 &&
 	    (!strncmp(ciphertext, "$dynamic_", 9) || !strncmp(ciphertext, "@dynamic=", 9)))
 	{
 		static char ct[496];
 		unsigned int len;
+		char search_char = (!strncmp(ciphertext, "@dynamic=", 9)) ? '@' : '$';
 		char *cp = strchr(&ciphertext[9], search_char), *cp2;
+
 		if (cp) {
 			++cp;
 			len = base64_valid_length(cp, e_b64_mime, flg_Base64_MIME_TRAIL_EQ_CNT, 0);