--left behavior on binary hashfiles with several hashes

[ngo]

Hashcat's --left flag prints the not-yet-cracked hashes from the input file. For binary hashfile formats (e.g. WPA2 -m 2500 .hccapx files) it correctly outputs binary data so that it can be fed back to hashcat again.

But the potfile_handle_left function that does this does not take into account that for binary hashfiles with multiple hashes they should not be delimited by a newline. E.g. if you want to combine two hccapx files you just concatenate them.

This results in the following. If you have a combined .hccapx input hashfile with two handshakes for different networks, the output of --left will be a file with the two handshakes separated by a newline.
When this file is later fed to hashcat, it will only recognise and load the first handshake.

So, the solution might be to either skip the newline output when handling --left for binary hash types or to make hashcat's input parsing more tolerant by allowing an additional newline between handshakes (though this potentially will have to be applied to all binary hash types).

[ngo]

It seems that potfile_handle_left calls outfile_write here: https://github.com/hashcat/hashcat/blob/master/src/potfile.c#L1065 and outfile_write unconditionally adds EOL here:
https://github.com/hashcat/hashcat/blob/master/src/outfile.c#L691

So, what would be the best strategy to patch this EOL addition?

One variant would be to accumulate a larger buffer with all the hashes in potfile_handle_left and doing one outfile_write in the end instead of calling outfile_write for each hash as it is done now.

[philsmd]

Thanks for the problem report. much appreciated. I can confirm it !

I think this is quite a rare situation when users use --left with binary hash types, but of course we should either support it or not allow it (and I think it was also allowed in the past, so we SHOULD fix it, I guess).

The problem as far as I just found out having a glance at the current source code is that the newline is always added here:

hashcat/src/main.c

Line 394 in d9e54cd

fwrite (EOL, strlen (EOL), 1, stdout);

(i.e. in the function main_potfile_hash_left from the file src/main.c)

so the problem is not only the src/potfile.c file (as you correctly mentioned above).

A simple fix like this would remove the newline in the case of a binary hash file:

diff --git a/src/main.c b/src/main.c
index d235db10..e7580f4e 100644
--- a/src/main.c
+++ b/src/main.c
@@ -387,11 +387,16 @@ static void main_potfile_hash_show (MAYBE_UNUSED hashcat_ctx_t *hashcat_ctx, MAY
 static void main_potfile_hash_left (MAYBE_UNUSED hashcat_ctx_t *hashcat_ctx, MAYBE_UNUSED const void *buf, MAYBE_UNUSED const size_t len)
 {
   outfile_ctx_t *outfile_ctx = hashcat_ctx->outfile_ctx;
+  hashconfig_t  *hashconfig  = hashcat_ctx->hashconfig;
 
   if (outfile_ctx->fp.pfp != NULL) return; // cracked hash was not written to an outfile
 
-  fwrite (buf, len,          1, stdout);
-  fwrite (EOL, strlen (EOL), 1, stdout);
+  fwrite (buf, len, 1, stdout);
+
+  if ((hashconfig->opts_type & OPTS_TYPE_BINARY_HASHFILE) == 0)
+  {
+    fwrite (EOL, strlen (EOL), 1, stdout);
+  }
 }
 
 static void main_potfile_num_cracked (MAYBE_UNUSED hashcat_ctx_t *hashcat_ctx, MAYBE_UNUSED const void *buf, MAYBE_UNUSED const size_t len)

but the problem is that we should really try to find a good/clean solution... and therefore we should decide if potfile_handle_left is responsible of adding the newlines or if the "EVENT" EVENT_POTFILE_HASH_LEFT -> main_potfile_hash_left in the file implementing the hashcat API should be responsible for the newline.

Therefore, I think we need a little bit of help/confirmation from @jsteube here.... for instance what I'm wondering is why we have this code section:

hashcat/src/potfile.c

Lines 1016 to 1025 in d9e54cd

	if (module_ctx->module_hash_binary_save != MODULE_DEFAULT)
	{
	char *binary_buf = NULL;
	int binary_len = module_ctx->module_hash_binary_save (hashes, salt_idx, digest_idx, &binary_buf);
	if ((hashconfig->opts_type & OPTS_TYPE_BINARY_HASHFILE) == 0)
	{
	binary_len--; // no need for the newline
	}

to me this means that in case of a binary hash format that has and uses the module_hash_binary_save function, we also check OPTS_TYPE_BINARY_HASHFILE is NOT set (i.e. it's not a binary hash file) and if true we remove one byte/char (which should be the newline). Remember: this doesn't really affect the final newline that is added, because the other one is in src/main.c which is added later on... to me this binary_len-- makes currently no sense, but maybe I'm just missing something.
Anyway, we could think of ADDING a new line in case it is NOT a binary file before the event is triggered:

hashcat/src/potfile.c

Line 1086 in d9e54cd

EVENT_DATA (EVENT_POTFILE_HASH_LEFT, final_buf[final_pos].hash_buf, final_buf[final_pos].hash_len);

IF and ONLY IF the format is not a binary hash format ?

So again, there is this question about who is responsible for the newline-add-logic ? The caller (and in this case at the same time the functions implementing the hashcat API) itself or the internal hashcat function responsible for the the potfile handling potfile_handle_left. I guess in theory it would make sense to say the potfile handler function should do the logic (even if the patch above should work in all of the cases), but of course there are always advantages and disadvantages of both approaches .

Thanks again

[ngo]

Considering binary hash formats, there is another problem:
take -m 6241 (truecrypt_ripemd160_aes_boot) as example. For this --left just prints out the filename:

% hashcat --left -m 6241 ./hashcat_ripemd160_aes_boot.tc 
./hashcat_ripemd160_aes_boot.tc

It seems that this module misses the custom module_hash_binary_save function and therefore cannot output binary in --left.

I've searched through modules and found that only the following have a custom module_hash_binary_save:

• 2500, 2501 - the hccapx case i mentioned
• 16800, 16801 - WPA PMKID, but the example hash format is text, not binary
• 22000, 22001 - WPA PMKID+EAPOL, also with textual example hash format

So it seems to me that currently this problem only affects -m 2500/2501 - for other binary hashes the binary output is just not supported.

The case with PMKID is rather strange - the hashes are non-binary (both the input format and the --left output), but the modules contain a custom module_hash_binary_save for whatever reason.

RE: usage of --left for binary hashes: I'm currently using this for an automated distributed cracking management setup as a distributed substitute for --remove (i.e. a node runs a wordlist on a hashfile, updates a centralized potfile as a result, and after that the central server makes a new hashfile with uncracked hashes using --left, which is then ready to be send to other rounds of cracking).

[ngo]

Thank you!