Applying the rule duplicate not working as expected

[tabularasa007]

Here's how to recreate the problem:

1. A simple rule file which nothing but the rule 'd' inside (=duplicate).
2. A simple dict with the lines

     a
     bb
     ccc
     ....
     tttttttttttttttttttttttttttttttttttttttt

Now do:
hashcat.bin --stdout -a0 testdict.txt -r testrule.rule

The result looks rather strange:

aa
bbbb
ccccgg
ddddeeee
eeeeeeeegg
ffffffffgggg
ggggggggoooooo
hhhhhhhhiiiiiiii
iiiiiiiiiiiikkkkkk
jjjjjjjjjjjjkkkkkkkk
kkkkkkkkkkkkoooooooooo
llllllllllllmmmmmmmmmmmm
mmmmmmmmmmmmmmmmoooooooooo
nnnnnnnnnnnnnnnnoooooooooooo
oooooooooooooooo
ppppppppppppppppqqqqqqqqqqqqqqqq
qqqqqqqqqqqqqqqqqqqqssssssssssssss
rrrrrrrrrrrrrrrrrrrrssssssssssssssss
sssssssssssssssssssswwwwwwwwwwwwwwwwww
tttttttttttttttttttttttttttttttttttttttt

[Chick3nman]

Confirmed as an issue, I can reproduce this behavior.

[Chick3nman]

@willcrozi Appears to be related to #3537, introduced at the same time but not fixed with the previous work.

[willcrozi]

Looks good to me. Apologies for introducing this. Thanks @tabularasa007 for reporting and @Chick3nman for fixing.

Tested as follows:

inputs:

$ cat testrule.rule 
d

$ cat gen-test-words
#!/bin/bash
REPS="${1:-1}" # $1: per pwd repetition count, defaults to 1
PW_MAX=256     # hashcat pwd length limit

len=1
while true; do
  for ch in {a..z}; do
    pwd=$(printf "${ch}%.0s" $(seq $len))

    # special case: script used as test input
    if (( REPS == 1 )); then
      printf "${pwd}%.0s" $(seq $REPS)
      printf "\n"
    else
      # hashcat drops input pwds above PW_MAX length
      if (( len > PW_MAX )); then exit 0; fi
    
      if (( (len * REPS) < PW_MAX )); then
        printf "${pwd}%.0s" $(seq $REPS)
        printf "\n"
      else
        # replicate hashcat behaviour when mangled pwd is too large
        printf "$pwd\n"    
      fi
    fi  

    ((len++))

  done
done

baseline:

$ ./gen-test-words 2 | head -n 300 | md5sum
ec050da89f32bd08bf92d18176ad6d30  -

broken:

$ git log --oneline -n1
8d044fa07 (HEAD, origin/master, origin/HEAD, master) Use -Wno-format-overflow cflags for unrar sources on non-clang compiler
$ { make clean; make; } > /dev/null

$ ./gen-test-words 1 | head -n 300 | ./hashcat -a0 --stdout -r testrule.rule | md5sum
93a82dc5052cc2ebd0715402dd3f4e28  -

fixed:

$ git log --oneline -n1
0f697fd66 (HEAD -> patch-1, Chick3nman/patch-1) Fixes #3596: bug in --stdout w/ rules
$ { make clean; make; } > /dev/null

$ ./gen-test-words 1 | head -n 300 | ./hashcat -a0 --stdout -r testrule.rule | md5sum
ec050da89f32bd08bf92d18176ad6d30  -

[willcrozi]

Thinking about it maybe it's a good idea to get rid of BUF_SZ altogether now it's not really used anywhere, i.e. replace (on line 99):

  #define BUF_SZ (PW_MAX / sizeof(u32))

  u32 plain_buf[BUF_SZ] = { 0 };

with just:

  u32 plain_buf[PW_MAX / 4] = { 0 };

It's only used once and the _SZ suffix leads one to think it's a byte sizing and could be used with things like memset.

[Chick3nman]

Thinking about it maybe it's a good idea to get rid of BUF_SZ altogether now it's not really used anywhere, i.e. replace (on line 99):

  #define BUF_SZ (PW_MAX / sizeof(u32))

  u32 plain_buf[BUF_SZ] = { 0 };

with just:

  u32 plain_buf[PW_MAX / 4] = { 0 };

It's only used once and the _SZ suffix leads one to think it's a byte sizing and could be used with things like memset.

I considered going through and trying to reduce the variables but I don't think it really does anything to help clean up the code. Up to you if you think it's better.

[l3clelVl]

I'm still experiencing an issue with duplicate (d) and --stdout. Is there a workaround? Below is how to replicate:

┌──(kali㉿kali)-[~/Desktop]
└─$ echo -e ":\nd" > duplicateword.rule

┌──(kali㉿kali)-[~/Desktop]
└─$ cat duplicateword.rule
:
d

┌──(kali㉿kali)-[~/Desktop]
└─$ echo -e "The\nquick\nbrown\nfox\njumps\nover\nthe\nlazy\ndog\n0\n1\n2\n3\n4\n5\n6\n7\n8\n9" > SingleWordsAndNumbers.txt

┌──(kali㉿kali)-[~/Desktop]
└─$ cat SingleWordsAndNumbers.txt
The
quick
brown
fox
jumps
over
the
lazy
dog
0
1
2
3
4
5
6
7
8
9

┌──(kali㉿kali)-[~~/Desktop]
└─$ hashcat -r duplicateword.rule SingleWordsAndNumbers.txt --stdout | more
The
TheTyu
quick
quickquic{
brown
brownbrowo
fox
foxfo}
jumps
jumpsjumw
over
over~er
the
thetle
lazy
lazyloy
dog
dogdg
0
00
1
--More--

[willcrozi]

I'm still experiencing an issue with duplicate (d) and --stdout. Is there a workaround? Below is how to replicate:

Hi, it seems @Chick3nman's fix above hasn't made it into a hashcat release yet. If you're running a hashcat release the workaround would be one of:

1. build from git master, or...
2. patch source of last release (6.2.6) with relevant fixes to src/stdout.c

If you go with option 2 above and use v6.2.6 as baseline you'll need: 6d3cf36, c23ee33, and 0f697fd

What output does hashcat --version give?

[l3clelVl]

Thank you for the follow-up and the options!

My output:
┌──(kali㉿kali)-[~]
└─$ hashcat --version
v6.2.6

As for my use case and anyone in a similar situation: I went with a short python script, because I realized the "-d" wouldn't output iterative permutation for each word. [https://github.com/l3clelVl/HomeGrown/blob/main/Python/IterativelyPermutateEachWordIntoList.py]

import sys

from itertools import permutations

if len(sys.argv) != 3:
    print("Usage: python generate_combinations.py <input_file> <output_file>")
    sys.exit(1)

input_file = sys.argv[1]
output_file = sys.argv[2]

# Create a list of words from the input file
with open(input_file, "r") as file:
    words = file.read().split()

# Create a new file for the combinations
with open(output_file, "w") as newfile:
    for word in words:
        newfile.write(word + "\n")

    for word1 in words:
        for word2 in words:
            newfile.write(word1 + word2 + "\n")

print(f"Combinations generated from {input_file} and saved to {output_file}.")