Further de-duplication for rules cracking

[dogfud]

Great project, thanks for taking the time.

Food for thought .. typically when using hashcat I like to run through and pull out the straight matches, then switch to rules like Korelogic or the build-in set. To that end, having various permutations in the file reduces efficiency because the rules will catch them anyway .. example, having "password" in the list would suffice since "Password0", "p455w0rd" and "Pa55word" would all be generated by the most common mungers. Sure, rules on top of a munged version might produce more words, but there are better ways of layer rules on top of each other in a more deliberate way.

Anyway, as long as you are on the path of creating derivative password lists, one that is normalized for munging rules would be something to think about. For my purposes I just strip out the easy stuff -- tolower it all, strip off leading and trailing single digits, replace mid-stream digits with corresponding letters, etc.

cheers

$ egrep '^[Pp][aA4][sS5]{2}w[oO0]rd[0-9]{0,2}$' Top125Thousand-probable.txt | head
password
password1
passw0rd
Password
Password1
pa55word
password2
pa55w0rd
password12
password01
$ egrep '^[Pp][aA4][sS5]{2}w[oO0]rd[0-9]{0,2}$' Top125Thousand-probable.txt | wc -l
106

[berzerk0]

Thanks for bringing this up, I had thought about it and then forgotten about it again.

This doesn't fit the 'most popular' idea a full 100%. I'd say it fits it 75%, however.

My thinking is this:
When you mutate a password, especially a common one, you're LIKELY to create a common password.
It's not as guaranteed to be likely as if it were located frequently amongst password wordlists, but it is a solid second line of defense once you've exhausted the full list.
And if my understanding of hashcat is correct, if you run the Un-mutated list first, it'll remember your attempts on the hash file and therefore skip over any mutations you create in your second pass that uses rules.

Secondly, isn't it less computationally expensive to use a premade list that includes mutations than to generate them yourself while simultaneously cracking? This is not a rhetorical question, I am pretty new to this.

Long story short - Yes

...but isn't at the top of the priority list. I'd say it's most likely to come in after 2.0 but if working on 2.0 gets boring I'll work on it after 1.1

Edit - "Yes" became "No" after discussion and thought

Edit: or after 1.2

[berzerk0]

Now you've got me thinking about it in more depth.
So you'd want, let's say, the top 125thousand, 'purified' into something resembling dictionary words and strings of numbers?

That means de-1337-ifying, taking out leading and trailing symbols and numbers (if it is word based), etc?

[dogfud]

I think it really depends on the practical use case you are trying to support with the list, because really they are quite different from each other:

One would be an online brute-force authentication attack. For this you want an inclusive list of words, sorted by likelihood of use. This could be useful for attacking a single identity, or alternatively if the identities are enumerable then "trolling" for accounts with common passwords. Either way, rules probably not used, and besides John or Hashcat could be used to compute those from a base list if needed.

Second use case is an off-line brute force attack on a password hash. This is the use case I was speaking to because it probably does include rules-based munging, and to that end would gain some efficiencies by weeding out the redundant base words.

[dogfud]

You know, the more I chew on this I'm starting to think you are on the right track already, and that anybody using rules should just as well come up with their own regex filters to weed out the "noise" apropos of the specific rules they intend to apply.

Thanks anyway for entertaining the idea ;)

[Lexus89]

Perhaps this can be of use: https://github.com/digininja/deleet/tree/tuning

[berzerk0]

Decided to have this functionality implemented through use of third-party software.

Check out https://thesprawl.org/projects/pack for this kind of functionality.