Check passwords against hacked password databases [$300]

[ghost]

haveibeenpwned.com has an API to check if passwords have been released in a previous dump.
They also provide a sha1-hashed download of their password dump at https://haveibeenpwned.com/Passwords .
Their API guideline is available here: https://haveibeenpwned.com/API/v2

Expected Behavior

Ability to check all or a selection of passwords against a password dump (either downloaded or online)

Current Behavior

Currently, KeepassXC does not check against a list of passwords that has been previously compromised.

Possible Solution (bash)

#!/usr/bin/env bash
PASSWORD=('p@55w0rd' 'P@55w0rd' 'password')
ONLINE=false
RELDIR="../pwnedPasswords"
OFFLINE_DATABASE=("${RELDIR}/pwned-passwords-1.0.txt" "${RELDIR}/pwned-passwords-update-1.txt" "${RELDIR}/pwned-passwords-update-2.txt")
check() {
	if ${ONLINE} ; then
		RESPONSE_POST=$(curl -s -o /dev/null -w "%{http_code}" --data "Password=${1}" "https://haveibeenpwned.com/api/v2/pwnedpassword")
		sleep 1.6
		RESPONSE_GET=$(curl -s -o /dev/null -w "%{http_code}" "https://haveibeenpwned.com/api/v2/pwnedpassword/${1}")
		RESPONSE=${RESPONSE_GET}
		sleep 1.6
		if [ ${RESPONSE} -eq 200 ]; then 
			echo "Bad password: ${1} (GET: ${RESPONSE_GET}, POST: ${RESPONSE_POST})"
		elif [ ${RESPONSE} -eq 404 ] ; then 
			echo "Good password: ${1} (GET: ${RESPONSE_GET}, POST: ${RESPONSE_POST})"
		elif [ ${RESPONSE} -eq 429 ] ; then
			echo "Too fast (wait >1.5s)"
		fi
	else
		if ! ${2} ; then
			CHECK=($(echo -n ${i} | sha1sum))
		else
			CHECK="${1}"
		fi
		LINES=$(cat ${OFFLINE_DATABASE[@]} | grep -i ${CHECK} | wc -l)
		if [ ${LINES} -gt 0 ] ; then
			echo "Bad password: ${1}"
		else
			echo "Good password: ${1}"
		fi
	fi
}

for i in ${PASSWORD[@]}; do 
	PASSWORDHASH=($(echo -n ${i} | sha1sum))
	check ${i} false
	check ${PASSWORDHASH[0]} true
done 

Note: Above script checks offline for security (slow). It can be modified to check online for testing.

Context

I would like to check for passwords that have been compromised in a breach or set of breaches.
I can currently do this manually with a modified version of the above script, but it is cumbersome and not as secure.

EDIT:
echo ${i} -> echo -n ${i}
Add note to script that it is meant for proof of concept and can be modified for offline use.
Modify script to check offline

[TheZ3ro]

I don't really like sending plaintext passwords over the web. Ok it's https but you need to really trust haveibeenpwnd. Sending sha1 is slightly better but still a no-no IMHO.

I think the best choice is to let user import a wordlist of pwned password and let KeePassXC do the comparison.

Also I will prefer this instead: https://haveibeenpwned.com/API/v2#BreachesForAccount

This is related to #551

[weslly]

Quoting haveibeenpwned.com/Passwords page itself:

Do not send any password you actively use to a third-party service - even this one!

[ghost]

I had been looking at using the downloaded password hashes, but I figured that online would be faster/easier to test against (the password hashes, when downloaded and unzipped are about 13 GB).

That is what I had intended on suggesting, but I haven't been able to figure out why my sha1sums are different from haveibeenpwned.com's hashes.

EDIT: needed to use echo -n instead of just echo.

[droidmonkey]

Honestly if you are using a randomized password greater than 12 characters the probability of being on that list are miniscule. It's not worth the effort to incorporate this. The entropy meter is far more useful.

[weslly]

Sending passwords to any third-party that isn't its own target website is just a really bad idea from a security standpoint and we shouldn't even offer the possibility to do so, IMHO.

But if you really want to do it, you can export your database to a CSV file and parse it with your script.

[mihaiile]

I would also vote against implementing this, it is not a good approach to send passwords/hashes of passwords to unknown websites on the web. Having the files locally would not help since they are huge and if you use a smaller version then what is the point in checking against a subset...

[ghost]

@weslly @mihaiile When testing, I don't like waiting 5 minutes to parse 13+ GB of text (especially when testing password). And yes, I agree that it is a really bad idea to check a third-party site directly.

@mihaiile I think it should be possible to implement a fast binary search on the text files (I believe they are in alphabetical order), if only by starting at byte (size / 2), reading to next newline, reading whole newline, checking if we need to go up/down, and going from there.

[phoerious]

SHA1-hashed passwords are basically plaintext. Never use SHA1 for passwords.

[TheZ3ro]

Just an update to this issue:
2 days ago Troy Hunt released v2 of his service pwnedpasswords.com with great improvements implementing k-Anonymity search.

Briefly, you perform locally the SHA1 of your password (that you want to check against the haveibeenpwned service), you take the first 5 characters of the hash and make an API request.
The service then respond with all the stored hashes that start with those 5 characters along with the count of how many times that specific hash is frequent.

If your local hash matches one of the returned one, your password SHA1 is already known and you should change it.
If not, the password is free to go.

Note that if you always use a newly random generated password of a discreet length this will be a no problem at all.

---

If you want to learn more check out "Cloudflare, Privacy and k-Anonymity" section in [1] or "How it works" in [2]

[1] https://www.troyhunt.com/ive-just-launched-pwned-passwords-version-2/
[2] https://blog.agilebits.com/2018/02/22/finding-pwned-passwords-with-1password/

[phoerious]

I am still absolutely against this. Such a database can only be useful in two cases:

• you use and re-use a trivial password (then you are using your password manager wrong)
• you are using a service that is saving your passwords in plain text (don't use them!)

In any other case, all that can leak to an attacker is a properly hashed and salted password and you can't provide a lookup database for that, so the service is useless.

But guess how such a database can suddenly be filled with useful passwords... By you sending them your password to check if it has been compromised. I don't want to discredit the publisher of that database, but do we know what they are doing with our passwords?

I strongly advise against sending out your passwords. EVER! Even if it's just the first five characters of the password's SHA-1 hash. SHA-1 is NOT a hash function suitable for password hashing and the fact that you cannot salt the hash, makes it really easy to pre-calculate rainbow tables. Or just use a GPU brute forcer. With a beefy GPU cluster, you can hash up to 8 billion passwords per second with a fast hash function like SHA-1 these days. For a completely random ASCII password (128 possible characters), ALL possible combinations of an 8-character password can be generated in about 104 days. Taking into account that you will find the correct hash on average after 50% of the time, that's 52 days. That still sounds like a lot (actually, it doesn't, but it's still a bit of time and money blown), but then, we are using a totally random password and if you feel like a database to check your password against is useful, you are probably using a lot less characters. With only English uppercase and lowercase letters and numbers, you crack an 8 character password in 7.5 hours (3.7 on average). Scary, right? So don't use those databases. Use a strong password instead! Actually, use many. You have a damn password manager!

[zatricky]

@phoerious you have a serious misunderstanding/under-estimation of how much thought has been put into the service. Pwned passwords uses k-anonymity. The password or it's hash is never sent to any 3rd-party. Ever:
Local computer makes a non-salted hash of your password, sends the first 5 characters of the hash to HIBP. HIBP responds with all the matching compromised hashes - which the local computer can then use to verify actual matches.

See
https://haveibeenpwned.com/API/v2#SearchingPwnedPasswordsByRange
https://en.wikipedia.org/wiki/K-anonymity

[droidmonkey]

@zatricky see #551

This application in no way whatsoever will ever silently send password hashes, usernames, or whatever over the network to be checked against a database. We will, however, offer multiple password checking (offline, possibly online too) services.

[zatricky]

"in no way whatsoever" and "possibly online too" contradict each other.

#551's initial issue description doesn't mention online services, which is why I hadn't posted there. The other mentions of HIBP have been shot down unceremoniously (and of course unfairly in my mind) as a bad idea.

I'll move further comments there however.

[TheZ3ro]

@zatricky The service is well thought, but It doesn't have much utility if you already use a password manager.

In a password manager you can generate very long random passwords every time (and you should), so the probability of one being leaked is pretty low and the leak needs to be plaintext since you can't crack a very long password easily if it's stored as hash.
We need to teach users to generate new long password every time and never re-use them, instead of avoiding known-bad passwords.
Also PwnedPassword suffer of true negatives, for example "password" is present but "password1" isn't and you will think that the latter is a safe password, but it's not.
More about it: https://keepassxc.org/blog/2018-02-24-pwned-database/

An integration with haveIbeenpwnd is much more usefull.

[zatricky]

Going through #551 I realised it is more about UI than anything else, whereas this topic is more about backend support for things like the HIBP service. Discussion re HIBP there would actually be off-topic/spammy.

I agree that you shouldn't need a service such as HIBP if you only use generated passwords. Legacy dictates however that most users don't exclusively use generated passwords and don't follow password hygiene processes (something #551 will help rectify).

I don't agree with your example as a weakness - in that it addresses "knownpass" but not "knownpass1". It's not ever intended to address "knownpass1". It's intended to address the fact that "knownpass" is a known password hackers can actively use in their password-guessing attempts.

[zatricky]

The concerns I see so far:

• sending passwords out to 3rd-party services
  Don't do that. That would be stupid anyway. HIBP has disabled that specific facility - and with the range lookup, passwords can be checked without sending the password
• doing things silently without user consent/knowledge
  That's nonsense. Why would you do that without user knowledge? Educate the user. Give them "Useful tools"*. If of course you realise that it's not sending passwords out, then by all means, do it silently and make the users safe. :P
• I don't like it
  ... that's not a good reason
• I don't want to do it myself
  ... then at least acknowledge that PRs will be given guidance toward acceptance

*Useful tools:

• User types/pastes a password into Keepass -> Give them an option to check it
• User imports passwords -> Give them an option to check it
• User generates a password using the random-pass-generator. Check it's entropy - but don't bother with stupid things like pw databases
• User runs the "hygiene wizard"/whatever-you-guys-come up with in Integrated Password Analyzer and Health Check [$170] #551: Give them an option to check the passwords

[rugk]

Mozilla announced it's integrating HIBP into (a) Firefox (extension?). And… drumm roll a competitor (i.e. a different password manager) is integrating this feature!

I've explained this in detail in issue #2073, but here are the main point's on how I consider such an integration a very nice thing:

• obviously, you can make it opt-in, like the favicon fetching that also uses network requests e.g.
• it's way of sending only a part of the hash and returning multiple results make it anonymous so that HIBP can not realistically get any sense of the password
• more services like NextCloud have already integrated it

So you may wonder why use it, if we already use a password manager?

1. I want to know, which services expose my account details. You may say these services in the list save passwords with bad hashes and so on. That's totally true and a good argument to integrate that feature: Before a breach nobody can know their security practices, so getting aware that my password has been breached is a good indicator for the "quality" of that service.
2. I use a password manager, but the service can still be get hacked. I.e. if the password get's leaked, one must still change that. That's what such an integration would help with.
3. I have a lot of mail aliases, so checking each one manually in the web version is just not feasible.

Bottom line is: Please reopen this issue. HIBP's API has been improved quite much, you already use network stuff for unimportant things (favicons), so you may use it for this here, too.

BTW everyone who wants this, upvote the initial issue.

[rugk]

As for the anonymity again, some extracts from Troy Hunt's blog, where he explained that in detail:

By looking at the hash prefix sent to the service, I have no idea what the password is. It could be any one of those 477 or it could be something totally different, I don't know. Of course, I could always speculate based on the prevalence of each password but it would never be anything more than that - speculation.
[…]In response, a collection of hashes is returned that match that prefix (477 on average). By looking at the hash prefix sent to the service, I have no idea what the password is. It could be any one of those 477 [i.,e. the breached ones, he obviously knows, but these are compromised anyway] or it could be something totally different[, because 5 chars of an SHA-1 hash say not really much about the content], I don't know.

The Cloudflare article get's deeper into the math:

Whilst for secure hashing algorithms it is computationally inefficient to invert the hash function, it is worth noting that as the length of a SHA-1 hash is a total of 40 hexadecimal characters long and 5 characters is utilised by the Hash Prefix, the total number of possible hashes associated with a Hash Prefix is 16^{35} ≈ 1.39E42.

(highlighting by me)
Yet again: The number of hashed passwords associated with a 5-char prefix is 16^{35}! If it takes a computer only one millisecond to guess one hash, it would still take 4,42 * 10^31 years. That's much more than the age of the universe…

Small FAQ from me:
But SHA-1 is not secure for passwords!
Yes, but that's not the point here. Of course for storing passwords it is not secure, but here it is just used as a way to not sent the first 5 plain-text chars to the server. These are way too low to reverse it.
But the password is not salted!
Indeed, that's the point here. Actually salting makes the password more unique, which is not what we want. We rather want the part of the SHA-1 hash to be most generalized, so it is the same as many other passwords. As Cloudflare says:

In essence, we turn the table on password derivation functions; instead of seeking to salt hashes to the point at which they are unique (against identical inputs), we instead introduce ambiguity into what the client is requesting.

Also, please don't get into the "I don't want this/have no use case for this. This feature must be crap." attitude. Obviously many people requested it and the initial reason to close this issue (i.e. SHA-1 with full hash) is no longer valid. Please re-open this.I thought KeePassXC wants to do better and integrate nice features compared to KeePassX.

[droidmonkey]

I want to integrate new features, but not feel-good snake oil. The website is literally a giant advertisement for 1Password. If you can't see that, look closer at how many "SIGN UP FOR 1PASSWORD" links there are on every page. The dude is probably getting kickbacks for each referral sign up.

[rugk]

Come on… you don't have to like that "dude", yes. I also don't agree with everything he says, but can we please get over such personal disagreements and rely on facts (I've demonstrated the math and use cases above and you can see it is certainly not snake-oil.). Also, do you think Firefox, NextCloud, etc. would integrate such a feature, if it would not be worth it? (Okay, there they likely also catch many users without password managers, but the principle of changing your password of hacked service does not change.)

Also, sure he mentions 1Password and also states he likes it – after all it is a personal blog, so that's totally okay –, but of course he also mentions Firefox there, so is it a giant advertisement for Firefox, too???
And as for your sign up statement, the only text/link on the site itself is, because of the Disqus comments:

But let me reiterate: This is a personal blog and of course he may like that service and feature it in other articles, but well… that totally does not matter for that feature here.
So please discuss the feature and not random persons. It does not matter at all, whether you like the person there or not.

[droidmonkey]

We can just agree that maybe it will be implemented, maybe in the password analyzer, but not a top priority.

[wolframroesler]

Thanks for the input again. Great community here :)

Now sorting the HIBP table in descending order by how many times a password has been exposed. Also shortened the title a bit (don't want to use more horizontal space than necessary, in both the table header and cells).

Won't use HIBP padding mode, it's not worth the cost (not even to add a checkbox to offer it as an option -- very few users would understand what the checkbox is about, and KeePassXC settings have more than enough checkboxes already.)

So, I'd call online HIBP checks done. Will submit a PR so you can check it out yourselves, and then continue with offline checks.

Stay healthy everyone.

[wolframroesler]

@droidmonkey How's the current planning for release 2.6.0? Would you like HIBP online checks in 2.6.0 already? Asking because from my point of view the PR could be merged, yet it's marked as WIP because offline checks aren't implemented yet. If you wish I'll un-WIP the PR so that online checks can go into 2.6.0, leaving offline checks for the 2.7.0.

[droidmonkey]

This will make 2.6.0. We are coming close to feature lock, perhaps in 2 weeks. Leave offline checks for another version.

[wolframroesler]

Awesome. PR un-WIPed. Looking forward to your review. Will submit a second PR for offline checks when they are done (probably not within the next two weeks).

[rugk]

Your BIOS password has been exposed "millions of times"… 🤣

[wolframroesler]

@rugk we really need the option to mark entries as "I know this is a bad password, you don't have to tell me anymore" (#4168), which would exclude them from Health Check as well as from HIBP. My ATM PIN has also been exposed millions of times (just like any other sequence of four digits) but there's nothing I can do about it, so there's no point in reporting it over and over again.

[OLLI-S]

I know that this topic is closed, but there are no column headers in the HIBP report:

Users don't know what the columns mean...

Edit
And I recognized that expired entries are also reported in the HIBP report...

[droidmonkey]

Oops that might be my fault

[OLLI-S]

I really appreciate your work, @droidmonkey and @wolframroesler .
You all at KeePassXC are really doing a fantastic job...

[wolframroesler]

@OLLI-S

And I recognized that expired entries are also reported in the HIBP report...

Of course, why should they not be?

[OLLI-S]

I have 116 expired entries in my database. All of them are for services, that I no longer use and some of them are services that are no longer available.
But I don't want to delete them, it might be the case that information stored there (the old problem: if you don't delete them, then you never need the data, but after you deleted them you need some information that is no longer present).

So I moved them all to a folder called "Abgelaufen" (what means "Expired") where I also set "Search" and "Auto-Type" to "Disabled" (so they are not found and not used for Auto-Type.
I did not want to move them to a new database (want to have everything in one file).

So these entries are old, mainly never used but it is disturbing if they inflate the health check.
So here I see the following possible solutions:

1. You implement a feature in the properties of a folder called "Health Check" where I can select that this folder should not be checked in the health check.
2. You implement an option "Exclude expired entries from health check (either in the settings of KeePassXC or in the Health-Check window)
3. You tell the maniac user that is writing these lines to move the expired entries into a new database. 😋

I really hope you implement 1. or 2. because I think there are other people that have expired entries and want to exclude them from the health check and HIBP checl.

[droidmonkey]

I like option 2, could also be a database setting that is stored in the custom data.

[droidmonkey]

See #4531 for fixes.

[OLLI-S]

When I wrote the last comment, then the 4th option did not came into my mind:

4. After you implemented Health Check - Exclude entries from check (Health Check - Exclude entries from check #4168) I just need to select the entries and exclude them from check.
   This requires that I can select multiple entries in the password list, right click and then exclude them all. So I will create a new issue for this idea....

I think this is the best and simple solution.

[OLLI-S]

I created the new issue Health Check - Exclude MULTIPLE entries from check (#4533)

[droidmonkey]

Locking this conversation as it is very long and multiple other issues have spawned from it.