-
-
Notifications
You must be signed in to change notification settings - Fork 736
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Option to detect hash collision #170
Comments
AFAIK, hash collisions start becoming a real problem if the number of hashes is about 2 ^ (N/2) with N being the bit length of the hash. So for sha256 that is 2 ^ 128 hashes. Each chunk is about 2^16 bytes (default, or likely bigger if you backup a lot of data and adapt --chunker-params). So this means that we talk about 2^144 Bytes of data in your backup. In decimal, that is a 1 followed by 43 zeros. I guess you'll have a lot of other problems before that happens, right? So the question here is really how much overhead one wants to add just to deal with such a improbable case. Currently we can not easily change the id_hash, but I have plans to make that more flexible (together with more flexible crypto). |
One check I could imagine (not sure if it is done already), is if we detect we already have some specific hash to additionally compare the length of the current input data to the length of the stored data that originally generated that hash. This would add a few bits (8..10?) of safety margin and is likely to be implementable rather cheaply (and also without significant runtime overhead). Relevant code: cache.Cache.seen_chunk and also .add_chunk. It does not compare the size yet. BTW, if chunk length matches, this also means either this was a relatively small file and the file size matches (whole file = 1 chunk) or that the rolling hash used for the content defined chunking triggered at the beginning and/or end of the chunk. This seems to add some more bits of safety than the mere length check. |
@pcgalando ^^^ see pull request. Guess this is all we should do here. |
detect inconsistency / corruption / hash collision, closes #170
…#170 added a check that compares the size of the new chunk with the stored size of the already existing chunk in storage that has the same id_hash value. raise an exception if there is a size mismatch. this could happen if: - the stored size is somehow incorrect (corruption or software bug) - we found a hash collision for the id_hash (for sha256, this is very unlikely)
Just to rephrase what @ThomasWaldmann said in a slightly different way: In 2014 the total amount of data produced in the world was 3.5 zettabytes (Whitby, Seagate). Even if you backed up every single byte of information humankind produced (hello NSA), it'd take 22300745198530623141535 years to have a 50% chance of SHA256 a collision. Let's meet here again in a billion years and review if any changes are needed. |
given the exponential growth of things, a change of hash algorithm is potentially helpfull in a few decades |
SHA-1 vs. SHA-256 and BLAKE2b-256 (the latter two are used by Borg, the
former isn't) are quite different. Collisions are usually reported and
generated against the plain hash function, while for SHA-2 we use the
HMAC construction, which is a different / more difficult target for
anything but "exhaustive search" (= brute force, or for 256 bits, brütal
förce).
…On 23.02.2017 17:04, Jeremie Miserez wrote:
Google can now generate SHA-1 collisions:
https://security.googleblog.com/2017/02/announcing-first-sha1-collision.html
Might be worth considering to reopen this.
|
Quoting myself: Update: |
No, this is not quite correct. Also note that this is something about ideal generic case...
So the "quality" of data set is often more important as its quantity. Thus your supposed 2128 can be abrupt 232 or even still smaller. As counterexample for your theory please see this article (more as 400 collisions were found for MM3-128 on 2 millions paths).
BTW, related "such a improbable case" - please see above. Which overhead exactly do you mean?
You say, as would it be an advantage, where in sense of "perfect" hashing over variable data-set with unknown quality it is really questionable (rather a disadvantage). Anyway, I would like to ask - how one can completely disable the deduplication? (because as long as suggested option is not available, I don't see it as safe and cannot really use borg for backup purposes). |
Borg stores data chunks into a key/value store using MAC(plaintext) as the key. So what is your suggestion for storage of plaintext that results in a MAC collision with a different plaintext? |
this was the changeset that closed this issue - it is doing an additional length-check comparing the current chunk with hash X with the length of the stored chunk with hash X. we can easily and efficiently do that, because we have hash and size in the local in-memory chunks cache and we do a lookup into that chunks cache anyway to detect whether we already have the data for hash X. a different length would mean we have a collision, but not vice versa, of course. otoh, the suggestion by @sebres to compare the current chunk data to the stored chunk data would be very expensive as the storage is often remote and the data would need transferring, authentication, decryption, decompression and byte-for-byte comparison. and this is not a rare case (only happening for collision scenario), but it would be needed for the very frequent real-duplicate scenario also - before you have done it, you do not know whether it is a duplicate or a collision. |
So and? I wanted just point to the subset issue by the hashing... Through certain "compression" of large number of bits to small number of bits (what the hashing doing), you'd restrict the whole data set (the hash can be applied to) to a sub set related to the quality of data set and its size and the hash size (no matter which function is used, MAC or whatever, as long as it is not PHF).
But do you not have the original chunks (as part of file it was created from)? If the encryption is deterministic, the same verification process is also possible with encrypted/compressed data (don't need to decompress/decrypt it). |
@sebres is wrong in his descriptions. Properties of the files have no effect on the chance of collision. And people are more generally misunderstanding how small the chance of collision is. Even with billions of chunks, the chance of collision is much, much, much smaller than the chance that an asteroid hits the earth and destroys all life in the next second. Borg does not need to worry about 256-bit hash collisions, and it would be a waste of effort to compare chunks against each other when the hashes match, something that happens very frequently when using borg. |
And you can surely explain (or better provide a mathematic proof) how depending to dataset properties expected 264 going to 218 in the practice.
Oh, really? |
All the comments above (and likely future comments below as well as people never learn) against collision detection use wrong assumption: if low probability then won't happen soon. If you roll a dice, the probability of "6" is less then the probability of "not 6". This doesn't mean that "6" won't happen soon, just that it is less likely than "not 6". Still "6" can happen on the very first roll. Now, if you have a dice with 2^256 sides, mechanics is the same: it's highly unlikely, but it may happen nevertheless. The "dice" does not know the results of previous rolls, nor is it anchored somehow in abstract "today", so any "soon" and "distant future" are irrelevant. So the real question is not whether it happens or not, but what to do when it happens. If one looses HDD, and restores all the data from Borg, some people may feel uneasy knowing that there's a non-zero chance that some of the restored files may be corrupt in a subtle way (like injection of random material inside critical data used to operate hazardous equipment, etc.), and there's no way to check that it didn't happen. Unless you took precautions, which users gathering their wisdom from incorrect answers on stackexchange do not take. One possible workaround would be to checksum all files Borg is going to backup, and include such manifest file to backup. On restore check files manually. It is inherently racy, but so is the backup process itself. It's really Borg's job, but let's point a finger: Restic doesn't do it too!. To be completely fair any checksum is probabilistic by nature due to pigeonhole principle, so the probability of Borg with 256-bit keys to inject random data in a file is exactly the same as the probability of collision of SHA256 for different files. However in latter case colliding file will unlikely be anything but random noise, while in the injection case it may still look structured and intact. Feel the difference.
Please take it easy :) |
Correction: I meant "the math is exactly the same", concrete values depend on number of chunks and files respectively. But the point is that probabilities are of roughly the same magnitude. |
Well, not quite correct - the math is more worse here, because of certain "compression" of chunk to the hash in non perfect way. Sure the number 2128 is enormous, but only if one really operates with 2128 entries (e. g. in case of perfect hash function), but that is not true in case of hashing over variable data set with SHA similar algorithms. But you are right, for someone which drive is currently lost and is discovering that some archive in the backup is "damaged" too it does not really matter how small the probability was exactly. |
(snip)
@sebres Disclaimer: I am also not a crypto expert. Regarding the above changeset mentioned by @ThomasWaldmann, doesn't this code mean that (perhaps excluding some even rarer edge cases) the only way borg will silently skip a new chunk with a hash that collides with an existing chunk is if-and-only-if the new chunk also happens to be exactly the same size as the existing chunk? I've tried to understand the concerns you've been trying to raise in this issue, and AFAICT so far you have been talking about probabilities only of a hash collision. For your concerns to be more relevant to the borg code wouldn't you also need to also consider the chunk size checks that borg conducts in the case where borg encounters a chunk hash that already exists? |
The function, used as checksum or else as key of the hash table, has basically nothing with cryptography either (and should also not be cryptographic strong). The distribution and dispersion quality is more important here.
Sure, and it'd decrease the probability of a collision (in about 221/2, so ca. 1448 times) by variable sizes of files smaller than 221. |
I've logged into Github and went through 2FA to reply to this thread, and typed a long reply, read it, and discarded it. I can't help but sum my experience reading your points up with a meme. You seem to imply a feasible way to generate collisions for HMAC-SHA-256 exists and also that a way to generate classes of inputs to HMAC-SHA-2 such that a large number of output bits can be controlled without knowledge of the key (which at this time not feasible even for measly HMAC-MD5, and also not for HMAC-SHA1, despite SHA1'ttered; that's because HMAC doesn't rely on preimage resistance or collision resistance to uphold its security). You say using a cryptographic hash for chunk IDs is unnecessary; but that is actually a paramount part of the design. By the way, a handy approximation for the probability of at least one random collision of a PRF with N output values and k inputs is k²/2N (for large k, N). For example, 1e12²/2**257 is ~4e-54. |
Just as explanation of coming question "why Inf?":
|
The question whether you get collisions hashing every possible chunk (yes, of course) is not relevant. The question whether it is possible to get a collision is not interesting, because obviously you can. The question of interest is how high the probability is. This question has been answered in detail in this issue and the mailing list several times now. |
OK, back to the school! |
Care to elaborate where that factorial is coming from? Remember, we are simply counting all possible chunks [1]. And yes, that formula is obviously not correct, though the error is quite marginal. [1] 256 1 byte chunks + 65536 2 byte chunks + 16777216 3 byte chunks + ... + 256^2e7 20 MiB chunks |
...if you use MAC(ciphertext) as a key, something that I never implied in my question. You said that MAC(plaintext) is an essential part. I ask why. Why can't we use just any HASH(plaintext) for the key? Why do we need authentication here?
It's absolutely not clear what you mean. I believe there's one-to-one correspondence between plaintext block of 256 bits and its ciphertext block of 256 bits (given fixed key and Nonce/IV of course), otherwise it's not clear how decryption would work.
I have read all the docs just recently, but somehow missed this part. Could you please provide more specific reference? Where exactly should I see?
For fixed key and Nonce - sure. For different Nonce - different ciphertext. Do you expect it some other way?
But how this protects against hypothetical chunk permutation bug in Borg? What I see is that individual chunks are authenticated (both before and after decryption), and that the list of the supposed order of chunks is also authenticated. What I don't see is any check (cryptographic or not) that Borg actually did assemble chunks in the required order (which would be the essence of the hypothesized bug). I think checksum of the whole file would provide such a check and also catch the case when an event of extremely low but non-zero probability actually realizes itself (I also fail to see why the exact probability computation is important as long as it is established that it is greater than zero, but if you people have fun, then why not?). But let's do it one step at a time: why can't we use just any HASH(plaintext) for the key? Why do we need it to be MAC? |
OK, let us forget the permutations of N distinct objects here, so let change factorial to sum, it would simply change nothing, because it is not important at all - 10108 is the same as 10100000000, this number has 100M zeros and although it is still not comparable to a Googolplex, but many many times larger as a Googol.
So we are speaking about Googol1000000. Just repeat above-mentioned number (zeros of that) 1 million times. And now compare it with 2256, which is:
then you'd estimate how many chunks with collisions theoretically exist by hashing using 256-bit hash. As long as you cannot do that the decision for the issue should be obvious. |
Ah, I see. Borg is designed to be deduplicating and assumes f(chunk) == f(chunk') => chunk == chunk'. Using this as a key is the reason it is there. That's why it needs to be a cryptographic hash (to avoid your backups being malleable by having collisions in that particular f); not necessarily a MAC (more below). This immediately implies that the system is probabilistic -- at least on paper -- since the domain of f() is much larger than its image.
It's not used for authentication; if you replaced HMAC() with HASH() (plain SHA-2, say) in an encrypted repository it would still be secure (because the manifest is HMAC'd, and contains the hashes of the archives and so on).
For a block cipher you're absolutely correct. But when we are talking about a plaintext or a ciphertext in this context, we (I) didn't mean a single block, but rather the entire chunk, up to ~20 MiB in length. In Borg's case AES-CTR is used to encrypt that, using a global counter as a nonce (so it actually isn't deterministic w.r.t. to the plaintext). A different scheme might use a random nonce; every time you'd encrypt the same plaintext, you'd get a different ciphertext. Yet another scheme might be fully deterministic, same plaintext results in always the same ciphertext.
Yup.
Changing the ordering of chunks in a file would be a very specific attack/change (e.g. from a file consisting of chunks 1, 2, 3 to it consisting of chunks 1, 3, 2 ; each number representing a different (HMAC-)SHA-256 value). The easier to think about attack is to adversarially generate a chunk that has the same ID as another chunk. E.g. replacing the contents of /bin/sudo with a backdoored version.
Yes, storing a separate hash of the entire file contents would be possible for Borg to do. However, verifying H(H(part1) || H(part2) || ...) implies the same confidence as verifying H(part1 || part2 || ...) for all practical intents and purposes, unless the particular H used is broken. |
I too can't follow the factorial part (yep, I didn't pay enough attention at school), however even if we consider just default minimal chunk size of 512KiB = 524288 bytes = 4194304 bits, there are 2^4194304 different 512KiB chunks that map to 2^256 different keys. So each key has at least 2^(4194304 - 256) chunks targeting it. Like with pages of Tolstoy's "War and Peace", most of them are just noise, but some do have meaning, and to my understanding nothing in the Universe prevents them to appear in a single backup.
I have to refer you to docs:
So no clients trying to exploit hash malleability.
Well, actually it is used for authentication (see two instances of
Doesn't Borg encrypt it? IIRC MAC(plaintext) is inside ciphertext (and if not it should - some security experts believe that MAC(plaintext) may reveal some information about plaintext - or so I heard).
I'd rather say that it is a deterministic process parametrized by encryption key and Nonce/IV. It simply couldn't be otherwise.
This is simply impossible with CTR mode (Nonce reuse flaw), and highly undesirable in any other mode, so I don't view it as a possibility. The very purpose of Nonce/IV is exactly to avoid this as much as possible.
I was talking about a hypothetical bug, you again read more than there was.
...or Borg has a bug. However I don't agree with the claim "for all practical intents and purposes", and haven't seen much of a justification from your exchange with Sebres (or I wasn't able to comprehend it), so we are back to where we started. Thanks for elaborations! :) |
The execution environment is trusted, but we do assume that the attacker is able to place arbitrary files in your backup input set, because that is a very common and easy to achieve capability (e.g. web server with a file upload form somewhere).
No, MAC(plaintext) is the "public" ID of the object, that will eventually end up at Repository.put. Yes, a MAC in general may reveal information about its input. HMAC provides guarantees beyond those required of a MAC; it's a PRF (pseudo-random function; making some assumptions about the underlying hash function) and does not leak anything about the input to the output.
The immediate purpose of not reusing a nonce/IV is to use different keystreams for different messages (in the case of block-based streamciphers like AES-CTR or ChaCha20; similar idea for other modes). As you noted, different modes react differently to nonce/IV reuse. It is not outright trivial but not particularly hard to create deterministic schemes such as AES-SIV. One concern with these is that, since ciphertext is strictly a function of key and plaintext, encrypting the same plaintext multiple times reveals that the ciphertexts belong to the same plaintext (this is called IND-CPA, ciphertext indistinguishability; no deterministic scheme can provide IND-CPA). This is not an issue for Borg, since in that case the IDs would be the same anyway, which we tell the server openly. And also, because of deduplication, we wouldn't generally encrypt the same plaintext twice (except maybe around borg --repair etc.). AES-SIV btw. uses CTR mode internally; the CTR nonce is essentially the MAC of the plaintext.
It pretty much comes down to what we expect of a cryptographic hash: Given an unbroken cryptographic hash function H and two messages msg1, msg2, then |
I contradict expressly this sentence - I had read it in original and it is great and even vintage classic from first to last byte. :) As for the relation N(chunks) vs. N(hashes)... the adherents of "use hash as unique ID without data comparison" (so |
Sadly no designer of any content addressed storage system ever stopped to consult the Senior Expert Software Developer :) Edit: While the discussion with kroki is interesting (but perhaps something better suited for the mailing list) the original issue #170 has been discussed to death even before the current iteration of this ticket and no merit will be found in beating a dead horse (or entertaining an ob(li)vious troll) any further. [Shouldn't you be using svnhub instead? 🤔] |
Tell me still one that operates with backup data. If you mean git and co: then it is pretty simple - yes the assumption is there, but a collision is simply not important: If two revisions in git have the same hash, it would treat those commits as identical. In case of broken backup you've no chance to repair it without original data (which you're missing if you currently need a restore process). |
Oh I don't know, Centera, Venti, IBM Tivoli, Acronis, Veeam (how do you think client-side dedup works to reduce traffic?), duplicati, bup, restic, Tahoe lafs, tarsnap, IPFS... just off the top of my head. |
Oh come on. As for many proprietary products from of that list, you want really tell me, you know its internal architecture? OK. |
Apropos EMC: one customer of me uses Centera Cluster as a storage for his bank's core business, archiving billions of documents (transactions whatever) per year with a retention time up to 30 years. |
I find it plainly amusing that you are completely oblivious to the BER of processors, memory, storage and interconnects, not to speak of random and correlated failure rates of entire components. But hey, it's digital, right? It's always correct; neither random nor systematic failures occur.
There's that thing called "documentation". Vendors tend to explain how their software does things, because gasp that's relevant to applying the software.
Imagine trying to call someone liar for saying Centera is a CAS... literally the headline feature of the entire product. This will be my last reply to you; further attempts at trolling will not be met with a response. |
OK, I'll try to explain it with a combinatorial example (because I assume my math would be not understood again)... Firstly I'm definitely right with the implication that common value of all possible chunk variants (in all known "universes") affects the calculation, as well with the estimation of its impact to the probability calculation ultimately. So the enormous relation Googol1000000 to 2256 having huge negative impact to the chance of a collision (and significant enlarges the probability), see formulas below. As well as that the properties of chunks/files have the effect well (in fact also drastically, I can also explain that later).
Still again - Wrong. The number of all possible chunks is relevant in probability calculation (see below).
Your approximation is wrong too - simply because the number of every possible chunk (yes, of course) is relevant. To understand the correct estimation (or else all the dependencies here)... we can use following approximation: Let's play cards (we starting firstly with 32 classic cards):
The probability to find two cards of the same type (so catch a collision) in some 3 ( The explanation is simply, on iteration number...
The formula is also growing recursively (by If one take an attentive look at the formula, one would see that the result will grows if This is pretty well visible in the following tables:
Thus by constant The numbers speak for themselves and I can only repeat my statement: I can not provide you the exact estimation how large it can be right now (need a supercomputer, or should possibly try to calculate it in CUDA), but I hope you recognize the tendency and understand now the seriousness of the issue. For people that don't believe the formulas, I wrote a program estimating the probability (iterative) as well as using a recursive calculation (probability tree), which confirms the numbers (I'll provide the link as long as I get pushed it to GH). And if someone would still one time call me a troll, I'll find him :] |
Here are 2 program variants (in tcl but also in python) - https://github.com/sebres/misc/tree/id-hash-collision |
FYI, in #4884 I tried to summarize main points of this thread with the conclusion that only "weak" solution is possible, and that it's up to proponents of collision awareness to provide the patch. I'm not a Borg user and I won't volunteer, so let me unsubscribe :).
I'm ashamed to admit that it is still in my "to read" list.
Two valid points indeed.
You wrapped your mind around "attack" scenarios so tightly, that you do not see any other possibilities. The are accidental collisions in the world, without any regard to cryptographic hash properties. Just don't forget to keep that in mind ;). |
Said I'm gone - and back again! Just another point for @enkore:
True for some content-addressed storage system (I believe there are some that do check bit-by-bit, and there are some that at least do store full file checksum). However your generalization about encryption is not correct: the goal of cryptography is to protect against deliberate collisions. It does not, nor it really can (leaving aside quantum encryption) protect against me guessing your key by accident. If by accident it so happens that you sign some document with your signature, and I sign my with mine, and such signatures collide bit-by-bit, it's not a security issue. We wouldn't even know that they had collided. Unless we try to put both documents into one content-addressed storage keyed by signatures and someone tries to deliberately exploit the fact. I have the impression that your subconscious strong objection against everything Sebres says is that you perceive that from his claims it follows it is possible to deliberately find collisions, and that he knows how to do it. I didn't follow everything he said, but I'm sure it's not his claim (though it is true that there are ways to reduce search space when you deliberately trying to create a collision, Google used these in their "shattered" PDFs). Now I said all I could, gone :). |
I added also my summarizing #4884 (comment) there. |
Just to keep an idea that just came to my mind. It solves the storage part of the collision problem. For efficiency reasons, we could only apply it for efficiently detected collision cases (EDCC).
(d is the current to-be-stored data and d' is some data already stored in the repo. the different length is required to efficiently differentiate a collision from data deduplication, see: a912c02 ) We currently just raise a "oh, we're fscked!" fatal exception in that case (and, AFAIK, nobody has seen it yet). Instead of that exception, we could do this: Currently: Instead, we could: x would usually be nothing (a zero length byte sequence), but the code would never require that. In EDCC, x would be chosen to be some generated non-empty string until there is no hash collision any more (not with d' nor with any other stored data d''). We neither want to generate x+d so it results in a real duplicate nor so that it produces another accidental hash collision. Then we would need to store d+x together with len(d) (or len(x)). The storage layer would handle that transparently, adding x when storing and removing it when fetching. This should behave well until too much of the 2^256 key space is occupied (assuming H spreads well enough over that space). |
If you mean the math - you can simply download my py- or tcl-program (playing cards;) and try it by yourself with different number of K/k and N (don't try toooo large numbers;). |
#170 (comment) adding to that: The sequence of x values tried for collision handling of some problematic data d needs to be reproducible, because the same collision that happened for some backup also might happen again for all subsequent backups (if the problematic data d is still there). If x would be random, we would produce different d+x for each backup. |
Sure, because en masse the spreading by hash sieve affects more the chunks of equal size (there are many many times more colliding blocks in large chunks as in short chunks).
As already said it would marginal decrease a collision chance (see my last sentence #170 (comment)),
But unfortunately it does not (as every other adder+modulo hashing algorithm of the world). |
@sebres it sounds like you assume that the chunker cuts equal sized chunks if the file size is > 2^21. That is not the case, the chunking points depend on the file contents. There is a minimum and maximum chunk size though, for technical reasons (storage efficiency, buffer size). |
And "well enough" meant "well enough for our purposes" and the purpose is storing way less chunks than 2^256. |
If my idea sounds valid / doable, I would open a ticket for "solving storage of detected hash collisions", meaning the EDCCs. |
This does not really matter, because the max size alone make half of all possible blocks ever (5e108 from 10e108).
But as I already revealed above - the max size of chunk so the number of possible chunks (big
Accumulated to the table, the effect by increment of ALL possible (
Note that by max size 221 the number big So the "well enough" means either "well enough yet".
More interesting is the question what happens with two large files containing different content but building same hash Ha for first part of file 2^21 bytes, also if hashes for second part(s) Hb and Hc are different? |
Hash collisions are unlikely when using SHA256, but as the number of files/chunks increases so does the probability, most de-duplication system do some sort to sanity check; file based - check file size and possibly type, block based - either generate fingerprint in addition to hash or read the block already in archive storage (ie: paranoid mode option in Symantec PureDisk) to verify the archive data really matches new data before incrementing the link counter and discarding the new data. Easy to test functionality if you are able to change the hash function to md4.
The text was updated successfully, but these errors were encountered: