docs(guides/concepts): add concept docs for hashes

[0x-r4bbit]

License: MIT
Signed-off-by: Pascal Precht pascal.precht@gmail.com

[0x-r4bbit]

CI is failing but unrelated to my changes.

[Stebalien]

While we do need to make this as simple as possible, I want to avoid misleading users by oversimplifying. This is especially important for things like hashes (security implications).

I'd just:

1. Explain that a hash serves as a fingerprint or a unique (ish but we don't need to go into that) "name" for a piece of content.
2. Explain how we use hashes.
3. Link to what a cryptographic hash is.
4. Explain the properties we care about (maybe)? We don't actually need all the standard properties of a normal cryptographic hash. Really, we just need collision resistance. That allows us to, e.g., use the identity function as a "hash" function.

[Stebalien]

I'd like to be more technically accurate. When we say "hash", we usually mean a cryptographic hash and cryptographic hashes have a certain set of properties. While we need to avoid getting too into the weeds, I don't want to be technically incorrect. For example, a hash function returns a "hash", usually a number (not an alphanumeric string), but it could also be some other piece of structured information.

[Mr0grog]

a hash function returns a "hash", usually a number (not an alphanumeric string)

How about: “…returns a fixed-length value. The particular value depends on the given hash algorithm in use, such as such as SHA-1 (used by Git), SHA-256, or BLAKE2, but a given hash algorithm always returns the same value for a given input. Have a look at the full list of hash functions for more.”

When we say "hash", we usually mean a cryptographic hash and cryptographic hashes have a certain set of properties.

I think this is well-covered by the second section. What we should add there is a statement that says IPFS only supports cryptographic hashes for its content identifiers.

[Stebalien]

It may be computationally infeasible given current computers but that doesn't mean "theoretically unique". That can be trivially refuted with the pigeon-hole principal.

[Mr0grog]

What’s a better wording here, then? Someone unfamiliar with hashes but with good critical thinking is going to naturally wonder how we can use hashes to address things when the hash is so small (“why should I be confident two 20 megabyte files won’t hash to the same value?”). That’s an important question.

“Generally unique?” “Unlikely to match the hash of any other input?” “Uncommon?” “Rare?” (I’d like to avoid “collide” here, if we can, because it would be a little tautological.)

[Stebalien]

My issue was with the "theoretically" part. We can probably just say "unique" and then explain right after "it's believed to be computationally infeasible to generate two messages that have the same hash". Is that too technical?

[Mr0grog]

Oh, 👍 then. I misunderstood your concern a little. Let’s just drop the “theoretically” :)

[Stebalien]

Not necessarily a property of cryptographic hash functions.

[Mr0grog]

Yeah, bcrypt is a good counter-example. It’s ideal for its use-case specifically because it’s not fast 😛

We should just drop this point. I’m guessing you pulled this list from Wikipedia’s cryptographic hash page, @PascalPrecht? It looks similar. (I wonder if I should go edit that page…)

[0x-r4bbit]

Yes I did take that from Wikipedia as I couldn't remember the characteristics I read in a book a while ago :) Will update.

[Mr0grog]

CI is failing but unrelated to my changes.

Yeah, CI is broken for this project and waiting for ipfs-inactive/dev-team-enablement#42 to be fixed :\

[0x-r4bbit]

Thank you for your feedback @Stebalien !

Yea, this was really just a first take on this without really knowing the style of writing you're looking for in the concepts section.

Explain how we use hashes

Okay just to clarify though, this would go into CIDs and Multihash no? For that there'd be a separate document, so I'm not sure if we wanna overlap here. But maybe you mean it in a different way.

[Stebalien]

Okay just to clarify though, this would go into CIDs and Multihash no? For that there'd be a separate document, so I'm not sure if we wanna overlap here.

Good point.

[Mr0grog]

This looks like a really great start! I added a few notes inline, but the big thing that I think needs addressing here to to clarify how this relates back to IPFS as a whole (it’s kind of implied in the last bit of each section, but it could be more clearly stated).

Maybe something like this?

Hashes are a core part of how content is addressed in IPFS. Because they are short, deterministic, and relatively unique to the content they represent, IPFS can use them to effectively “name” a piece of content and then ask other IPFS nodes for the content with that name. That’s what allows IPFS to store content in many places, instead of having to address content by the location or computer it lives in, like traditional web addresses do. (In general, this concept is called “content addressing.”) For more on this, see CIDs [link to CID concept doc].

IPFS also uses hashes for plenty of other things, too (like securing and encrypting communications).

But if you’ve got some tweaks for that or a completely different and better way to write it, go for it :)

It might also be good to link to the Wikipedia page on cryptographic hash functions or something. In general, I’m trying to follow these guidelines for what to cover in a concept doc. (I shared these at the Berlin summit a couple weeks ago, but only just now got them posted here; you can be forgiven for not having had any guidelines to follow!)

P.S. Sorry for taking so long to review. I was more behind than I realized this week.

[Mr0grog]

a hash function returns a "hash", usually a number (not an alphanumeric string)

How about: “…returns a fixed-length value. The particular value depends on the given hash algorithm in use, such as such as SHA-1 (used by Git), SHA-256, or BLAKE2, but a given hash algorithm always returns the same value for a given input. Have a look at the full list of hash functions for more.”

When we say "hash", we usually mean a cryptographic hash and cryptographic hashes have a certain set of properties.

I think this is well-covered by the second section. What we should add there is a statement that says IPFS only supports cryptographic hashes for its content identifiers.

[Mr0grog]

I would just say:

As an example, the input:

[Mr0grog]

Should be “However, the exact same input generates…” (there’s no verb for “exactly” to describe). I think I’d also say “using SHA-256” or “with SHA-256” instead of “by” here.

[0x-r4bbit]

Thanks! Always happy to learn better english grammar :D

[Mr0grog]

We should either define “digest” at the top (alongside “hash,” since they are synonyms) or just stick with the term “hash” (probably simpler).

It might also be good to say “base 16” instead of “hexadecimal,” say “hexadecimal (base 16),” or to to say “hexadecimal” in the next paragraph, just to clearly link these two. People should generally know these are the same thing, but it’s a lot easier in general (and especially for non-native english speakers) to stick with the same terminology throughout.

[0x-r4bbit]

We should either define “digest” at the top (alongside “hash,” since they are synonyms) or just stick with the term “hash” (probably simpler)

Yep, I'll go with "hash".

[0x-r4bbit]

People should generally know these are the same thing, but it’s a lot easier in general (and especially for non-native english speakers) to stick with the same terminology throughout.

I have to admit, I just recently realised that base 16 is hexadecimal because I've never seen that wording being used like that in this context before. So I'll go with base 16 (or hexadecimal) here.

[Mr0grog]

Another example might be good here, e.g:

For example, the SHA-256 hash of “Hello world” from above can be represented in base 32 as:

mtwirsqawjuoloq2gvtyug2tc3jbf5htm2zeo4rsknfiv3fdp46a

---

An easy way to play around with these in JS is to use the multibase package:

const hashBytes = crypto.createHash('sha256').update('Hello world').digest();
multibase.encode('base32', hashBytes).toString();

(Just remember to strip off the multibase prefix :)

[0x-r4bbit]

So I think if we want to put code snippets into this concepts guide, we should make them "complete" as in, providing all the import statements needed etc.

E.g. in the code above it's unclear where crypto and multibase come from for someone who's not familiar with those APIs.

[Mr0grog]

Oh, whoa, I wasn’t trying to suggest we put that snippet in here at all. Sorry. That was a note for you. The only thing I was suggesting we add was the base 32 bit:

For example, the SHA-256 hash of “Hello world” from above can be represented in base 32 as:

mtwirsqawjuoloq2gvtyug2tc3jbf5htm2zeo4rsknfiv3fdp46a

[Mr0grog]

(From above) we should note why this matters: IPFS only accepts cryptographic hashes for identifying content.

[Mr0grog]

Yeah, bcrypt is a good counter-example. It’s ideal for its use-case specifically because it’s not fast 😛

We should just drop this point. I’m guessing you pulled this list from Wikipedia’s cryptographic hash page, @PascalPrecht? It looks similar. (I wonder if I should go edit that page…)

[Mr0grog]

What’s a better wording here, then? Someone unfamiliar with hashes but with good critical thinking is going to naturally wonder how we can use hashes to address things when the hash is so small (“why should I be confident two 20 megabyte files won’t hash to the same value?”). That’s an important question.

“Generally unique?” “Unlikely to match the hash of any other input?” “Uncommon?” “Rare?” (I’d like to avoid “collide” here, if we can, because it would be a little tautological.)

[Stebalien]

I think this is well-covered by the second section. What we should add there is a statement that says IPFS only supports cryptographic hashes for its content identifiers.

Some pedantic nitpicker (me) will come along and point out that we use the identity hash (for small pieces of data) which isn't a cryptographic hash. However, I don't know how to explain this without confusing the reader.

[0x-r4bbit]

Thanks @Mr0grog for the detailed feedback!

I wanted to give you guys just a quick update that I haven't forgotten about this one and am just busy atm but will send an update for this one tomorrow!

Thanks again for taking the time to write such detailed feedback, I appreciate it!

[Mr0grog]

Some pedantic nitpicker (me) will come along and point out that we use the identity hash (for small pieces of data) which isn't a cryptographic hash.

Ha, probably true. Maybe something like this?

In order to use hashes to identify content, they have to be short, deterministic, and relatively unique — so that means IPFS generally sticks to cryptographic hashes.

(Optionally then qualify that with “but any hash with those properties works“ or “for short enough content, the content itself can be used” if we really want, but it might be fine to just leave it — that satisfies the nitpickers without adding new, confusing info for someone who’s just learning how hashing works.)

Maybe this fits in with my suggested language above about how hashes relate to IPFS as a whole. We can figure a way to put this and that together that sounds good.

[Stebalien]

The shorter statement is probably fine.

[0x-r4bbit]

I'll commit my changes on top of this one before squashing them so you both can see exactly what changes I've applied (instead looking at a new rebased commit). Hope that's okay with you

[0x-r4bbit]

@Mr0grog I tried to address the "why it matters for IPFS" here, I'm not really sure what more to say about it here except for the fact that IPFS uses hashes to identify content and that due to the things given above, hases are a perfect fit for that.

Happy to change that.

[Mr0grog]

Hmmmm, I hear you. I think the other concept worth addressing here is why that’s important to a distributed system like IPFS. Someone who understands content addressing will probably already know that (or be able to work it out for themselves), but someone reading the basics of hashes here most certainly won’t really know much about content addressing.

So maybe adding something like this?

It turns out these features also mean we can use a cryptographic hash to identify any piece of data: the hash is unique to the data we calculated it from and it’s not too long (a hash is a fixed length, so the SHA-256 hash of a 1 Gigabyte video file is still only 32 bytes) so sending it around the network doesn’t take up a lot of resources.

That’s critical for a distributed system like IPFS, where we want to be able to store and retrieve data from many places. A computer running IPFS can ask all the computers near it whether they have a file with a particular hash and, if one of them does, they send back the whole file. Without a short, unique identifier like a cryptographic hash, that wouldn’t be possible. This technique is called “content addressing” — because the content itself is used to form an address, rather than information about the computer and disk location it’s stored at.

[0x-r4bbit]

@Mr0grog notice that I've added the require statements here just for completeness' sake

[Mr0grog]

OK. Sorry, I now realize I was really unclear with my earlier comment. (In my head, the horizontal rule was ending the main comment and starting a different discussion. Reading it now I see how that was not obvious.)

I’m not sure this whole bit with the code sample fits in really well here — we jump from some basic examples of hashes and their formatting to code that uses a whole bunch concepts we haven’t covered in the doc. That might be a little much.

[0x-r4bbit]

@Mr0grog @Stebalien okay, added another commit on top with changes that should integrate your feedback. Please let me know if there's more changes needed :)

Once this gets a green light I'll squash them and update the PR for merge.

[Stebalien]

This looks like a good place to start, clean and doesn't go into any unnecessary detail.

[0x-r4bbit]

No worries! I'll remove it and update the PR first thing tomorrow (it's late night here) Otherwise if this has to land today, feel free to apply the changes :)

…

On Mon, Jul 30, 2018, 11:58 PM Rob Brackett ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In content/guides/concepts/hashes.md <#81 (comment)>: > + +Notice that the second hash is longer than the first one. This is because SHA-1 creates a 160 bit hash, while SHA-256 creates a 256 bit hash. Also, the prepended `0x` is just an indicator that tells us that the following hash is represented as a base 16 (or hexadecimal) number. + +Hashes can be represented in different bases (`base2`, `base16`, `base32`, etc.). In fact, IPFS makes use of that as part of its [Content Identifiers](cid.md) and supports mulitiple base representations at the same time, using the [Multibase](https://github.com/multiformats/multibase) protocol. + +For example, the SHA-256 hash of "Hello World" from above can be represented as base 32 as: + +``` +mtwirsqawjuoloq2gvtyug2tc3jbf5htm2zeo4rsknfiv3fdp46a +``` + +An easy way to play around with these in JavaScript is to use the [multibase package](https://github.com/multiformats/js-multibase): + +``` +const crypto = require('crypto'); +const multibase = require('multibase'); OK. Sorry, I now realize I was really unclear with my earlier comment. (In my head, the horizontal rule was ending the main comment and starting a different discussion. Reading it now I see how that was not obvious.) I’m not sure this whole bit with the code sample fits in really well here — we jump from some basic examples of hashes and their formatting to code that uses a whole bunch concepts we haven’t covered in the doc. That might be a little much. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#81 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAbKsrj2SAIt_FoPxvLLOrOc5ceEtChlks5uL4GFgaJpZM4VX3re> .

[Mr0grog]

Ugh, sorry for the miscommunication on the code example. This looks pretty good to me, though I do think we could improve on the ending just a bit (see my comment inline).

[Mr0grog]

For someone new to hashes reading this document, I think this line is going to make no sense. If we keep this code sample, this line definitely need some rewording to explain what the multibase prefix is :\

[Mr0grog]

(But to be clear, I think it would be better to remove the code sample entirely. I’m just open to keeping it if you disagree.)

[Mr0grog]

Hmmmm, I hear you. I think the other concept worth addressing here is why that’s important to a distributed system like IPFS. Someone who understands content addressing will probably already know that (or be able to work it out for themselves), but someone reading the basics of hashes here most certainly won’t really know much about content addressing.

So maybe adding something like this?

It turns out these features also mean we can use a cryptographic hash to identify any piece of data: the hash is unique to the data we calculated it from and it’s not too long (a hash is a fixed length, so the SHA-256 hash of a 1 Gigabyte video file is still only 32 bytes) so sending it around the network doesn’t take up a lot of resources.

That’s critical for a distributed system like IPFS, where we want to be able to store and retrieve data from many places. A computer running IPFS can ask all the computers near it whether they have a file with a particular hash and, if one of them does, they send back the whole file. Without a short, unique identifier like a cryptographic hash, that wouldn’t be possible. This technique is called “content addressing” — because the content itself is used to form an address, rather than information about the computer and disk location it’s stored at.

[Stebalien]

Go to sleep. There's always tomorrow.

[Mr0grog]

Otherwise if this has to land today, feel free to apply the changes :)

Nope, there’s no rush on it. As @Stebalien said, get your rest!

[0x-r4bbit]

Alright, updated the PR and added the changes. @Mr0grog I could pretty much take your entire proposal in #81 (comment), just did some tiny little changes.

[Mr0grog]

(Rebased your commit off master so we get a nice preview and passing integration.)

[Mr0grog]

Added a minor URL and grammar fix, but now this looks ➕💯 👍

Will merge shortly unless there’s anything left you want to add, @PascalPrecht.

[0x-r4bbit]

Thanks @Mr0grog !

Will merge shortly unless there’s anything left you want to add,

This has now landed but there'd be two questions I have:

1. I think the fix commit could've as well been squashed into the first commit, so we prevent having that fix for a problem that was introduced in this PR commit in the overall history. Not super sure what the guidelines are for that... are you trying to keep a clean commit history? (not that a fix commit isn't clean, but the fix has been applied on something that was introduce in this PR)

2. Slightly related to that: The JS contribution guidelines for IPFS describe a particular commit message format: https://github.com/ipfs/community/blob/master/js-code-guidelines.md#commits I just noticed in this very moment that this project actually doesn't use them (sorry!) so I guess it would've been better if my commit looked more like the rest of the history... is this project moving to those guidelines at some point too, or shall I stick to what's there right now for future contributions?

I don't mean to be picky here (just in case it comes across like that), just trying to clarify things for myself so I can better help and contribute in the future :)

Hope this makes sense!

[Mr0grog]

I think the fix commit could've as well been squashed into the first commit, so we prevent having that fix for a problem that was introduced in this PR commit in the overall history.

Personally I’m not a big fan of squashing to that degree, especially when the commits in question are different people’s work. I prefer to be able to see the progression of effort in the final history, as long as it doesn’t make for a really confusing story (it can be really helpful to step through a series of well-factored commits to see how a feature evolved or was thought through, and to know what approaches were actually tried before and didn’t work out well). But I also don’t have super strong feelings about it and there’s no specific stance on this across the IPFS project, so you’ll probably find different maintainers and repos using different approaches. ¯\_(ツ)_/¯

I just noticed in this very moment that this project actually doesn't use them (sorry!) so I guess it would've been better if my commit looked more like the rest of the history…

Oh, no, this project should be using them and I’m glad you did; it’s only my own malfeasance in working too quickly to get this repo slammed together into something shippable that caused me to overlook those guidelines. You are all good 👍

[0x-r4bbit]

But I also don’t have super strong feelings about it and there’s no specific stance on this across the IPFS project, so you’ll probably find different maintainers and repos using different approaches. ¯_(ツ)_/¯

Totally! Didn't mean to be picky, was just wondering how you felt about it :)

it’s only my own malfeasance in working too quickly to get this repo slammed together into something shippable that caused me to overlook those guidelines.

Got it, thanks for taking the time everyone!