hg2atp/01_hash_all_the_things.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hash all the things\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CID\n",
    "\n",
    "### Theory\n",
    "\n",
    "Link rot is a thing because URLs tell you where to find some particular content by addressing some entity providing it. \n",
    "If that entity goes away, the link is broken. \n",
    "\n",
    "Enter the [CID](https://github.com/multiformats/cid), or **Content IDentifier**.\n",
    "\n",
    "CIDs are a way to address content itself. If you have a CID, you can verify that the content you have is the content you want. This is\n",
    "generally useful in distributed systems because you can ask for content and *anyone* can return it to you. They could lie and give \n",
    "you something else, but the CID lets you verify cryptographically.\n",
    "\n",
    "This is useful to ATP for the same reason. All content shared has an associated, canonical CID. **No service assigns \n",
    "this CID, it is a property of the content itself.**\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Applied\n",
    "\n",
    "If you don't want to use python but have a CID, check out IPFS's [CID inspector](https://cid.ipfs.tech/#zdpuAx7GYAybGShxy9wvkK5eJt6a5G47tz5z5yeFcDqChfYE3)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from multiformats.cid import CID  # pip install multiformats\n",
    "from multiformats.multihash import get as get_hash_func"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Expanded, a cid is,\n",
    "\n",
    "```<cid> ::= <cid-version><multicodec><multihash>```\n",
    "\n",
    "where,\n",
    "\n",
    "- `<cid-version>` refers to the verison of the CID spec used\n",
    "- `<multicodec>` refers to the codec used to encode the content. [there are lots of them](https://github.com/multiformats/multicodec/blob/master/table.csv).\n",
    "- `<multihash>` refers to the hash algorithm used to generate the hash\n",
    "\n",
    "Imagine we have some data,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The original data\n",
    "hw_data = b\"hello world\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now imagine we had some magic blob store referenced by this. So long as we all agree to use sha2-256 everywhere, this would work. \n",
    "\"Here is my digest, give me the data.\" But that's kinda a fairy tale because we don't all agree and technology changes for good reasons. \n",
    "So we need something a bit more robust. \n",
    "\n",
    "A decent start would be to use a [multihash](https://github.com/multiformats/multihash), \n",
    "which is a self-describing hash,\n",
    "\n",
    "```\n",
    "multihash ::= <hash function code><digest size><hash function output>\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1220b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hw_digest = get_hash_func(\"sha2-256\").digest(hw_data).hex()\n",
    "hw_digest"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But that doesn't solve the problem completely. Now we can negotiate the shared\n",
    "algorithm, but how is the digest encoded? Is it hex? Base64? Base58? Something else?\n",
    "There is still ambiguity. Moreover, I'm assuming we're just sharing bytes. But\n",
    "what if we were sharing json documents? We need some way of agreeing how \n",
    "to encode the data."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "This motivates the CID. Expanded, a cid is,\n",
    "\n",
    "```<cid> ::= <cid-version><multicodec><multihash>```\n",
    "\n",
    "where,\n",
    "\n",
    "- `<cid-version>` refers to the verison of the CID spec used\n",
    "- `<multicodec>` refers to the codec used to encode the content ([there are lots of them](https://github.com/multiformats/multicodec/blob/master/table.csv))\n",
    "- `<multihash>` refers to the hash algorithm used to generate the hash\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'base58btc - cidv1 - raw - (sha2-256 : 256 : B94D27B9934D3E08A52E52D7DA7DABFAC484EFE37A5380EE9088F7ACE2EFCDE9)'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hw_cid = CID(\n",
    "    base=\"base58btc\", \n",
    "    version=1,\n",
    "    codec=\"raw\", \n",
    "    digest=hw_digest\n",
    ")\n",
    "\n",
    "hw_cid.human_readable\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert hw_cid.digest.hex() == hw_digest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, I can ask the magic blob store if it has the data for the following CID instead,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'zb2rhj7crUKTQYRGCRATFaQ6YFLTde2YzdqbbhAASkL9uRDXn'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "str(hw_cid)  # base58btc encoded, which saves some bytes and removes some ambiguity"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The magic blob store may decode the CID for some reason (maybe the internally don't use base58btc),"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CID('base58btc', 1, 'raw', '1220b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "CID.decode(str(hw_cid))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "but in any case it returns the data we requested. Which we can validate. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "returned_data = b\"hello world\"\n",
    "\n",
    "hw_cid.hashfun.digest(returned_data).hex() == hw_digest"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Extropolate from here:\n",
    "\n",
    "If you have a CID, you can get the data *from anyone who might have it*. \n",
    "\n",
    "- Trust isn't part of the process. They provide, you validate.\n",
    "- Any particular provider can fail so long as someone retains the data and provides it to you in some way\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Restated, the CID describes some `dag-cbor`-encoded content that has been hashed with `sha2-256`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'base58btc - cidv1 - raw - (sha2-256 : 256 : B94D27B9934D3E08A52E52D7DA7DABFAC484EFE37A5380EE9088F7ACE2EFCDE9)'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hw_cid.human_readable"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CBOR\n",
    "\n",
    "### Theory\n",
    "\n",
    "[CBOR](https://cbor.io/) is Concise Binary Object Representation. \n",
    "\n",
    "For our purposes, it's JSON in binary form. \n",
    "\n",
    "So think \"JSON\" but say \"CBOR\" and you're most of the way there."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Applied"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cbor2  # pip install cbor2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'\\xa2eactorq@generativist.xyzqburritos_per_week\\x02'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc = {\n",
    "    \"actor\": \"@generativist.xyz\",\n",
    "    \"burritos_per_week\": 2\n",
    "}\n",
    "\n",
    "as_cbor = cbor2.dumps(doc)\n",
    "as_cbor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'actor': '@generativist.xyz', 'burritos_per_week': 2}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from_cbor = cbor2.loads(as_cbor)\n",
    "from_cbor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DAG-CBOR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a CID for *something* produced on ATP,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'base58btc - cidv1 - dag-cbor - (sha2-256 : 256 : ADA48B7C8394D2855F97E9E47EC0EA63D57778E1AF14283EA1A0C2D6A86DC1A0)'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mystery_cid = CID.decode(\"zdpuAx7GYAybGShxy9wvkK5eJt6a5G47tz5z5yeFcDqChfYE3\")\n",
    "mystery_cid.human_readable"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We know what everything in that human readable format means \n",
    "except the `dag` part of the `dag-cbor` encoding. To motivate why we need `dag-cbor` consider the following,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from collections import OrderedDict\n",
    "\n",
    "order_1 = OrderedDict([('a', 1), ('b', 2)])\n",
    "order_2 = OrderedDict([('b', 2), ('a', 1)])\n",
    "\n",
    "cbor2.dumps(order_1) == cbor2.dumps(order_2)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CBOR does not have an ordered dict, so the order is arbitrary. (I enforced an order by using `OrderedDict` just to show this is true.) If you were to take a hash of the cbor encoded map, you would introduce a new source of fragility. `dag-cbor` removes this by [enforcing some strict rules](https://github.com/ipld/specs/blob/master/block-layer/codecs/dag-cbor.md#strictness) on what is allowed in the map as well as a canonical encoding.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import dag_cbor\n",
    "\n",
    "dag_cbor.encode(order_1) == dag_cbor.encode(order_2)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "psychonaut-hgjsIY8y-py3.10",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}