Skip to content

Commit

Permalink
Refactor use case 2 to add code
Browse files Browse the repository at this point in the history
  • Loading branch information
iknite committed Jun 3, 2019
1 parent d2b304c commit d0e8962
Show file tree
Hide file tree
Showing 7 changed files with 227 additions and 78 deletions.
Binary file added rtd_docs/docu.tar.gz
Binary file not shown.
19 changes: 11 additions & 8 deletions rtd_docs/source/home/quickstart.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,17 @@ For each step we will use the **QED CLI** facility.
The client will talk to the QED server and the snapshot store, so it must be
configured for that proposal.

The involved variables are the following ones, and we will use pre-defined
values for this quickstart.
.. note::

.. code-block:: shell
In production deployments, the following variables are required, and you
need to configure it, but for this quickstart we will use the
pre-defined values, so *you don't need to configure it for now*.

.. code-block:: shell
--api-key string Set API Key to talk to QED Log service (default "my-key")
--endpoints string REST QED Log service endpoint list http://ip1:port1,http://ip2:port2... (default [http://127.0.0.1:8800])
--snapshot-store-url string REST Snapshot store service endpoint http://ip:port (default "http://127.0.0.1:8888")
--api-key string Set API Key to talk to QED Log service (default "my-key")
--endpoints string REST QED Log service endpoint list http://ip1:port1,http://ip2:port2... (default [http://127.0.0.1:8800])
--snapshot-store-url string REST Snapshot store service endpoint http://ip:port (default "http://127.0.0.1:8888")
1. Environment set up
Expand All @@ -36,7 +39,7 @@ environment is as easy as:
.. code::
$ git clone https://github.com/BBVA/qed.git
$ cd deploy/docker
$ cd qed/deploy/docker
$ docker-compose up -d
This simple environment comprises 3 services: **QED Log server**,
Expand All @@ -51,7 +54,7 @@ Listing there these 3 services.

.. important::

To use the QED client using docker (and forget about installing golang -among other stuff-), do the following:
To use the ``qed_client`` command using docker (and forget about installing golang -among other stuff-), do the following:

.. code::
Expand Down
2 changes: 1 addition & 1 deletion rtd_docs/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ Contents

use_cases/commits
use_cases/documentation
use_cases/tweets
.. use_cases/tweets

.. toctree::
:maxdepth: 2
Expand Down
63 changes: 63 additions & 0 deletions rtd_docs/source/internals/why_is_secure.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@

Trust the untrustable
---------------------

Using today's technologies are by far trusted by default. A myriad of problems
can emerge in the neccessity to protect sensitive data, and even the when the
maximum level of isolation and protection are in place, you can always ask
**Who whatches the Watchmen?**

This trust dilemma is called in *distribution systems* a `Bizantine Fault
Tolerant`_ service.

.. _`Bizantine Fault Tolerant`: https://en.wikipedia.org/wiki/Byzantine_fault

Cryptographic proofs
--------------------

QED address this problem by using a internal tree storage that are
*statistically impossible* to alter without detection.

This is mainly because some inherent properties of the cryptographic
algorithms we use. From the original event source is fast and coherent to
create a *cryptographic hash* but **statistically impossible** to find other
input that could create the same output.

The other interesting property of the cryptographic hashers are the
**sparsity** of the hashes. This mean that similar inputs provides completely
different results, and the *distance* between those results are really wide.

This both properties are *abused* in QED in order to create a tamper evident
storage, even on untrustable environments.


Understanding the QED storage
+++++++++++++++++++++++++++++

QED stores all the transactions in a append-only tree. this allows us to track
the previous and future transactions that where sent to the QED server.

In order to prevent tamperings, we use a `Merkle tree`_. Which is a
cryptographic sum between adjacent elements in a tree fashion. This allows us
to make a lot of cryptographic hashes, between the last inserted elements and
all the previous ones.

.. image:: /_static/images/Hash_Tree.svg

Since the append-only storage can grow really fast, we need a way to find
previously inserted transactions, so we use another cryptographic tree, to
prevent tampering in finding the stored transactions.

How a Proof can be used
+++++++++++++++++++++++

Once a transaction is stored, we publish the final sum of all the cryptographic
nodes in a public, distributed storage.

If the need to prove that some transaction exists we return and audit path of
the current QED storage. Any alteration of the history will be evident and we
can't determine if the transaction is the same as it was included in the QED
server in first place. Only if the history is coherent the proof will be verified.

A final note along the auditable proofs is that it must be verified outside the
QED server in order to allow transparency.
19 changes: 19 additions & 0 deletions rtd_docs/source/notes.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
README.md:

Q.E.D in README.md http://127.0.0.1:8080/index.html#why-q-e-d
remove why, put features (Alvaro)

trust_model
quitar fondo (Alvaro)
flechas dobles

Internal a TODO

Use Case...

anadir link a quickstart
alon(g)side
qed command igual que en el quickstart

docu
fingerprint
92 changes: 68 additions & 24 deletions rtd_docs/source/use_cases/commits.rst
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
Commit certification
====================

In this Use Case we will discuss how to certify artifacts from source code,
In this use case we will show how to certify artifacts from source code,
building exactly what the developer intended to publish.

Theory and Operation
Expand All @@ -14,7 +14,7 @@ Not to mention third-party storages, like github.
In order to create the transparency we will need some actors being the **event source** that **auditors** will need to detect tamperings.

In this Use Case QED allows transparency in that regard, by allowing developers
publish both code and a fingerprint ``F1(SOURCE)`` (more on this later...), and
publish both code and a event ``F1(SOURCE)`` (more on this later...), and
the ``F2(BINARY)``.

.. image:: /_static/images/Uc1.png
Expand All @@ -26,6 +26,49 @@ The **DEVs** acts as event source for the transparency of the GIT REPO,
and the **BUILD** stage, creator of the artifact, acts as event source for
the transparency of the ARTIFACT REPO.

We will use two mapping functions ``F1`` and ``F2`` to identify the original
content as an unique **event**

Mapping Source Code to ``F1`` event
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

In orden to translate abstract content to a **QED event** we need to identify
what makes it unique, and how can we detect changes if the original content is
tampered.

For this event, the original commit hash, and a SHA256 of the files (excluding
.git folder) will provide a concise information that will change when even a
single character is changed.

.. note::

``F1`` event example:

.. code:: json
{
"commit_hash": "4b1a0b7be7b5982dc778e76adacbb6348632ff4d",
"src_hash": "b9261acdcc979434d37ed8211ad6014309752cb6a02705a40dc8dbaf9cdcd89b",
}
Mapping Binaries to ``F2`` event
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

For this event, the SHA256 digest of the binary, will be simple and good to
detect changes.

.. note::

``F2`` event example:

.. code:: json
{
"artifact_hash": "pcdcc979434d37e4b1a0b4309752cb6a0277c778e76adacbb6348632ff4d",
}
Auditors
++++++++

Expand All @@ -44,39 +87,40 @@ source as valid.
Creating transparency in a GIT repository
-----------------------------------------

.. important::
.. warning::

Visit 'Installation' and 'Quick start' sections before typing the following code snippets.
The following snippets are atop :ref:`Quick start`. please visit it to
configure the required code.

Creating a fingerprint is crucial to allow **auditors** generate trust around
Creating a event is crucial to allow **auditors** generate trust around
the repository that we need to rely on.

In this example we using the **commit_hash** and the **source_hash** to
univocally identify a source code.

.. code:: shell
# Create the source code fingerprint
# Create the source code event
commit_hash=$(git rev-parse HEAD)
src_hash=$(echo $(find . -type f -not -path "./.git/*" -exec sha256sum {} \; | sort -k2) | sha256sum | cut -d' ' -f1)
cat > fingerprint.json <<EOF
cat > event.json <<EOF
{
"commit_hash": "${commit_hash}",
"src_hash": "${src_hash}",
}
EOF
Alonside publishing to the git repo (or using a githook) now you can push the
fingerprint to QED.
event to QED.
.. code:: shell
# pushing the fingerprint to QED server
qed client \
# pushing the event to QED server
qed_client \
add \
--event "$(cat fingerprint.json)"
--event "$(cat event.json)"
Once the QED stores the fingerprint event ``F1(SOURCE)``, it will be verified
Once the QED stores the event event ``F1(SOURCE)``, it will be verified
and proved only and only if the code retrieved is exactly the same. This will prove
untampered once the ``BUILD`` stage fetch the source code from the git repo.
Expand All @@ -85,45 +129,45 @@ untampered once the ``BUILD`` stage fetch the source code from the git repo.
# Verify the proof
# please note the --auto-verify flag, without this flag the operation
# will returns the cryptographic proof
qed client \
qed_client \
membership \
--event "$(cat fingerprint.json)" \
--event "$(cat event.json)" \
--auto-verify
Creating transparency in the Artifacts Repository
-------------------------------------------------
Once we create the ``BINARY`` in the BUILD stage we can create the fingerprint
Once we create the ``BINARY`` in the BUILD stage we can create the event
``F2(BINARY)`` by using the content of the file.
.. code:: shell
# Create the artifact fingerprint
# Create the artifact event
artifact_hash=$(sha256sum archived/gin | cut -d' ' -f1 )
cat > bin_fingerprint.json <<EOF
cat > bin_event.json <<EOF
{
"artifact_fingerprint": "${artifact_hash}",
"artifact_hash": "${artifact_hash}",
}
EOF
And push the binary fingerprint to QED alonside to push the binary to the Artifact
And push the binary event to QED alonside to push the binary to the Artifact
repo. Ad you can see there is a repeating pattern of ``event-source -> [QED|Untrusted-source] <- auditor`` in the
way QED creates the transparency.
.. code:: shell
# pushing the artifact fingerprint to QED server
qed client \
# pushing the artifact event to QED server
qed_client \
add \
--event "$(cat bin_fingerprint.json)"
--event "$(cat bin_event.json)"
And Finally verify the proof.
.. code:: shell
# Verify the proof
qed client \
qed_client \
membership \
--event "$(cat bin_fingerprint.json)" \
--event "$(cat bin_event.json)" \
--auto-verify

0 comments on commit d0e8962

Please sign in to comment.