deploy: f7abe26

README.md

@@ -0,0 +1,99 @@
+Welcome to the Hydra Proof-of-Concept (POC) documentation.
+
+This technical documentation does contain some additional information about the
+architecture and inner workings of a `hydra-node` and the [Hydra Head
+protocol](https://eprint.iacr.org/2020/299.pdf).
+
+:warning: This project is still prototypical and exploratory work - it is NOT ready for production (yet). :warning:
+
+Thus, the documentation here is also a work in progress and is certainly not
+complete. However, we do want to improve it and would like to hear from any
+[questions](https://github.com/input-output-hk/hydra-poc/#question-contributing)
+you might have (so we can at the very least compile an FAQ).
+
+# Hydra Head protocol
+
+The greater [vision of
+Hydra](https://iohk.io/en/blog/posts/2020/03/26/enter-the-hydra-scaling-distributed-ledgers-the-evidence-based-way/)
+involves a whole suite of layer-two protocols to achieve greater scalability in
+many different use cases.
+
+The [Hydra Head](https://eprint.iacr.org/2020/299.pdf) protocol is one of them
+and forms the foundation for more advanced deployment scenarios and introduces
+isomorphic, multi-party state channels. This is also the protocol on which we
+focused most so far and implemented a proof of concept for.
+
+There exist various flavors and extensions of the Hydra Head protocol, but let's
+have a look at a full life cycle of a basic Hydra Head and how it allows for
+isomorphic state transfer between layer 1 and layer 2.
+
+![](images/hydra-head-lifecycle.svg)
+
+A Hydra Head is formed by a group of online and responsive participants. They
+**init** a Head by announcing several Head-specific parameters including the
+participants list. Then each of the participants **commits** unspent transaction
+outputs (UTXO) from the Cardano main-chain to it, before all the UTXO are
+**collected** and made available in a Hydra Head as initial state (**U0**).
+
+While open, they can use the Hydra Head via a `hydra-node` just the same as they
+would be using the Cardano blockchain via a `cardano-node` by submitting
+transactions to it (that's the **isomorphism** property). When UTXO are spent
+and new UTXO are created in a Hydra Head, all participantes are required to
+acknowledge and agree on the new state in so-called snapshots (**U1..n**)
+
+Any participant can **close** the Head using an agreed state, when for example
+they wish to use some UTXO on the mainnet or another party misbehaves or stalls
+the Head evolution. There is a mechanism to **contest** the final state on the
+main chain for a Head-specific contestation period, which a **fanout**
+transaction does distribute in the end.
+
+This is not the full picture though, as the protocol also allows to **abort**
+Head initialization and protocol extensions for incremental commits and
+decommits, as well as optimistic head closures (without contestation period) are
+possible.
+
+# Hydra Node Architecture
+
+We use _Architecture Decision Records (ADR)_ for a lightweight technical
+documentation about our principles and significant design decisions. The
+architecture itself then is just a result of all accepted ADRs, which have not
+been deprecated or superseeded. An up-to-date index of still relevant ADRs is
+kept [here](./adr/README.md).
+
+The following diagram represents the internal structure of the Hydra Node and the interactions between its components.
+
+![](images/hydra-architecture-direct.jpg)
+
+**Legend**:
+- Grayed boxes represent components which are not developed yet
+- Black boxes represent components which are expected to be used as _black box_, eg. without any knowledge of their inner workings.
+- Arrows depict the flow of data (Requests, messages, responses...)
+- We represent some components that are not part of the Hydra node proper for legibility's sake
+
+## Components
+
+> **TODO**: Move as haddock comments to each module/function
+
+Please refer to each component's internal documentation for details.
+
+* The [HydraNode](https://github.com/input-output-hk/hydra-poc/blob/d24c04e138acd333c3d47f97bb214957785fde08/hydra-node/src/Hydra/Node.hs) is a handle to all other components' handles
+  * This handle is used by the main loop to `processNextEvent` and `processEffect`
+* The [HeadLogic](https://github.com/input-output-hk/hydra-poc/blob/d24c04e138acd333c3d47f97bb214957785fde08/hydra-node/src/Hydra/HeadLogic.hs) component implements the Head Protocol's _state machine_ as a _pure function_.
+  * The protocol is described in two parts in the [Hydra paper](https://iohk.io/en/research/library/papers/hydrafast-isomorphic-state-channels/):
+    * One part detailing how the Head deals with _clients input_, eg. [ClientRequest](https://github.com/input-output-hk/hydra-poc/blob/d24c04e138acd333c3d47f97bb214957785fde08/hydra-node/src/Hydra/HeadLogic.hs#L43):
+    * Another part detailing how the Head reacts to _peers input_ provided by the network, eg. [HydraMessage](https://github.com/input-output-hk/hydra-poc/blob/d24c04e138acd333c3d47f97bb214957785fde08/hydra-node/src/Hydra/HeadLogic.hs#L78):
+* The [OnChain](https://github.com/input-output-hk/hydra-poc/blob/d24c04e138acd333c3d47f97bb214957785fde08/hydra-node/src/Hydra/Node.hs#L154) client implements the _Head-Chain Interaction_ part of the protocol
+  * Incoming and outgoing on-chain transactions are modelled as an [OnChainTx](https://github.com/input-output-hk/hydra-poc/blob/d24c04e138acd333c3d47f97bb214957785fde08/hydra-node/src/Hydra/HeadLogic.hs#L88) data type that abstracts away the details of the structure of the transaction.
+* The [Network](https://github.com/input-output-hk/hydra-poc/blob/d24c04e138acd333c3d47f97bb214957785fde08/hydra-node/src/Hydra/Network.hs) component provides the Node an asynchronous messaging interface to the Hydra Network, e.g to other Hydra nodes
+  * Incoming and outgoing messages are modelled as [HydraMessage](https://github.com/input-output-hk/hydra-poc/blob/d24c04e138acd333c3d47f97bb214957785fde08/hydra-node/src/Hydra/HeadLogic.hs#L78) data type
+  * We have a [ouroboros-network](https://github.com/input-output-hk/ouroboros-network/tree/master/ouroboros-network-framework) based implementation of the network component (we had another one using [ZeroMQ](https://zeromq.org/))
+    * The [Ouroboros](https://github.com/input-output-hk/hydra-poc/blob/d24c04e138acd333c3d47f97bb214957785fde08/hydra-node/src/Hydra/Network/Ouroboros.hs) based network layer implements a dumb [FireForget](https://github.com/input-output-hk/hydra-poc/blob/d24c04e138acd333c3d47f97bb214957785fde08/hydra-node/src/Hydra/Network/Ouroboros/Type.hs#L27) protocol. Contrary to other protocols implemented in Ouroboros, this is a push-based protocol
+* The main constituent of the Head's state is the [Ledger](https://github.com/input-output-hk/hydra-poc/blob/d24c04e138acd333c3d47f97bb214957785fde08/hydra-node/src/Hydra/Ledger.hs) which allows the head to maintain and update the state of _Seen_ or _Confirmed_ transactions and UTxOs according to its protocol.
+  * [MaryTest](https://github.com/input-output-hk/hydra-poc/blob/d24c04e138acd333c3d47f97bb214957785fde08/hydra-node/src/Hydra/Ledger/MaryTest.hs) provides a more concrete implementation based on a Mary-era Shelley ledger, but with test cryptographic routines
+* Structured logging is implemented using [IOHK monitoring framework](https://github.com/input-output-hk/iohk-monitoring-framework) which provides backend for [contra-tracer](https://hackage.haskell.org/package/contra-tracer) generic logging
+  * Each component defines its own tracing messages as a datatype and they are aggregated in the [HydraLog](https://github.com/input-output-hk/hydra-poc/blob/d24c04e138acd333c3d47f97bb214957785fde08/hydra-node/src/Hydra/Logging/Messages.hs) datatype. Specialized `Tracer`s can be passed around from the top-level one using `contramap` to peel one layer of the onion
+  * Configuration of the main tracer is done via the [withTracer](https://github.com/input-output-hk/hydra-poc/blob/d24c04e138acd333c3d47f97bb214957785fde08/hydra-node/src/Hydra/Logging.hs) wrapper function
+* Metrics and monitoring are piggy-backed on tracing events:
+  * Monitoring collection is configured at start of the hydra-node
+  * Traced events are [interpreted](https://github.com/input-output-hk/hydra-poc/blob/d24c04e138acd333c3d47f97bb214957785fde08/hydra-node/src/Hydra/Logging/Monitoring.hs) as contributing to some specific metric value without trace producers needing to be aware of how this process happens
+  * Metrics are exposed using [Prometheus](https://prometheus.io/docs/instrumenting/exposition_formats/) format over URI `/metrics` from an HTTP server started on a configurable port.

adr/0001-record-architecture-decisions.md

@@ -0,0 +1,29 @@
+# 1. Record architecture decisions
+
+Date: 2021-06-07
+
+## Status
+
+Accepted
+
+## Context
+
+We are in search for a means to describe our technical architecture.
+
+We are a small team working in a very lean and agile way (XP), so we naturally
+prefer also light-weight documentation methods which also accomodate change
+easily.
+
+## Decision
+
+* We will use _Architecture Decision Records_, as described by Michael Nygard in
+  this
+  [article](http://thinkrelevance.com/blog/2011/11/15/documenting-architecture-decisions).
+* We will follow the convention of storing those ADRs as Markdown formatted
+  documents stored under `docs/adr` directory, as exemplified in Nat Pryce's
+  [adr-tools](https://github.com/npryce/adr-tools). This does not imply we will
+  be using `adr-tools` itself.
+
+## Consequences
+
+See Michael Nygard's article, linked above.

adr/0002-reactive-core.md

@@ -0,0 +1,34 @@
+# 2. Reactive core
+
+Date: 2021-06-07
+
+## Status
+
+Accepted
+
+## Context
+
+We are looking for a way of expressing the Hydra Head protocol logic in a Hydra node.
+
+The Hydra Head protocol is defined as a _State machine_ in the paper, whose transitions are inputs that come from different sources which can emit outputs to other instances of the state machine or the mainchain. See the [FC2021](https://iohk.io/en/research/library/papers/hydrafast-isomorphic-state-channels/) paper for details
+
+It should also be easy to review / feed-back to researchers.
+
+We are familiar with React's [redux](https://react-redux.js.org/) way of structuring applications, which in turn is inspired by [The Elm Architecture](https://guide.elm-lang.org/architecture/) which itself is a simplification of [Functional Reactive Programming](https://en.wikipedia.org/wiki/Functional_reactive_programming) principles.
+
+We have experienced benefits with _Event Sourcing_ in the domain of persistence in the past
+
+## Decision
+
+Implements the Hydra Head core logic as a _loop_ that:
+1. Consumes _input events_ from an event _queue_,
+2. Applies each _event_ to the current _state_ yielding potentially an _updated state_ and a sequence of _effects_,
+3. Execute all _effects_.
+
+## Consequences
+
+The internal state is only ever changed through _Events_.
+
+The core state machine _transition_ function _is pure_ and reviewing it requires minimal Haskell knowledge.
+
+Side-effects are all handled at the level of the `Node`.

adr/0003-asynchronous-duplex-api.md

@@ -0,0 +1,32 @@
+# 3. Asynchronous Duplex Client API
+
+Date: 2021-06-07
+
+## Status
+
+Accepted
+
+## Context
+
+The [_reactive_ nature of the Hydra node](0002-reactive-core.md) means that
+clients produce a _stream_ of _inputs_ to a node which in turns issues a stream
+of _outputs_ representing the outcome of previous inputs or resulting from
+interaction with peers in the network.
+
+For example, a client may send a _command_ as _input_, upon which the node might
+do something. When that something is finished, a _output_ does indicate that.
+However, there might also be an _output_ emitted to the client when another peer
+interacted with "our" node.
+
+Queries, messages by clients which do only fetch information from the node, are
+not in scope of this ADR.
+
+## Decision
+
+* We use a single, full-duplex communication channel per client connected to a Hydra node
+* This is implemented using a simple [Websocket](https://datatracker.ietf.org/doc/html/rfc6455) with messages corresponding to `Input`s and `Output`s.
+
+## Consequences
+
+* Clients needing a synchronous API need to implement it on top
+* Clients can receive _outputs_ decorrelated from any _inputs_ and at any time

adr/0004-use-handle-to-model-effects.md

@@ -0,0 +1,44 @@
+# 4. Use Handle to model Effects
+
+Date: 2021-06-08
+
+## Status
+
+Accepted
+
+## Context
+
+Given we are structuring Hydra node as a [reactive core](0002-reactive-core.md) we need a way to ensure a strict separation of pure and impure (or effectful) code.
+
+We want to be able to test those impure/effectful parts of the code. This requires a means for exchanging the actual implementation for e.g. the function to send messages over a network.
+
+Also we want the ability to swap implementations not only for testing, but also be able
+to accommodate different usage scenarios, e.g. use a different middleware
+depending on peer configuration.
+
+In Haskell there are various common _patterns_ to model effects:
+  * [Tagless final encoding](http://okmij.org/ftp/tagless-final/index.html) also known as _MTL-style_ although using typeclasses to implement is [not necessary](https://www.foxhound.systems/blog/final-tagless/), whereby Effect(s) are expressed as typeclass(es) which are propagated as constraints
+  * [Free monads](https://reasonablypolymorphic.com/blog/freer-monads/), or any variant thereof like Eff, freer, extensible-effects, whereby effect(s) are expressed as ADTs which are _interpreted_ in the context of an _Effect stack_
+  * [Handle](https://jaspervdj.be/posts/2018-03-08-handle-pattern.html) pattern also known as _record-of-functions_ whereby effects are grouped together in a datatype with a single record constructor
+
+(These tradeoffs also appear in other functional languages like
+[F#](https://medium.com/@dogwith1eye/prefer-records-of-functions-to-interfaces-d6413af4d2c3))
+
+There is not one most favored solution though and we all have various
+experiences with these techniques.
+
+## Decision
+
+Effectful components of the Hydra node (our code) will be defined using the _Handle pattern_.
+
+There might be other techniques in use because of libraries used etc.
+
+## Consequences
+
+For example, the network component is defined as:
+  ```hs
+  newtype Network m = Network
+    { broadcast :: MonadThrow m => HydraMessage -> m ()
+    }
+  ```
+There might be multiple `createNetwork :: m (Network m)` functions

adr/0005-use-io-sim-classes.md

@@ -0,0 +1,35 @@
+# 5. Use io-classes
+
+Date: 2021-06-08
+
+## Status
+
+Accepted
+
+## Context
+
+Although we try to contain the use of IO at the outskirt of the Hydra node using [Handle pattern](0004-use-handle-to-model-effects.md) and [Reactive core](0002-reactive-core.md), low-level effects are still needed in various places, notably to define concurrently executing actions, and thus need to be tested
+
+Testing asynchronous and concurrent code is notoriously painful
+
+The ouroboros consensus test suite and [hydra-sim](https://github.com/input-output-hk/hydra-sim) simulation have demonstrated the effectiveness of abstracting concurrent primitives through the use of typeclasses (MTL-style pattern) and being able to run these as pure code, harvesting and analysing produced execution traces.
+
+There are other such libraries, e.g. [concurrency](https://hackage.haskell.org/package/concurrency) and [dejafu](https://hackage.haskell.org/package/dejafu), as well as the venerable [exceptions](https://hackage.haskell.org/package/exceptions) (for abstracting exception throwing).
+
+## Decision
+
+For all IO effects covered by the library, use functions from typeclasses exposed by [io-classes](https://github.com/input-output-hk/ouroboros-network/tree/e338f2cf8e1078fbda9555dd2b169c6737ef6774/io-classes). As of this writing, this covers:
+  * All STM operations through `MonadSTM`
+  * Time and timers through `MonadTime` and `MonadTimer`
+  * Concurrency through `MonadAsync`, `MonadFork`
+  * Exceptions through `MonadThrow`, `MonadCatch` and `MonadMask`
+
+## Consequences
+
+We can use `io-sim` to evaluate IO-ish functions easily
+
+Instantiation to concrete IO is pushed at the outermost layer, eg. in the `Main` or tests.
+
+As some of these functions and typeclasses clash with the
+[cardano-prelude](https://github.com/input-output-hk/cardano-prelude) we might
+want to define a custom prelude (candidate for another ADR)

adr/0006-network-broadcasts-all-messages.md

@@ -0,0 +1,37 @@
+# 6. Network Broadcasts all messages
+
+Date: 2021-06-08
+
+## Status
+
+Accepted
+
+## Context
+
+The simplified Head protocol in the [Hydra
+paper](https://iohk.io/en/research/library/papers/hydrafast-isomorphic-state-channels/)
+requires _unicast_ and _multicast_ messaging between participants. However, this
+can be simplified to only _multicast_ by also sending `AckTx` messages to all
+participants and removing the necessity for `ConfTx`.
+
+There is already a battle-tested implementation for _broadcasting_ messages over
+networks with any kind of topology (mesh), namely the
+[TxSubmission](https://github.com/input-output-hk/ouroboros-network/tree/master/ouroboros-network/src/Ouroboros/Network/TxSubmission)
+protocol of `ouroroboros-network`.
+
+If the network connects only to interested peers, _broadcast_ is essentially the
+_multicast_ required by the protocol. If this is not the case, some addressing
+scheme is required and _broadcast_ would be a waste of resources.
+
+## Decision
+
+* All messages emitted by a Hydra node through the Network component are _broadcasted_ to _all_ nodes in the network
+* This implies the emitter shall itself receive the message
+
+## Consequences
+
+* The network layer is responsible for ensuring sent messages effectively
+  reaches all nodes in the network. How this is achieved is left as an
+  implementation detail, i.e. whether it uses relaying or not.
+* We need to make sure all Head participants are connected to the same network.
+

adr/0007-with-pattern-component-interfaces.md

@@ -0,0 +1,46 @@
+# 7. Use with-pattern based component interfaces
+
+Date: 2021-06-08
+
+## Status
+
+:hammer_and_wrench:
+
+TBD:
+* Naming of `Callback` and `Component`
+* Demonstrate the `Covariant` consequence
+* Provide context about tying the knot
+
+## Context
+
+The _with pattern_ or _bracket pattern_ is a functional programming idiom, a
+particular instance of _Continuation-Passing Style_, whereby one component that
+controls some resource that is consumed by another component of the system, is
+created via a function that takes as argument a function consuming the resource,
+instead of returning it. This pattern allows safe reclaiming of resources when
+the "wrapped" action terminates, whether normally or unexpectedly.
+
+TODO "Tying the knot"
+
+## Decision
+
+We use this pattern to provide interfaces to all _active components_, which
+exchange messages with other components of the system. A prototypical signature
+of such a component could be:
+
+  ```hs
+  type Component m = outmsg -> m ()
+  type Callback m = inmsg -> m ()
+
+  withXXX :: Callback m -> (Component m -> m a) -> m a
+  ```
+
+Note that `withXXX` can also allocate resources in order to provide `Component`
+or use the `Callback`, e.g. fork threads which invoke `Callback`, but also make
+sure they are cleaned up.
+
+## Consequences
+
+Components can be layered on top of another to provide additional behavior given the same interface. This also similar to "decorating" in the object-orientation world.
+
+If the `Component` is agnostic about the messages it consumes/produces, it can be defined as a [`Contravariant` functor](https://hackage.haskell.org/package/base-4.15.0.0/docs/Data-Functor-Contravariant.html) and the `Callback` part as a (covariant) `Functor`. This makes it possible to use `map` and `contramap` operations to transform messages.