Wip progress

book/src/SUMMARY.md

@@ -6,12 +6,14 @@
     - [From Source](./installation/from_source.md)
     - [Docker](./installation/docker.md)
 - [Usage](./usage/README.md)
-- [Sources](./sources/README.md)
-    - [Node-to-Node](./sources/n2n.md)
-    - [Node-to-Client](./sources/n2c.md)
-- [Reducers](./reducers/README.md)
-- [Storage](./storage/README.md)
-    - [Redis](./storage/redis.md)
+- [Configuration](./configuration/README.md)
+    - [Sources](./configuration/sources.md)
+    - [Reducers](./configuration/reducers.md)
+    - [Storage](./configuration/storage.md)
+    - [Enrich](./configuration/enrich.md)
+    - [Intersect](./configuration/intersect.md)
+    - [Chain](./configuration/chain.md)
+    - [Policy](./configuration/policy.md)
 - [Advanced Features](./advanced/README.md)
     - [Swarm Mode](./advanced/swarm_mode.md)
 - [Guides](./guides/README.md)

book/src/configuration/README.md

@@ -0,0 +1,43 @@
+# Configuration
+
+Scrolls daemon must be configured using a single `.toml` file. For the purpose of testing out Scrolls you can use the provided configuration located in `testdrive/simple/daemon.toml`. See below for an example config with explantions.
+
+```toml
+# get data from a relay node
+[source]
+type = "N2N"
+address = "relays-new.cardano-mainnet.iohk.io:3001"
+
+# You can optionally enable enrichment (local db with transactions), this is needed for some reducers
+[enrich]
+type = "Sled"
+db_path = "/opt/scrolls/sled_db"
+
+# enable the "UTXO by Address" collection
+[[reducers]]
+type = "UtxoByAddress"
+# you can optionally prefix the keys in the collection
+key_prefix = "c1"
+# you can optionally only process UTXO from a set of predetermined addresses
+filter = ["addr1qy8jecz3nal788f8t2zy6vj2l9ply3trpnkn2xuvv5rgu4m7y853av2nt8wc33agu3kuakvg0kaee0tfqhgelh2eeyyqgxmxw3"]
+
+# enable the "Point by Tx" collection
+[[reducers]]
+type = "PointByTx"
+key_prefix = "c2"
+
+# store the collections in a local Redis
+[storage]
+type = "Redis"
+connection_params = "redis://127.0.0.1:6379"
+
+# start reading from an arbitrary point in the chain
+[intersect]
+type = "Point"
+value = [57867490, "c491c5006192de2c55a95fb3544f60b96bd1665accaf2dfa2ab12fc7191f016b"]
+
+# let Scrolls know that we're working with mainnet
+[chain]
+type = "Mainnet"
+```
+

book/src/configuration/chain.md

@@ -0,0 +1 @@
+# Chain

book/src/configuration/enrich.md

@@ -0,0 +1,7 @@
+# Enrich
+
+## Fields
+- type: `"Sled" | "Skip"`
+- db_path **(*)**: `"<dirpath>"`
+
+(*) Available only with `type = "Sled"`

book/src/configuration/intersect.md

@@ -0,0 +1,15 @@
+# Intersect
+
+Scrolls provides 4 different strategies for finding the intersection point within the chain sync process.
+
+- `Origin`: Scrolls will start reading from the beginning of the chain.
+- `Tip`: Scrolls will start reading from the current tip of the chain.
+- `Point`: Scrolls will start reading from a particular point (slot, hash) in the chain. If the point is not found, the process will be terminated with a non-zero exit code.
+- `Fallbacks`: Scrolls will start reading the first valid point within a set of alternative positions. If point is not valid, the process will fallback into the next available point in the list of options. If none of the points are valid, the process will be terminated with a non-zero exit code.
+
+
+## Fields
+- type: `"Tip" | "Origin" | "Point" | "Fallbacks"`
+- value **(*)**: `(u64, String) | Vec<(u64, String)>`
+
+**(*)** Use `(u64, String)` with `type = "Point"` and `Vec<(u64, String)>` with `type = "Fallbacks"`

book/src/configuration/policy.md

@@ -0,0 +1 @@
+# Policy

book/src/configuration/redis.md

@@ -0,0 +1 @@
+# Redis

book/src/configuration/reducers.md

@@ -0,0 +1,146 @@
+# Reducers
+
+## address_by_asset
+
+### Description
+### Output Format
+
+<br />
+<br />
+<hr />
+
+## address_by_txo
+
+### Description
+### Output Format
+
+<br />
+<br />
+<hr />
+
+## addresses_by_stake
+
+### Description
+### Output Format
+
+<br />
+<br />
+<hr />
+
+## asset_holders_by_asset_id
+
+### Description
+### Output Format
+
+<br />
+<br />
+<hr />
+
+## balance_by_address
+
+### Description
+### Output Format
+
+<br />
+<br />
+<hr />
+
+## block_header_by_hash
+
+### Description
+### Output Format
+
+<br />
+<br />
+<hr />
+
+## last_block_parameters
+
+### Description
+### Output Format
+
+<br />
+<br />
+<hr />
+
+## point_by_tx
+
+### Description
+### Output Format
+
+<br />
+<br />
+<hr />
+
+## pool_by_stake
+
+### Description
+### Output Format
+
+<br />
+<br />
+<hr />
+
+## supply_by_asset
+
+### Description
+### Output Format
+
+<br />
+<br />
+<hr />
+
+## tx_by_hash
+
+### Description
+### Output Format
+
+<br />
+<br />
+<hr />
+
+## tx_count_by_address
+
+### Description
+### Output Format
+
+<br />
+<br />
+<hr />
+
+## tx_count_by_native_token_policy_id
+
+### Description
+### Output Format
+
+<br />
+<br />
+<hr />
+
+## utxo_by_address
+
+### Description
+### Output Format
+
+<br />
+<br />
+<hr />
+
+## utxo_by_stake
+
+### Description
+### Output Format
+
+<br />
+<br />
+<hr />
+
+## utxos_by_asset
+
+### Description
+### Output Format
+
+<br />
+<br />
+<hr />
+

book/src/configuration/sources.md

@@ -0,0 +1,12 @@
+# Sources
+
+Sources represent the "origin" of the events processed by Scrolls. Any compatible source is responsible for feeding blockchain data into crolls's pipeline for further processing. This section describes the currently available sources included as part the main Scrolls codebase.
+
+### Node-to-Node
+The Node-to-Node (N2N) source uses Ouroboros mini-protocols to connect to a local or remote Cardano node through a tcp socket bearer and fetches block data using the ChainSync mini-protocol instantiated to "headers only" and the BlockFetch mini-protocol for retrieval of the actual block payload.
+
+
+### Node-to-Client 
+The Node-to-Client (N2C) source uses Ouroboros mini-protocols to connect to a local Cardano node through a unix socket bearer and fetches block data using the ChainSync mini-protocol instantiated to "full blocks".
+
+

book/src/configuration/storage.md

@@ -0,0 +1,12 @@
+# Storage
+
+Storage backends are "pluggable", any key-value storage mechanism is a potential candidate. Our backend of preference is Redis (and TBH, the only one implemented so far). It provides a very high "read" throughput, it can be shared across the network by multiple clients and can be used in cluster-mode for horizontal scaling.
+
+
+## Available backends
+
+### Redis
+
+In case you don't have a redis instance running in your system, you might want to look at our [Testdrive](../guides/testdrive.md) guide for a minimal docker example providing a local redis instance. You can also look at the official Redis [Installation guide](https://redis.io/docs/getting-started/installation/).
+
+See our [Redis-cli basics](../guides/redis.md) guide to get started playing with Redis from the command line.

book/src/installation/docker.md

@@ -1,9 +1,11 @@
 # Docker
 
-Installation using the Docker image:
+Installation using the Docker image. Use the following command:
 
 ```
 docker pull ghcr.io/txpipe/scrolls:v0.5.0
 ```
 
 Check [Github Packages](https://github.com/txpipe/scrolls/pkgs/container/scrolls) for newer versions.
+
+Check the [Testdrive](../guides/testdrive.md) guide for a minimal example that uses docker-compose to spin up a local Redis instance and a Scrolls daemon.

book/src/reducers/README.md

@@ -134,6 +134,7 @@
 <br />
 <br />
 <hr />
+
 ## utxos_by_asset
 
 ### Description

book/src/usage/README.md

@@ -6,55 +6,5 @@ Once you have Scrolls and Redis installed in your system (see [Installation](../
 /path/to/scrolls daemon --config /path/to/config.toml
 ```
 
-If you have no experience using Redis, you can follow the [Redis-cli guide](../guides/redis.md) to learn the very basics needed to fetch data from redis using the command line application provided by Redis. For any real application you would need to use a client library in your language of preference. See our [Python](../guides/python.md) and [NodeJS](../guides/nodejs.md) guides.
+Check the [Configuration](../configuration/index.md) section of the manual for further information. If you have no experience using Redis, you can follow the [Redis-cli guide](../guides/redis.md) to learn the very basics needed to fetch data using the command line application provided by Redis. For any real application you would need to use a client library in your language of preference. See our [Python](../guides/python.md) and [NodeJS](../guides/nodejs.md) guides.
 
-## Configuration
-Scrolls daemon must be configured using a single `.toml` file. For the purpose of testing out Scrolls you can use the provided configuration located in `testdrive/simple/daemon.toml`. See below for an example config with examplantions.
-
-```toml
-# get data from a relay node
-[source]
-type = "N2N"
-address = "relays-new.cardano-mainnet.iohk.io:3001"
-
-# You can optionally enable enrichment (local db with transactions), this is needed for some reducers
-[enrich]
-type = "Sled"
-db_path = "/opt/scrolls/sled_db"
-
-# enable the "UTXO by Address" collection
-[[reducers]]
-type = "UtxoByAddress"
-# you can optionally prefix the keys in the collection
-key_prefix = "c1"
-# you can optionally only process UTXO from a set of predetermined addresses
-filter = ["addr1qy8jecz3nal788f8t2zy6vj2l9ply3trpnkn2xuvv5rgu4m7y853av2nt8wc33agu3kuakvg0kaee0tfqhgelh2eeyyqgxmxw3"]
-
-# enable the "Point by Tx" collection
-[[reducers]]
-type = "PointByTx"
-key_prefix = "c2"
-
-# store the collections in a local Redis
-[storage]
-type = "Redis"
-connection_params = "redis://127.0.0.1:6379"
-
-# start reading from an arbitrary point in the chain
-[intersect]
-type = "Point"
-value = [57867490, "c491c5006192de2c55a95fb3544f60b96bd1665accaf2dfa2ab12fc7191f016b"]
-
-# let Scrolls know that we're working with mainnet
-[chain]
-type = "Mainnet"
-```
-### The source section
-This section specifies the origin of the data. The special type field must always be present and containing a value matching any of the available built-in sources. The rest of the fields in the section will depend on the selected type. See the [Sources](../sources/index.md) section for a list of available options and their corresponding config values.
-### The enrich section
-### The reducers section
-See the [Reducers](../reducers/index.md) section of the manual.
-### The storage section
-See the [Storage](../storage/index.md) section of the manual.
-### The intersect section
-### The chain section