Merge pull request #288 from cerc-io/cleanup-readmes

Watcher docs updates

.github/CONTRIBUTING.md

@@ -0,0 +1,15 @@
+# Contributing
+
+To update versions in all packages run the following:
+
+```bash
+yarn version:set <VERSION>
+yarn version:set-codegen <VERSION>
+```
+
+Example:
+
+```bash
+yarn version:set 0.2.20
+yarn version:set-codegen 0.2.20
+```

README.md

@@ -1,58 +1,56 @@
 # watcher-ts
 
-## Setup
+![Cute Panopticon](./docs/watchers-graphic.png)
 
-This project uses [yarn workspaces](https://classic.yarnpkg.com/en/docs/workspaces/).
+Watchers make managing data in Dapp development as frictionless as possible. They do this by querying, transforming, and caching Ethereum state data cheaper and faster compared to existing solutions. This data also comes with evidence for generating cryptographic proofs to provide verification that the data is authentic. Public watchers are found in the [packages](/packages) directory.
 
-Install packages (Node.JS v16.13.1):
+Go [here](https://github.com/cerc-io/stack-orchestrator/tree/main/app/data/stacks/erc20) for a quick start demo of setting up the stack and deploying/using the erc20 watcher via the Laconic Stack Orchestrator.
 
-```bash
-yarn
-```
 
-Build packages:
+## Prerequisites
 
-```bash
-yarn build
-```
+### User Mode
 
-## Tests
+- `laconic-so` [Install](https://github.com/cerc-io/stack-orchestrator#setup)
 
-* [graph-node](./packages/graph-node/README.md)
+The Laconic Stack Orchestrator provides all the tools to quickly get started with watchers.
+
+### Developer Mode
+
+- `yarn` [Install](https://yarnpkg.com/getting-started/install)
+- `postgres` [Install](https://www.postgresql.org/download/)
+
+You'll need the above if you plan on digging into this repo, writing your own watchers, or experimenting with watchers not currently supported by Stack Orchestrator. See [CONTRIBUTING.md](.github/CONTRIBUTING.md) for more information.
 
 ## Services
 
-The default config files used by the watchers assume the following services are setup and running on localhost:
+The default configuration files used by watchers assume the following services are setup and running on localhost:
 
-* `vulcanize/go-ethereum` on port 8545
-* `vulcanize/ipld-eth-server` with native GQL API enabled, on port 8082
+* [cerc-io/go-ethereum](https://github.com/cerc-io/go-ethereum/tree/v1.10.25-statediff-v4) on port 8545
+* [cerc-io/ipld-eth-server](https://github.com/cerc-io/ipld-eth-server) with native GQL API enabled, on port 8082
+* [cerc-io/ipld-eth-db](https://github.com/cerc-io/ipld-eth-db) to populate the postgres database for `ipld-eth-server`
 
-### Note
+These services are dockerized by the Laconic Stack Orchestrator (`laconic-so`). [Use it](https://github.com/cerc-io/stack-orchestrator) unless you plan on digging into those codebases. For more information on setting up these services up by hand, see [here](/docs/README.md)
 
-* In `vulcanize/ipld-eth-server`, add the following statement to `[ethereum]` section in `environments/config.toml`:
+## Setup
+
+From the root of this repository, run:
+
+`yarn && yarn build`
 
-  `chainConfig = "./chain.json" # ETH_CHAIN_CONFIG`
+to download dependencies and build all the watchers.
 
-## Watchers
+Orient yourself with the available CLI commands [here](docs/cli.md) and in some cases, watchers have in-depth demos (e.g. [mobymask](/packages/mobymask-watcher))
 
-* [eden-watcher](./packages/eden-watcher/README.md)
-* [erc20-watcher](./packages/erc20-watcher/README.md)
-* [erc721-watcher](./packages/erc721-watcher/README.md)
-* [graph-test-watcher](./packages/graph-test-watcher/README.md)
-* [mobymask-watcher](./packages/mobymask-watcher/README.md)
 
-## Development
+[//]: # (TODO: ## Generating Watchers)
 
-* To update versions in all packages run the following:
+## Tests
+
+* [graph-node](./packages/graph-node/README.md)
 
-  ```bash
-  yarn version:set <VERSION>
-  yarn version:set-codegen <VERSION>
-  ```
+## Further Reading
 
-  Example
+[//]: # (TODO: link to docs.laconic.com when ready)
 
-  ```bash
-  yarn version:set 0.2.20
-  yarn version:set-codegen 0.2.20
-  ```
+- [Watchers Blog Post](https://www.laconic.com/blog/laconic-watchers)

docs/README.md

@@ -0,0 +1,85 @@
+# Watcher Documentation
+
+This document is specifically focused on standing up a minimal core version of the Laconic Stack *without* using [Stack Orchestrator]( https://github.com/cerc-io/stack-orchestrator). If this is your first foray into the stack, start with Stack Orchestrator. To understand what is going on under the hood or to make contributions to this repo, this is a good place to start.
+
+There are 3 main components to setting up an environment for running watchers:
+- core services
+- configure postgres
+- edit config file
+
+After which you should be able to navigate to the `README.md` of any watcher and run through its demo using `yarn`. The common `yarn` CLI commands for watchers are documented [here](../cli.md).
+
+## Core services
+
+The following core services should be setup and running on localhost:
+
+* `cerc-io/go-ethereum` [v1.10.26-statediff-4.2.2-alpha](https://github.com/cerc-io/go-ethereum/releases/tag/v1.10.26-statediff-4.2.2-alpha)
+ on port 8545
+
+* `cerc-io/ipld-eth-server` [v4.2.2-alpha](https://github.com/cerc-io/ipld-eth-server/releases/tag/v4.2.2-alpha) with native GQL API enabled, on port 8082
+
+* `cerc-io/ipld-eth-db` [v4.2.2-alpha](https://github.com/cerc-io/ipld-eth-db/releases/tag/v4.2.2-alpha) is the postgres schema required for `ipld-eth-server`
+
+## Setup Postgres
+
+In this example, we use the `erc20-watcher`; for another watcher substitute with its name.
+
+Create a postgres database for the watcher:
+
+```bash
+sudo su - postgres
+createdb erc20-watcher
+```
+
+Create a postgres database for the job queue:
+
+```
+sudo su - postgres
+createdb erc20-watcher-job-queue
+```
+
+Enable the `pgcrypto` [extension](https://github.com/timgit/pg-boss/tree/master/docs#database-install) on the job queue database.
+
+```
+postgres@tesla:~$ psql -U postgres -h localhost erc20-watcher-job-queue
+Password for user postgres:
+psql (12.7 (Ubuntu 12.7-1.pgdg18.04+1))
+SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
+Type "help" for help.
+
+erc20-watcher-job-queue=# CREATE EXTENSION pgcrypto;
+CREATE EXTENSION
+erc20-watcher-job-queue=# exit
+```
+
+## Config File
+
+In each watchers' directory is a config file: `packages/<watcher>/environments/local.toml`:
+
+* Update the database connection settings.
+* Update the `upstream` config and provide the `ipld-eth-server` GraphQL API endpoint.
+* Select "active" vs. "lazy" watcher depending on its kind.
+
+For example:
+```toml
+[server]
+  kind = "active"
+
+[database]
+  type = "postgres"
+  host = "localhost"
+  port = 5432
+  database = "erc20-watcher"
+  username = "postgres"
+  password = "postgres"
+
+[jobQueue]
+  dbConnectionString = "postgres://postgres:postgres@localhost/erc20-watcher-job-queue"
+
+[upstream]
+  [upstream.ethServer]
+    gqlApiEndpoint = "http://127.0.0.1:8082/graphql"
+    rpcProviderEndpoint = "http://127.0.0.1:8081"
+```
+
+Now that your environment is setup, you can test run any watcher!

docs/cli.md

@@ -0,0 +1,127 @@
+## Watcher CLI commands
+
+Non-exhaustive list of (yarn) CLI commands available for watchers. Assumes you have an environment setup either [by hand](./README.md) or using [Stack Orchestrator](https://github.com/cerc-io/stack-orchestrator), and have built `yarn && yarn build` a specific watcher.
+
+If the watcher is an `active` watcher, run the job-runner:
+
+```bash
+yarn job-runner
+```
+
+```
+# For development.
+yarn server:dev
+
+# For specifying config file.
+yarn server -f environments/local.toml
+```
+
+If the watcher is `lazy` omit the above step, then run the server:
+
+```bash
+yarn server
+```
+
+This will enable the GraphQL playground at: `http://localhost:<port>/graphql`
+
+where `<port>` is set in the `environments/local.toml` of each watcher.
+
+To watch a contract:
+
+```bash
+yarn watch:contract --address <contract-address> --kind <contract-kind> --checkpoint <true | false> --starting-block <block-number>
+```
+
+* `address`: Address or identifier of the contract to be watched.
+* `kind`: Kind of the contract.
+* `checkpoint`: Turn checkpointing on (`true` | `false`).
+* `starting-block`: Starting block for the contract (default: `1`).
+
+Example:
+
+Watch a contract with its address and checkpointing on:
+
+```bash
+yarn watch:contract --address 0x1F78641644feB8b64642e833cE4AFE93DD6e7833 --kind ERC20 --checkpoint true
+```
+
+To fill a block range:
+
+```bash
+yarn fill --start-block <from-block> --end-block <to-block>
+```
+
+* `start-block`: Block number to start filling from.
+* `end-block`: Block number till which to fill.
+
+To create a checkpoint for a contract:
+
+```bash
+yarn checkpoint create --address <contract-address> --block-hash <block-hash>
+```
+
+* `address`: Address or identifier of the contract for which to create a checkpoint.
+* `block-hash`: Hash of a block (in the pruned region) at which to create the checkpoint (default: latest canonical block hash).
+
+To verify a checkpoint:
+
+```bash
+yarn checkpoint verify --cid <checkpoint-cid>
+```
+
+* `cid`: CID of the checkpoint for which to verify.
+
+To reset the watcher to a previous block number:
+
+```bash
+yarn reset watcher --block-number <previous-block-number>
+```
+
+* `block-number`: Block number to which to reset the watcher.
+
+To reset the job-queue:
+
+```bash
+yarn reset job-queue
+```
+
+To reset the state:
+
+```bash
+yarn reset state --block-number <previous-block-number>
+```
+
+## Import/Export State
+
+To export and import the watcher state:
+
+In the source watcher, export watcher state:
+
+```bash
+yarn export-state --export-file [export-file-path] --block-number [snapshot-block-height]
+```
+
+* `export-file`: Path of file to which to export the watcher data.
+* `block-number`: Block height at which to take snapshot for export.
+
+In the target watcher, run the job-runner:
+
+```bash
+yarn job-runner
+```
+
+Import watcher state:
+
+```bash
+yarn import-state --import-file <import-file-path>
+```
+
+* `import-file`: Path of file from which to import the watcher data.
+
+To inspect a CID:
+
+```bash
+yarn inspect-cid --cid <cid>
+```
+
+* `cid`: CID to be inspected.

packages/README.md

@@ -0,0 +1,20 @@
+# Watchers Directory
+
+Here contains all the publicly available watchers for your experimenting purposes. For general information on setting up the stack and the CLI commands used to interact with watchers, see the [docs](/docs) directory.
+
+## Customizing Watchers
+
+* Indexing on an event:
+
+  * Edit the custom hook function `handleEvent` (triggered on an event) in [hooks.ts](./src/hooks.ts) to perform corresponding indexing using the `Indexer` object.
+
+  * While using the indexer storage methods for indexing, pass `diff` as true if default state is desired to be generated using the state variables being indexed.
+
+* Generating state:
+
+  * Edit the custom hook function `createInitialState` (triggered if the watcher passes the start block, checkpoint: `true`) in [hooks.ts](./src/hooks.ts) to save an initial `State` using the `Indexer` object.
+
+  * Edit the custom hook function `createStateDiff` (triggered on a block) in [hooks.ts](./src/hooks.ts) to save the state in a `diff` `State` using the `Indexer` object. The default state (if exists) is updated.
+
+  * Edit the custom hook function `createStateCheckpoint` (triggered just before default and CLI checkpoint) in [hooks.ts](./src/hooks.ts) to save the state in a `checkpoint` `State` using the `Indexer` object.
+