README cleanup and simplification (#3104)

* Change docker-compose commands to docker compose built into Docker * Change to de-emphasize Docker Compose in favor of Helm/Kubernetes * General grammar and spelling cleanup * Move component documentation to docs/<component>/README.md * Move Docker Compose details from README.md to docs/installation.md * Remove references to "Beta" mirror node * Remove irrelevant TimescaleDB docs * Remove unused REST API performance images * Simplified mirror node overview Signed-off-by: Steven Sheehy <steven.sheehy@hedera.com>
hashgraph · Jan 7, 2022 · fbad635 · fbad635
1 parent 9961230
commit fbad635
Show file tree

Hide file tree

Showing 13 changed files with 284 additions and 434 deletions.
diff --git a/README.md b/README.md
diff --git a/docs/configuration.md b/docs/configuration.md
@@ -224,7 +224,8 @@ to configure the application.
 The following table lists the available properties along with their default values. Unless you need to set a non-default
 value, it is recommended to only populate overridden properties in the custom `application.yml`.
 
-See the monitor [documentation](monitor.md) for more general information about configuring and using the monitor.
+See the monitor [documentation](/docs/monitor/monitor.md) for more general information about configuring and using the
+monitor.
 
 Name                                                            | Default    | Description
 ----------------------------------------------------------------| -----------| ---------------------------------------

diff --git a/docs/grpc/README.md b/docs/grpc/README.md
@@ -0,0 +1,4 @@
+# gRPC API
+
+The gRPC API provides a protobuf defined interface for interacting with the mirror node. Currently only the Hedera
+Consensus Service (HCS) topic subscription is supported.
diff --git a/docs/images/rest_api_perf_test_architecture.png b/docs/images/rest_api_perf_test_architecture.png
diff --git a/docs/images/rest_api_perf_test_migration.png b/docs/images/rest_api_perf_test_migration.png
diff --git a/docs/installation.md b/docs/installation.md
@@ -1,16 +1,11 @@
 # Installation
 
-The Mirror Node can be ran [locally](#running-locally) or via [Docker](#running-via-docker-compose). To run locally,
-it'll first need to be built using Java. To run via Docker, either build locally, or pull latest images from GCR.
-
-```console
-docker-compose pull
-```
+The Mirror Node can be run [locally](#running-locally) or via [Docker](#running-via-docker-compose).
 
 ## Building
 
-Ensure you have AdoptOpenJDK 11 installed, then run the following command from the top level directory. This will
-compile a runnable Mirror Node JAR file in the `target` directory.
+To run locally, first build the project using Java. Ensure you have OpenJDK 11 installed, then run the following command
+from the top level directory. This will compile a runnable mirror node JAR file in the `target` directory.
 
 ```console
 ./mvnw clean package -DskipTests
@@ -20,14 +15,12 @@ compile a runnable Mirror Node JAR file in the `target` directory.
 
 ### Database Setup
 
-In addition to OpenJDK 11, you will need to install a database and initialize it. The Mirror Node
-utilizes [PostgreSQL](https://postgresql.org) v9.6 or [TimescaleDB](https://docs.timescale.com/latest/main) depending on
-the version of its database schema.
+In addition to OpenJDK 11, you will need to install a [PostgreSQL](https://postgresql.org) database and initialize it.
 
-For both databases, since [Flyway](https://flywaydb.org) will manage the database schema, the only required step is to
-run the database initialization script. Locate the SQL script
-at `hedera-mirror-importer/src/main/resources/db/scripts/init.sh` and edit the file to change the name and password
-variables at the top of the file as appropriate. Then make sure the application [configuration](configuration.md)
+Since [Flyway](https://flywaydb.org) will manage the database schema, the only required step is to run the database
+initialization script. Locate the SQL script at `hedera-mirror-importer/src/main/resources/db/scripts/init.sh` and edit
+the file to change the name and password variables at the top of the file to the desired values. Then make sure the
+application [configuration](configuration.md)
 matches the values in the script. Run the SQL script as a super user and check the output carefully to ensure no errors
 occurred.
 
@@ -47,9 +40,9 @@ java -jar hedera-mirror-importer/target/hedera-mirror-importer-*.jar
 Additionally, there is a Systemd unit file located in the `hedera-mirror-importer/scripts/` directory that can be used
 to manage the process. See the [operations](operations.md) documentation for more information.
 
-### GRPC API
+### gRPC API
 
-To run the GRPC API, first populate the configuration at one of the supported
+To run the gRPC API, first populate the configuration at one of the supported
 [configuration](configuration.md#grpc-api) paths, then run:
 
 ```console
@@ -91,7 +84,7 @@ npm test
 #### Prerequisites
 
 ``
-Go 1.16+
+Go 1.17+
 ``
 
 To start the Rosetta API ensure you have the necessary [configuration](configuration.md#rosetta-api) populated and run:
@@ -130,7 +123,7 @@ Currently, Rosetta CLI Validation supports only `DEMO` and `TESTNET`, where
 
 The `All-in-One` configuration aggregates the PostgreSQL, Importer, and Rosetta services into a single Dockerfile
 configuration. Configuration is based on the Rosetta specification,
-found [here](https://www.rosetta-api.org/docs/node_deployment.html). Data Persistence is based on Rosetta specification
+found [here](https://www.rosetta-api.org/docs/node_deployment.html). Data persistence is based on Rosetta specification
 as well, found [here](https://www.rosetta-api.org/docs/standard_storage_location.html). Exposed ports are `5432`
 (PostgreSQL) and `5700` (Rosetta).
 
@@ -201,67 +194,121 @@ docker run -e NETWORK=TESTNET \
 
 ## Running via Docker Compose
 
-Docker Compose scripts are provided and run all the mirror node components:
+Docker Compose scripts are provided and can run all the mirror node components. Containers use the following persistent
+volumes:
 
-- PostgreSQL/TimescaleDB database
-- GRPC API
-- Importer
-- Monitor
-- REST API
-- Rosetta API
-
-Containers use the following persisted volumes:
-
-- `./db` on your local machine maps to `/var/lib/postgresql/data` in the containers. This contains the files for the
-  PostgreSQL/TimescaleDB database. If the database container fails to initialise properly and the database fails to run,
-  you will have to delete this folder prior to attempting a restart otherwise the database initialisation scripts will
-  not be run.
+- `./db` on your local machine maps to `/var/lib/postgresql/data` in the db container. This contains the files for the
+  PostgreSQL database. If the database container fails to initialise properly and the database fails to run, you will
+  have to delete this folder prior to attempting a restart otherwise the database initialisation scripts will not be
+  run.
 
 ### Configuration
 
-#### TimescaleDB vs PostgreSQL
-
-To utilize the TimescaleDB database over the default PostgreSQL database, disable the PostgreSQL container and enable
-the TimescaleDB container.
-
-To achieve this the `docker-compose.yml` can be updated to set the postgres `db` service replicas to 0, while removing
-this same setting from the `timescaledb` service as follows:
-
-```yaml
-    ...
-    services:
-      db:
-        deploy:
-          replicas: 0
-      ...
-      timescaledb:
-      #    deploy:
-      #      replicas: 0
-      ...
-```
+Before starting, [configure](configuration.md) the application by updating the [application.yml](/application.yml)
+file with the desired custom values. This file is passed to Docker Compose and allows customized configuration for each
+of the mirror node components. The `application.yml` file contents represent the minimal set of fields required to
+configure requester pays and must be uncommented and filled in.
 
 ### Starting
 
-Before starting, [configure](configuration.md) the application by updating the [application.yml](../application.yml)
-file with the desired custom values. This file is passed to Docker compose and allows customized configuration for each
-of the sub modules.
-
 Finally, run the commands to build and startup:
 
 ```console
-docker-compose up
+docker compose up
 ```
 
-### Stopping
+### Verifying
+
+When running the mirror node using Docker, activity logs and container status for each module container can be viewed in
+the [Docker Desktop Dashboard](https://docs.docker.com/desktop/dashboard/) or the `docker` CLI to verify expected
+operation. You can also interact with mirror node APIs to verify their operation.
+
+First list running docker container information using and verify an `Up` status is present:
+
+```shell
+$ docker ps
+CONTAINER ID    IMAGE                                      COMMAND                 CREATED         STATUS          PORTS                   NAMES
+21fa2a986d99    gcr.io/mirrornode/hedera-mirror-rest:main  "docker-entrypoint.s…"  7 minutes ago   Up 12 seconds   0.0.0.0:5551->5551/tcp  hedera-mirror-node_rest_1
+56647c384d49    gcr.io/mirrornode/hedera-mirror-grpc:main  "java -cp /app/resou…"  8 minutes ago   Up 16 seconds   0.0.0.0:5600->5600/tcp  hedera-mirror-node_grpc_1
+```
+
+Using the IP address and port, the APIs' endpoints can be called to confirm data is processed and available.
+
+#### Database
 
-Shutting down the database container via `docker-compose down` may result in a corrupted database that may not restart
-or may take longer than usual to restart. In order to avoid this, shell into the container and issue the following
-command:
+The following log can be used to confirm the database is up and running:
 
-Use `docker ps` to get the name of the database container, it should be something like `hedera-mirror-node_db_1`.
+```shell
+database system is ready to accept connections
+```
+
+If you have PostgreSQL installed you can connect directly to your database using the following `psql` command and
+default [configurations](/docs/configuration.md):
 
-Use the command `docker exec -it hedera-mirror-node_db_1 /bin/sh` to get a shell in the container.
+```shell
+psql "dbname=mirror_node host=localhost user=mirror_node password=mirror_node_pass port=5432"
+```
+
+If `psql` is not available locally you can `docker exec -it <CONTAINER ID> bash` into the db container and run the same
+command above.
+
+#### Importer
+
+Logs similar to the following snippets can be used to confirm the Importer is downloading and persisting transactions to
+the database:
+
+```shell
+o.f.c.i.c.DbMigrate Schema "public" is up to date. No migration necessary.
+c.h.m.i.MirrorImporterApplication Started MirrorImporterApplication in 18.77 seconds
+...
+c.h.m.i.p.b.AccountBalanceFileParser Successfully processed 1474 items from 2022-01-05T18_30_00.150597126Z_Balances.pb.gz in 12.78 ms
+c.h.m.i.d.r.RecordFileDownloader Downloaded 1 signatures in 102.8 ms (9/s)
+c.h.m.i.p.r.RecordFileParser Successfully processed 2 items from 2022-01-05T18_16_24.581564299Z.rcd in 3.784 ms
+```
 
-`su - postgres -c "PGDATA=$PGDATA /usr/local/bin/pg_ctl -w stop"`
+#### gRPC API
+
+The gRPC container will display logs similar to the below at start:
+
+```shell
+c.h.m.g.MirrorGrpcApplication Started MirrorGrpcApplication in 15.808 seconds
+```
+
+To manually verify the gRPC streaming endpoint please consult the [operations](/docs/operations.md#verifying) guide.
+
+#### REST API
+
+The REST API container will display logs similar to the below at start:
+
+```shell
+Server running on port: 5551
+```
+
+To manually verify REST API endpoint please consult the [operations](/docs/operations.md#verifying-1) guide.
+
+#### Rosetta API
+
+The Rosetta API container will display logs similar to the below at start:
+
+```shell
+Successfully connected to Database
+Serving Rosetta API in ONLINE mode
+Listening on port 5700
+```
+
+To manually verify the Rosetta API endpoints follow the [operations](/docs/operations.md#verifying-2) details.
+
+#### Web3 API
+
+The Web3 API container will display logs similar to the below at start:
+
+```shell
+c.h.m.web3.Web3Application Started Web3Application in 27.808 seconds
+c.h.m.w.config.LoggingFilter /10.0.0.43 POST http://localhost:8545/web3/v1 in 8 ms: 200 OK
+```
+
+To manually verify the Web3 API endpoints run the [acceptance tests](/docs/web3/README.md#acceptance-tests).
+
+### Stopping
 
-You may now power down the docker image itself.
+Shut down the containers via `docker compose down`.
diff --git a/docs/monitor.md → docs/monitor/README.md b/docs/monitor.md → docs/monitor/README.md