From a388c676ee4c37c648b7ad66c348d0526f6bd181 Mon Sep 17 00:00:00 2001 From: Ori Pomerantz Date: Thu, 14 Sep 2023 16:08:46 -0500 Subject: [PATCH 1/5] WIP --- docs/pages/indexer.mdx | 79 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 79 insertions(+) diff --git a/docs/pages/indexer.mdx b/docs/pages/indexer.mdx index cdf0c486e9..d8996cf4d2 100644 --- a/docs/pages/indexer.mdx +++ b/docs/pages/indexer.mdx @@ -290,3 +290,82 @@ pnpm start:sqlite ``` Note that in your queries you need to specify the correct chainId for the chain you are indexing. + +## Using the indexer with PostgreSQL + +To run the indexer with [PostgreSQL](https://www.postgresql.org/), follow these steps: + +1. Download and install [PostgreSQL](https://www.postgresql.org/download/). + +1. To use the default setting (local PostgreSQL, using the `postgres` database, indexing the local `anvil` chain), run: + + ```sh copy + pnpm start:postgres:local + ``` + + Otherwise, set these environment variables: + + | Variable | Value | + | ------------ | ---------------------------------------------------------------- | + | DATABASE_URL | URL for the database, of the form `postgres:///` | + | CHAIN_ID | ChainID for the chain you index | + + And then run: + + ```sh copy + pnpm start:postgres + ``` + +### Reading information from PostgreSQL + +You can now use [`psql`](https://www.postgresql.org/docs/current/app-psql.html) to view the indexer information. + +1. Get the list of schemas to verify `__mud_internal` is there. + + ```sql copy + \dn + ``` + +1. Get the list of relations in the schema. + + ```sql copy + \dt __mul_internal.* + ``` + + You should get two tables, `chain` and `tables`. + +1. Get the list of chains (there should be just one). + + ```sql copy + select * from __mud_internal.chain; + ``` + +1. The `tables` table is a lot bigger, so first get the list of columns within the table. + + ```sql copy + \d+ __mud_internal.tables + ``` + +1. Get the list of MUD table names. + + ```sql copy + select name from __mud_internal.tables; + ``` + +1. Get the key and value schemas for `Counter`. + + ```sql copy + select key_schema, value_schema from __mud_internal.tables where name='Counter'; + ``` + +1. Get the list of tables for a specific world (if your world address is different, use that instead of `0x5FbDB2315678afecb367f032d93F642f64180aa3`). + + ```sql copy + \dt "0x5FbDB2315678afecb367f032d93F642f64180aa3__".* + ``` + +1. Get the value of the counter. + + ```sql copy + select * from "0x5FbDB2315678afecb367f032d93F642f64180aa3__"."Counter"; + ``` From b934d278936c80f207715ac8765e1c12f8b6aebb Mon Sep 17 00:00:00 2001 From: Ori Pomerantz Date: Thu, 14 Sep 2023 20:46:33 -0500 Subject: [PATCH 2/5] Add @ludns feedback --- docs/pages/indexer.mdx | 38 ++++++++++++++++++++++++++------------ 1 file changed, 26 insertions(+), 12 deletions(-) diff --git a/docs/pages/indexer.mdx b/docs/pages/indexer.mdx index d8996cf4d2..211e73635f 100644 --- a/docs/pages/indexer.mdx +++ b/docs/pages/indexer.mdx @@ -276,12 +276,14 @@ To understand the result, it is easiest to look at it inside Node. To index a different blockchain, you specify these environment variables: -| Variable | Value | -| --------------- | -------------------------------------------------------------------------------------------------------- | -| RPC_HTTP_URL | The URL to access the blockchain using HTTP | -| RPC_WS_URL | The URL to access the blockchain using WebSocket | -| START_BLOCK | The block to start indexing from. The block in which the `World` contract was deployed is a good choice. | -| SQLITE_FILENAME | `.db` | +| Variable | Value | +| ----------------------- | -------------------------------------------------------------------------------------------------------- | +| RPC_HTTP_URL | The URL to access the blockchain using HTTP | +| RPC_WS_URL | The URL to access the blockchain using WebSocket | +| START_BLOCK1 | The block to start indexing from. The block in which the `World` contract was deployed is a good choice. | +| SQLITE_FILENAME | `.db` | + +(1) This value is optional, but highly recommended in chains where the world was deployed after a large number of blocks had already passed. After you do that, start the indexer with this command: @@ -305,10 +307,16 @@ To run the indexer with [PostgreSQL](https://www.postgresql.org/), follow these Otherwise, set these environment variables: - | Variable | Value | - | ------------ | ---------------------------------------------------------------- | - | DATABASE_URL | URL for the database, of the form `postgres:///` | - | CHAIN_ID | ChainID for the chain you index | + | Variable | Value | + | ------------------------ | -------------------------------------------------------------------------------------------------------- | + | CHAIN_ID | ChainID for the chain you index | + | RPC_HTTP_URL1 | The URL to access the blockchain using HTTP | + | RPC_WS_URL1 | The URL to access the blockchain using WebSocket | + | START_BLOCK2 | The block to start indexing from. The block in which the `World` contract was deployed is a good choice. | + | DATABASE_URL | URL for the database, of the form `postgres:///` | + + (1) Not necessary in the case of a Lattice testnet. + (2) This value is optional, but highly recommended in chains where the world was deployed after a large number of blocks had already passed for performance reasons. And then run: @@ -329,7 +337,7 @@ You can now use [`psql`](https://www.postgresql.org/docs/current/app-psql.html) 1. Get the list of relations in the schema. ```sql copy - \dt __mul_internal.* + \dt __mud_internal.* ``` You should get two tables, `chain` and `tables`. @@ -364,8 +372,14 @@ You can now use [`psql`](https://www.postgresql.org/docs/current/app-psql.html) \dt "0x5FbDB2315678afecb367f032d93F642f64180aa3__".* ``` +1. Specify that the default schema is the world being indexed. + + ```sql copy + set search_path to "0x5FbDB2315678afecb367f032d93F642f64180aa3__"; + ``` + 1. Get the value of the counter. ```sql copy - select * from "0x5FbDB2315678afecb367f032d93F642f64180aa3__"."Counter"; + select * from "Counter"; ``` From c137a078fbcd522a7dfac4aab0e1f6af954bb29c Mon Sep 17 00:00:00 2001 From: Ori Pomerantz Date: Fri, 15 Sep 2023 12:21:46 -0500 Subject: [PATCH 3/5] Explain the world address better. --- docs/pages/indexer.mdx | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/docs/pages/indexer.mdx b/docs/pages/indexer.mdx index 211e73635f..9076576540 100644 --- a/docs/pages/indexer.mdx +++ b/docs/pages/indexer.mdx @@ -316,6 +316,7 @@ To run the indexer with [PostgreSQL](https://www.postgresql.org/), follow these | DATABASE_URL | URL for the database, of the form `postgres:///` | (1) Not necessary in the case of a Lattice testnet. + (2) This value is optional, but highly recommended in chains where the world was deployed after a large number of blocks had already passed for performance reasons. And then run: @@ -328,6 +329,9 @@ To run the indexer with [PostgreSQL](https://www.postgresql.org/), follow these You can now use [`psql`](https://www.postgresql.org/docs/current/app-psql.html) to view the indexer information. +Note: This example uses [the template](http://localhost:3000/tutorials/minimal). +If you are indexing a different `World`, change the world address and use the name of a table you have rather than `Counter`. + 1. Get the list of schemas to verify `__mud_internal` is there. ```sql copy @@ -342,7 +346,7 @@ You can now use [`psql`](https://www.postgresql.org/docs/current/app-psql.html) You should get two tables, `chain` and `tables`. -1. Get the list of chains (there should be just one). +1. Get the list of chains being indexed. ```sql copy select * from __mud_internal.chain; @@ -366,13 +370,14 @@ You can now use [`psql`](https://www.postgresql.org/docs/current/app-psql.html) select key_schema, value_schema from __mud_internal.tables where name='Counter'; ``` -1. Get the list of tables for a specific world (if your world address is different, use that instead of `0x5FbDB2315678afecb367f032d93F642f64180aa3`). +1. The schema for a `World` is `
__`. + Get the list of tables for a specific world. ```sql copy \dt "0x5FbDB2315678afecb367f032d93F642f64180aa3__".* ``` -1. Specify that the default schema is the world being indexed. +1. Specify the `World` schema is the default. ```sql copy set search_path to "0x5FbDB2315678afecb367f032d93F642f64180aa3__"; From 04092790cc245258759b9f83d701d66645cf6d72 Mon Sep 17 00:00:00 2001 From: Ori Pomerantz Date: Fri, 15 Sep 2023 12:22:31 -0500 Subject: [PATCH 4/5] Minor edits --- docs/pages/indexer.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/pages/indexer.mdx b/docs/pages/indexer.mdx index 9076576540..d454ae0f98 100644 --- a/docs/pages/indexer.mdx +++ b/docs/pages/indexer.mdx @@ -370,14 +370,14 @@ If you are indexing a different `World`, change the world address and use the na select key_schema, value_schema from __mud_internal.tables where name='Counter'; ``` -1. The schema for a `World` is `
__`. +1. The schema for a world is `
__`. Get the list of tables for a specific world. ```sql copy \dt "0x5FbDB2315678afecb367f032d93F642f64180aa3__".* ``` -1. Specify the `World` schema is the default. +1. Specify the world schema is the default. ```sql copy set search_path to "0x5FbDB2315678afecb367f032d93F642f64180aa3__"; From 933d05da28cf538c44c932f33ca80a56006aafcf Mon Sep 17 00:00:00 2001 From: Ori Pomerantz Date: Mon, 18 Sep 2023 19:42:09 -0500 Subject: [PATCH 5/5] Update the PostgreSQL instructions to explain how to delete the data --- docs/pages/indexer.mdx | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/docs/pages/indexer.mdx b/docs/pages/indexer.mdx index 632b02c37c..3fb503dcf5 100644 --- a/docs/pages/indexer.mdx +++ b/docs/pages/indexer.mdx @@ -357,7 +357,7 @@ If you are indexing a different `World`, change the world address and use the na 1. Get the list of chains being indexed. ```sql copy - select * from __mud_internal.chain; + SELECT * FROM __mud_internal.chain; ``` 1. The `tables` table is a lot bigger, so first get the list of columns within the table. @@ -369,13 +369,13 @@ If you are indexing a different `World`, change the world address and use the na 1. Get the list of MUD table names. ```sql copy - select name from __mud_internal.tables; + SELECT name FROM __mud_internal.tables; ``` 1. Get the key and value schemas for `Counter`. ```sql copy - select key_schema, value_schema from __mud_internal.tables where name='Counter'; + SELECT key_schema, value_schema FROM __mud_internal.tables WHERE name='Counter'; ``` 1. The schema for a world is `
__`. @@ -388,11 +388,22 @@ If you are indexing a different `World`, change the world address and use the na 1. Specify the world schema is the default. ```sql copy - set search_path to "0x5FbDB2315678afecb367f032d93F642f64180aa3__"; + SET search_path TO "0x5FbDB2315678afecb367f032d93F642f64180aa3__"; ``` 1. Get the value of the counter. ```sql copy - select * from "Counter"; + SELECT * FROM "Counter"; ``` + +### Deleting the PostgreSQL data + +PostgreSQL is a persistent data storage. +This means that if you restart the blockchain without manually deleting the data, your indexer will use old data and provide incorrect results. + +To avoid this, enter `psql` and run this command: + +```sql copy +DROP OWNED BY ; +```