Merge remote-tracking branch 'origin/main' into chap

doc/adr/0014-marconi-query-interface.rst

@@ -0,0 +1,37 @@
+ADR 14: Marconi Query Interface
+======================================================
+
+Date: 2022-10-27
+
+Author(s)
+---------
+
+Kayvan Kazeminejad <kayvan.kazeminejad@iohk.io>
+
+Status
+------
+
+Draft
+
+Context
+-------
+
+`Marconi <https://plutus-apps--763.org.readthedocs.build/en/763/adr/0004-marconi-initiative.html>`_ provides a general solution for indexing the blockchain data.  The query interface adds reporting capabilities on top of Marconi for both Haskell and non-Haskell applications.
+
+Decision
+--------
+
+- We will build the interface on top of the Marconi indexer with minimal impact on the Marconi infrastructure
+
+- The query interface may be used both as an executable or a Haskell library
+
+- The query interface supports `JSON-RPC 2.0 <https://www.jsonrpc.org/>`_ on top of HTTP
+
+- The query interface provides reporting of both memory and disk storage of indexers as described in `Marconi implementation <https://plutus-apps--763.org.readthedocs.build/en/763/adr/0010-marconi-indexer-rollbacks.html#implementation>`_
+
+Implications
+^^^^^^^^^^^^
+
+- No changes to the marconi infrastructure
+
+- we will remain with `SQLite <https://www.sqlite.org/index.html>`_ as our storage engine

doc/adr/index.rst

@@ -43,3 +43,4 @@ The general process for creating an ADR is:
    0011-support-return-and-total-collateral-when-building-transactions
    0012-commit-to-data-types-in-cardano-api
    0013-tx-validity-time-range-fix
+   0014-marconi-query-interface

marconi-mamba/README.md

@@ -1,78 +1,173 @@
-marconi-mamba
---
 
-This is an initial draft for the Marconi Mamba README file. Going forward, we will need to discuss and refine this document and/or the implementation so that they match.
+# Marconi-Mamba
 
-## Description
-marconi-mamba is a [JSON-RPC](http://www.simple-is-better.org/rpc/#differences-between-1-0-and-2-0) HTTP server on top of the [marconi](../marconi/README.md)
+Marconi-Mamba is a lightweight, customizable chain follower application and library for the Mamba project to index and query specific information from the Cardano blockchain.
+## Purpose
 
-## Playing with the JSON-RPC
-There is an example client/server JSON-RPC to experiment with the JSON-RPC clien/server. Here are the steps to build and execute the examples:
+The purpose of Marconi-Mamba is to make the core Marconi APIs available to non-Haskell applications.
 
-#### Build the project
+## Interface
 
-``` sh
-cabal build all
-cd marconi-mamba
-cabal build
+The interface for Marconi-Mamba uses [JSON-RPC](http://www.simple-is-better.org/rpc/#differences-between-1-0-and-2-0) over HTTP built on top of [Marconi](../marconi/README.md).
+
+
+```
+             Running on a single machine                    Internet
++----------------------------------------------------+
+|                                                    |                  +-------------+
+|   +----------------+                +-----------+  |                  |             |
+|   |                | node-to-client |           |  |     JSON-RPC     |    mamba    |
+|   |  cardano-node  +----------------+  marconi  +--+------------------+ application |
+|   |                |      IPC       |   mamba   |  |       HTTP       |             |
+|   +----------------+                +----+------+  |                  +------------ +
+|                                          |         |
+|                                          |         |
+|                                      +---+----+    |
+|                                      | SQLite |    |
+|                                      +--------+    |
+|                                                    |
++----------------------------------------------------+
 ```
 
-#### Execute the JSON-RPC server example
+## Requirements
+* Local instance of a compatible version of [cardano-node](https://github.com/input-output-hk/plutus-apps/blob/main/cabal.project#L246).
+* [Nix](https://nixos.org/download.html): the package manager
+* Enable [IOHK's binary cache](https://iohk.zendesk.com/hc/en-us/articles/900000673963-Installing-Nix-on-Linux-distribution-and-setting-up-IOHK-binaries)
+
+## Building from source
+To build Marconi-Mamba from the source files, use the following commands: 
 
 ``` sh
-cd ./marconi-mamba/examples
-./start-json-rpc-server.sh
+git clone git@github.com:input-output-hk/plutus-apps.git
+nix-shell
+cabal clean
+cabal update
+cabal build marconi-mamba
 ```
-Note that the example json-rpc-server relies on test data which may change in future.
 
-#### Execute the JSON-RPC client example
+The above process will build the executable in your local environment at this location:
 
 ``` sh
- $(cabal exec -- which examples-jsonrpc-client)
+ cabal exec -- which marconi-mamba
 ```
 
-## Experimenting with marconi-mamba
+## Command line summary
 
-Use the CLI to configure the runtime time environment. To get a list of available options:
+The following is a general synopsis of the command line options: 
 
 ``` sh
 $(cabal exec -- which marconi-mamba) --help
+marconi-mamba - Cardano blockchain indexer
 
 Usage: marconi-mamba --socket-path FILE --utxo-db FILE [--http-port HTTP-PORT]
                      (--mainnet | --testnet-magic NATURAL)
                      (--addresses-to-index ARG)
 
 Available options:
-  --socket-path FILE       Socket path to node
+  --socket-path FILE       Socket path to node.
   --utxo-db FILE           Path to the utxo database.
-  --http-port HTTP-PORT    JSON-RPC http port number, default is port 3000.
+  --http-port HTTP-PORT    JSON-RPC http port number. Default is port 3000.
   --mainnet                Use the mainnet magic id.
   --testnet-magic NATURAL  Specify a testnet magic id.
-  --addresses-to-index ARG Becch32 Shelley addresses to index. i.e
+  --addresses-to-index ARG Becch32 Shelley addresses to index, i.e.,
                            "--address-to-index address-1 --address-to-index
                            address-2 ..."
   -h,--help                Show this help text
 ```
 
+## Example of using Marconi-Mamba
+
+To use Marconi-Mamba, follow these steps: 
+1. Invoke the JSON-RPC server
+2. Interrogate the JSON-RPC endpoints
+3. Interrogate the REST endpoints
+
+These steps are described in more detail below. 
 
-Here is a sample invocation :
+## Invoking the JSON-RPC server
+
+The following is an example shell script for executing Marconi-Mamba in [preview-testnet](https://book.world.dev.cardano.org/environments.html#preview-testnet). 
+
+### Requirement
+
+Compatible version of [cardano-node](https://github.com/input-output-hk/plutus-apps/blob/main/cabal.project#L246) must be running with the socket-path as outlined below:
 
 ``` sh
+#!/usr/bin/env sh
+
+CARDANO_NODE_DIR=../cardano-node
+MAINNET_DIR="$CARDANO_NODE_DIR/.cardano-node/mainnet"
+PREVIEW_TESTNET_DIR="$CARDANO_NODE_DIR/.cardano-node/preview-testnet"
+
+CONFIG_DIR=$PREVIEW_TESTNET_DIR
+CARDANO_NODE_SOCKET_PATH="$CONFIG_DIR/cardano-node.socket"
+
+DB="./.marconidb/4"
+mkdir -p $DB
+
 $(cabal exec -- which marconi-mamba) \
     --testnet-magic 2 \
     --socket-path "$CARDANO_NODE_SOCKET_PATH" \
     --utxo-db "$DB/utxo-db" \
     --addresses-to-index addr_test1vpfwv0ezc5g8a4mkku8hhy3y3vp92t7s3ul8g778g5yegsgalc6gc \
     --addresses-to-index addr_test1vp8cprhse9pnnv7f4l3n6pj0afq2hjm6f7r2205dz0583egagfjah \
     --addresses-to-index addr_test1wpzvcmq8yuqnnzerzv0u862hmc4tc8xlm74wtsqmh56tgpc3pvx0f \
-    --addresses-to-index addr_test1vrvf7yfr2h79mtzqrpcn0ql98xrhs63k85w64u8py7709zsm6tsr6 \
+    --addresses-to-index addr_test1wrn2wfykuhswv4km08w0zcl5apmnqha0j24fa287vueknasq6t4hc \
+    --addresses-to-index addr_test1qr30nkfx28r452r3006kytnpvn39zv7c2m5uqt4zrg35mly35pesdyk43wnxk3edkkw74ak56n4zh67reqjhcfp3mm7qtyekt4 \
+    --addresses-to-index addr_test1wr9gquc23wc7h8k4chyaad268mjft7t0c08wqertwms70sc0fvx8w \
     --addresses-to-index addr_test1vqeux7xwusdju9dvsj8h7mca9aup2k439kfmwy773xxc2hcu7zy99
 ```
-Assumption:
-A suitable version of cardano-node is running and the environment variable `CARDANO_NODE_SOCKET_PATH` is set.
 
-#### Test with the rest-client
+## Interrogating the JSON-RPC endpoints
+
+``` sh
+|-----------+-----------+------------------------+---------------------------------------------|
+| HTTP Verb | Endpoints | RPC method             | Description                                 |
+|-----------+-----------+------------------------+---------------------------------------------|
+| POST      | json-rpc  | addresseesBech32Report | Retrieves user provided addresses           |
+| POST      | json-rpc  | utxoReport             | Retrieves TxRefs for an address             |
+| POST      | json-rpc  | utxoReportx            | Retrieves TxRefs for all provided addresses |
+| POST      | JSON-rpc  | echo                   | echo's user input to console                |
+|-----------+-----------+------------------------+---------------------------------------------|
+```
+
+## Interrogating the REST endpoints
+
+``` sh
+|-----------+-----------+-----------------------------------|
+| HTTP Verb | Endpoints | Description                       |
+|-----------+-----------+-----------------------------------|
+| GET       | addresses | Retrieves user provided addresses |
+| GET       | time      | current local time                |
+|-----------+-----------+-----------------------------------|
+```
+
+## Examples
+
+Here is a curl script to exploit the JSON-RPC server:
+
+``` sh
+curl -d '{"jsonrpc": "2.0" , "method": "utxoTxOutReport" , "params": "addr_test1vpfwv0ezc5g8a4mkku8hhy3y3vp92t7s3ul8g778g5yegsgalc6gc"  , "id": 12}' -H 'Content-Type: application/json' -X POST http://localhost:3000/json-rpc
+
+{
+  "id": 12,
+  "jsonrpc": "2.0",
+  "result": {
+    "bech32Address": "addr_test1vpfwv0ezc5g8a4mkku8hhy3y3vp92t7s3ul8g778g5yegsgalc6gc",
+    "txOutRefs": [
+      "00022d3d74304c089dc76620d6a39089c008862165c69200d82a34f53a6b93dc#0",
+      "000b106461150f070409f1183efc832a4a95c25f0083777648a1a2f3f9dbe8bc#0",
+      "001351bd6e2f09e275c4ab32570f639fb1ac17d9e6bf6ae8bf7d451aa368336f#0",
+      "00167959f52fab36abad607384a8254ce0e3d4d014147712f7caffbbe6b51385#0",
+      "0017ae0b0a0466c9728ae5040afa484d4f9797da2935db045f08de85aaec32ce#0",
+      "001a23cceafe105d7efd235cf258a7d0d78f2ce335607e8d52e7e25618c7bd90#0",
+      "001c537acf3c98d84d5cb419243dc1f8f14075a787e85360dced600b9b4ef5bb#0",
+      "00260d5e8f70dd061ae3ab4c090ab4d65e49f6501fde980eff1f35d66520f014#0",
+      "002c625c00587f65c606904bb661d3021d45a97ed254d13f117895a5a5dddfba#0",
+      "003dc4490c76a2aa7d3abe53a88918cdbee118fd5cf6436ea2c7e797abd03cdb#0"
+    ]
+  }
+}
+```
 
-To test with the [rest-client](https://github.com/pashky/restclient.el)
-- execute the JSON-RPC server example as outlined above
-- execute the invidual tests in [test-with-rest-client.http](./examples/test-with-rest-client.http)
+[test-json-rpc.http](./examples/test-json-rpc.http) contains additional example usage.

marconi-mamba/examples/README.md

@@ -0,0 +1,27 @@
+## Marconi-Mamba examples
+
+This library demonstrates end-to-end examples of creating and executing a sample JSON-RPC server and client. 
+
+### Example JSON-RPC server
+
+The main purpose of this example is to run Marconi-Mamba in isolation without a need for cardano-node.
+
+#### Usage
+
+``` sh
+$(cabal exec -- which  examples-json-rpc-server) --help
+Usage: examples-json-rpc-server [-d|--utxo-db FILENAME]
+                                (--addresses-to-index ARG)
+
+Available options:
+  -d,--utxo-db FILENAME    Path to the utxo database.
+                           (default: "./.marconidb/4/utxo-db")
+  --addresses-to-index ARG Becch32 Shelley addresses to index. i.e
+                           "--address-to-index address-1 --address-to-index
+                           address-2 ..."
+  -h,--help                Show this help text
+```
+
+### Assumption
+
+The --utxo-db points to a file that already contains utxo data.

marconi-mamba/examples/json-rpc-server/src/Main.hs

@@ -1,29 +1,28 @@
 {-
 -- Sample JSON-RPC server program
 --
+-- Often we need to test the JSON-RPC http server without the cermony of marconi, or marconi mamba.
+-- The purpose of this exampl JSON-RPC server is to test the cold-store, SQLite, flow.
+-- The assumption is that at some point in the past marconi had been executed and there is SQLite databse available
+-- The server uses CLI parameters to connect to SQLite
+-- See `start-json-rpc-server.sh` for detail
 -}
 module Main where
 
 import Cardano.Api (NetworkId (Mainnet))
 import Control.Concurrent (threadDelay)
 import Control.Concurrent.Async (race_)
-import Control.Concurrent.STM (atomically, putTMVar, takeTMVar, tryReadTMVar)
-import Control.Exception (bracket_)
+import Control.Concurrent.STM (atomically, putTMVar)
 import Control.Lens.Operators ((^.))
-import Control.Monad (unless)
-import Options.Applicative (Parser, execParser, help, helper, info, long, metavar, short, showDefault, strOption, value,
-                            (<**>))
+import Options.Applicative (Parser, execParser, help, helper, info, long, metavar, short, strOption, (<**>))
 
-import Marconi.Api.Types (DBQueryEnv, HasDBQueryEnv (queryComm), HasJsonRpcEnv (queryEnv),
-                          HasUtxoQueryComm (indexer, queryReq), TargetAddresses)
+import Marconi.Api.Types (DBQueryEnv, HasDBQueryEnv (queryTMVar), HasJsonRpcEnv (queryEnv), TargetAddresses,
+                          unUtxoIndex)
 import Marconi.Bootstrap (bootstrapHttp, bootstrapJsonRpc)
 import Marconi.CLI (multiString)
 import Marconi.Index.Utxo (Depth (Depth), open)
 
 
-defaultDbpath :: FilePath
-defaultDbpath = "./.marconidb/4/utxo-db"
-
 data CliOptions = CliOptions
     { _utxoPath  :: FilePath -- ^ path to utxo sqlite database
     , _addresses :: TargetAddresses
@@ -34,9 +33,7 @@ cliParser = CliOptions
     <$> strOption (long "utxo-db"
                               <> short 'd'
                               <> metavar "FILENAME"
-                              <>  showDefault
-                              <> value defaultDbpath
-                              <> help "Path to the utxo database.")
+                              <> help "Path to the utxo SQLite database.")
      <*> multiString (long "addresses-to-index"
                         <> help ("Becch32 Shelley addresses to index."
                                  <> " i.e \"--address-to-index address-1 --address-to-index address-2 ...\"" ) )
@@ -52,17 +49,12 @@ main = do
     env <- bootstrapJsonRpc dbpath Nothing addresses Mainnet
     race_ (bootstrapHttp env) (mocUtxoIndexer (env ^. queryEnv) )
 
+-- | moc marconi utxo indexer.
+-- This will allow us to use the UtxoIndexer query interface without having cardano-node or marconi online
+-- Effectively we are going to query SQLite only
 mocUtxoIndexer :: DBQueryEnv -> IO ()
-mocUtxoIndexer env = open "" (Depth 4) >>= innerLoop
+mocUtxoIndexer env =
+        open "" (Depth 4) >>= atomically . (putTMVar utxoIndexer) >> innerLoop
     where
-        utxoIndexer = env ^. queryComm . indexer
-        qreq = env ^. queryComm . queryReq
-        innerLoop ix = do
-            isquery <-  atomically $ tryReadTMVar qreq
-            unless (null isquery) $ bracket_
-                (atomically (takeTMVar qreq ) )
-                (atomically (takeTMVar qreq ) )
-                (atomically  (putTMVar utxoIndexer ix) )
-
-            threadDelay 10  --  inserts to sqlite
-            (innerLoop ix)
+        utxoIndexer = unUtxoIndex $ env ^. queryTMVar
+        innerLoop = threadDelay 1000 >> innerLoop

marconi-mamba/examples/start-json-rpc-server.sh

@@ -0,0 +1,32 @@
+#!/usr/bin/env sh
+
+PLUTUS_APPS_DIR=../..
+
+DB="$PLUTUS_APPS_DIR/.marconidb/7"
+mkdir -p $DB
+ls -la $DB
+
+## Noes
+## We use db-utils-exe/exe/Main.hs to get a list of most re occuring addresses from utxo-db
+## Therefor these addresses my not always be the best addresses to use.
+##
+cabal run examples-json-rpc-server -- \
+    --utxo-db "$DB/utxo-db" \
+    --addresses-to-index addr_test1qqwh05w69g95065zlr4fef4yfpjkv8r3dr9h3pu0egy5n7pwhxp4f95svdjr9dmtqumqcs6v49s6pe7ap4h2nv8rcaasgrkndk \
+    --addresses-to-index addr_test1wr9gquc23wc7h8k4chyaad268mjft7t0c08wqertwms70sc0fvx8w \
+    --addresses-to-index addr_test1qpp0xdvdexl4t3ur4svdkv3jjs2gy8fu6h9mrrsgvm7r20cmrhy4p5ukjknv23jy95nhsjsnud6fxkjqxp5ehvn8h0es2su3gt \
+    --addresses-to-index addr_test1qzu6at6s7r7yzpvw3mm2tsas34mc9nzrhda89w59wd78lfaw8084xldqyrvxe38z4wxqqdr9h86t8ruut8rrfezdftpstsnrd8 \
+    --addresses-to-index addr_test1vqslp49gcrvah8c9vjpxm4x9j7s2m4zw0gkscqh0pqjg4wcjzvcfr \
+    --addresses-to-index addr_test1wrn2wfykuhswv4km08w0zcl5apmnqha0j24fa287vueknasq6t4hc \
+    --addresses-to-index addr_test1qr30nkfx28r452r3006kytnpvn39zv7c2m5uqt4zrg35mly35pesdyk43wnxk3edkkw74ak56n4zh67reqjhcfp3mm7qtyekt4 \
+    --addresses-to-index addr_test1qrr6urmwy2nle3xppnjg9xcukasrwfyfjv9eqh97km84ra53q4hau90tjeldx0mv9eka2z73t9727xl8jny3cy8zetqsdjctpd \
+--addresses-to-index addr_test1vpfwv0ezc5g8a4mkku8hhy3y3vp92t7s3ul8g778g5yegsgalc6gc \
+--addresses-to-index addr_test1wz3937ykmlcaqxkf4z7stxpsfwfn4re7ncy48yu8vutcpxgnj28k0 \
+--addresses-to-index addr_test1vqeux7xwusdju9dvsj8h7mca9aup2k439kfmwy773xxc2hcu7zy99 \
+--addresses-to-index addr_test1qpxtftdcvh55twn2lum4uylxshzls6489q89pft3feesq2m8an0x234jtsqra93hcefmut6cyd6cdn535nkjukhph47ql6uvg7 \
+--addresses-to-index addr_test1qz8q8ymsty33sw7mh4s2wxj2pe4mcrlchxxa37z70l348cyjd3dlf08q9usapw5gt5t8cp8lju7wtwqzk5cj0gxaxyss6w8n66 \
+--addresses-to-index addr_test1vrvf7yfr2h79mtzqrpcn0ql98xrhs63k85w64u8py7709zsm6tsr6 \
+--addresses-to-index addr_test1vz8q8ymsty33sw7mh4s2wxj2pe4mcrlchxxa37z70l348cqk95hdw \
+--addresses-to-index addr_test1wqgden0j2d7pkqy3hu6kcj32swazzy6wg93a8c46pndptncdmz6tq \
+--addresses-to-index addr_test1qpe6s9amgfwtu9u6lqj998vke6uncswr4dg88qqft5d7f67kfjf77qy57hqhnefcqyy7hmhsygj9j38rj984hn9r57fswc4wg0 \
+--addresses-to-index addr_test1qz4ll7yrah8h5t3cv2qptn4mw22judsm9j9zychhmtuuzmszd3hm6w02uxx6h0s3qgd4hxgpvd0qzklnmahcx7v0mcysptyj8l