Add a ledger operations benchmarking analysis to db-analyser

ouroboros-consensus-cardano-tools/Documentation.md

@@ -42,7 +42,10 @@ db-analyser will get quite noisy
 
 ### --only-immutable-db
 
-By default db-analyser will process all blocks from the current chain. That is from the genesis up to the current tip. In order to do this it must first properly initialize the whole ChainDB. That means that before it even starts processing blocks it will:
+By default db-analyser will process all blocks from the chain database
+(`ChainDB`), from the genesis block up to the chain tip. In order to do this it
+must first properly initialize the whole database. That means that before it
+even starts processing blocks it will:
 
 1. look for latest snapshot stored in DB_PATH/ledger
 2. load that snapshot into memory
@@ -89,32 +92,128 @@ The user can limit the maximum number of blocks that db-analyser will process.
 
 ### Analysis
 
-Lastly the user must provide the analysis they want to run on the chain. They must select one of below:
+Lastly the user must provide the analysis that should be run on the chain:
 
-* `--show-slot-block-no` Will print the slot and block number of each block it process
+* `--show-slot-block-no` prints the slot and block number of each block it
+  process.
 
-* `--count-tx-outputs` Will print the block and slot number, tx out output for given block and the cumulative tx out output for all the blocks seen so far
+* `--count-tx-outputs` prints the block and slot number, tx out output for given
+  block and the cumulative tx out output for all the blocks seen so far.
 
-* `--show-block-header-size` Will show block header size for each block and also the maximum head size it has seen in the whole chain it processed
+* `--show-block-header-size` shows block header size for each block and also the
+  maximum head size it has seen in the whole chain it processed.
 
-* `--show-block-txs-size` Will print number of transactions and transactions size per each block
+* `--show-block-txs-size` prints number of transactions and transactions size
+  per each block.
 
-* `--show-ebbs` Will print all EBB blocks including their hash, previous block hash and a boolean value whether it is a known EBB (list of known EBBs stored in module `Ouroboros.Consensus.Byron.EBBs`)
+* `--show-ebbs` prints all EBB blocks including their hash, previous block hash
+  and a boolean value whether it is a known EBB (list of known EBBs stored in
+  module `Ouroboros.Consensus.Byron.EBBs`).
 
-* `--store-ledger SLOT_NUMBER` Will store a snapshot of a ledger state under `DB_PATH/ledger/SLOT_NUMBER_db-analyser`. If there is no block under requested slot number, it will create one on the next available slot number (and issue a warning about this fact).
+* `--store-ledger SLOT_NUMBER` stores a snapshot of a ledger state under
+  `DB_PATH/ledger/SLOT_NUMBER_db-analyser`. If there is no block under requested
+  slot number, it will create one on the next available slot number (and issue a
+  warning about this fact).
 
-* `--count-blocks` Will print out the number of blocks it saw on the chain
+* `--count-blocks` prints out the number of blocks it saw on the chain
 
+* `--benchmark-ledger-ops` applies the main ledger calculations to each block in
+  the chain database, and collect different metrics such as total time spent,
+  time spent doing garbage collection, etc. The ledger operations that this
+  command benchmarks are: forecasting, ticking and applying the header, and
+  ticking and applying a block. The benchmarking results are stored in the file
+  specified by the `out-file` option. In the [`scripts`](./scripts) directory we
+  provide a `plot-ledger-ops-cost.gp` script that can be used to plot the
+  benchmarking results. See this file for usage information.
+
+## Examples
+
+The use of this tool requires a chain database together with the configuration
+files that were used (by `cadano-node`) to populate it (by means of chain
+syncing). Assuming you ran `cardano-node` on your machine, it is quite handy to
+save the location to the working directory from which it was run. Eg:
+
+```sh
+export NODE_HOME=/path/to/local/copy/of/cardano-node/working/dir/
+```
+
+### Saving a snapshot
+
+Suppose we have a local chain database in reachable from `$NODE_HOME`, and we
+want to take a snapshot of the ledger state for slot `100`. Then we can run:
+
+```sh
+cabal run exe:db-analyser -- cardano \
+    --config $NODE_HOME/configuration/cardano/mainnet-config.json \
+    --db $NODE_HOME/mainnet/db \
+    --only-immutable-db --store-ledger 100
+```
+
+If we had a previous snapshot of the ledger state, say corresponding to slot
+`50`, it is possible to tell `db-analyser` to start from this snapshot:
+
+```sh
+cabal run exe:db-analyser -- cardano \
+    --config $NODE_HOME/configuration/cardano/mainnet-config.json \
+    --db $NODE_HOME/mainnet/db \
+    --analyse-from 50 \
+    --only-immutable-db --store-ledger 100
+```
+
+### Running the ledger operations benchmark
+
+To benchmark the ledger operations, using the setup mentioned in the foregoing
+examples, one could run the tool as follows:
+
+```sh
+cabal run exe:db-analyser -- cardano
+    --config $NODE_HOME/configuration/cardano/mainnet-config.json \
+    --db $NODE_HOME/mainnet/db \
+    --analyse-from 100 \
+    --benchmark-ledger-ops \
+    --out-file ledger-ops-cost.csv \
+    --only-immutable-db
+```
+
+The benchmarking command can be combined with `--num-blocks-to-process` to
+specify the application of how many blocks we want to process. Eg:
+
+```sh
+cabal run exe:db-analyser -- cardano
+    --config $NODE_HOME/configuration/cardano/mainnet-config.json \
+    --db $NODE_HOME/mainnet/db \
+    --analyse-from 100 \
+    --benchmark-ledger-ops \
+    --out-file ledger-ops-cost.csv \
+    --num-blocks-to-process 200
+    --only-immutable-db
+```
+
+#### Plotting the benchmark results
+
+Assuming you are at the top level of `ouroboros-network`, and the benchmark data
+has been written to a fine named `ledger-ops-cost.csv`, a plot can be generated
+by running:
+
+```sh
+gnuplot -e "bench_data='ledger-ops-cost.csv'" \
+        -e "out_file='results.png'" \
+           ouroboros-consensus-cardano-tools/scripts/plot-ledger-ops-cost.gp 
+```
+
+The plot will be written to a file named `results.png`. See the script file for
+more usage options.
 
 # db-synthesizer
 
 ## About
-This tool synthesizes a valid ChainDB, replicating cardano-node's UX. The blocks forged to synthesize the ChainDB won't contain any transactions.
+This tool synthesizes a valid ChainDB, replicating cardano-node's UX. The blocks
+forged to synthesize the ChainDB won't contain any transactions.
 
 A minimal test case is provided which incorporates a staked genesis and credentials for 2 forgers (cf. `test/config`).
 
 ## Running the tool
-When you run db-synthsizer without any arguments, it will print out usage information:
+When you run db-synthesizer without any arguments, it will print out usage information:
 
 ```
 cabal build ouroboros-consensus-cardano-tools:db-synthesizer

ouroboros-consensus-cardano-tools/app/DBAnalyser/Parsers.hs

@@ -114,6 +114,7 @@ parseAnalysis = asum [
               <> "blocks to put in the mempool at once."
         , metavar "INT"
         ]
+    , benchmarkLedgerOpsParser
     , pure OnlyValidation
     ]
 
@@ -139,6 +140,28 @@ parseLimit = asum [
   , pure Unlimited
   ]
 
+benchmarkLedgerOpsParser :: Parser AnalysisName
+benchmarkLedgerOpsParser =
+    BenchmarkLedgerOps <$> (benchmarkLedgerOpsFlagParser *> pMaybeOutputFile)
+  where
+    benchmarkLedgerOpsFlagParser =
+      flag' BenchmarkLedgerOps $ mconcat [
+            long "benchmark-ledger-ops"
+          , help $ "Maintain ledger state and benchmark the main ledger calculations for each block."
+                  <> " Prints one line of stats per block to the given output file "
+                  <> " (defaults to stdout)."
+          ]
+
+pMaybeOutputFile :: Parser (Maybe FilePath)
+pMaybeOutputFile =
+  optional $
+    strOption
+      (  long "out-file"
+      <> metavar "FILE"
+      <> help "Optional output file. Default is to write to stdout."
+      <> completer (bashCompleter "file")
+      )
+
 blockTypeParser :: Parser BlockType
 blockTypeParser = subparser $ mconcat
   [ command "byron"

ouroboros-consensus-cardano-tools/ouroboros-consensus-cardano-tools.cabal

@@ -56,6 +56,7 @@ library
                      , serialise
                      , cardano-strict-containers
                      , text
+                     , text-builder
                      , transformers
                      , transformers-except
 
@@ -70,6 +71,8 @@ library
                      , cardano-ledger-shelley
                      , cardano-protocol-tpraos
                      , cardano-prelude
+                     , cardano-ledger-shelley-ma
+                     , cardano-slotting
 
                      , ouroboros-consensus
                      , ouroboros-consensus-byron
@@ -91,6 +94,7 @@ library
                      , Cardano.Node.Protocol.Byron
                      , Cardano.Node.Protocol.Cardano
                      , Cardano.Node.Protocol.Shelley
+                     , Cardano.Tools.DBAnalyser.Analysis.BenchmarkLedgerOps.SlotDataPoint
 
   default-language:    Haskell2010