cardano-tools: new project (squashed)

cabal.project

@@ -18,6 +18,7 @@ packages: ./ouroboros-network-testing
           ./ouroboros-consensus-shelley
           ./ouroboros-consensus-shelley-test
           ./ouroboros-consensus-test
+          ./ouroboros-consensus-cardano-tools
           ./ntp-client
           ./cardano-client
 
@@ -84,6 +85,9 @@ package ouroboros-consensus-cardano
 package ouroboros-consensus-cardano-test
   tests: True
 
+package ouroboros-consensus-cardano-tools
+  tests: True
+
 
 package cardano-binary
   tests: False

ouroboros-consensus-cardano-tools/.gitignore

@@ -0,0 +1,2 @@
+test/disk/chaindb/
+

ouroboros-consensus-cardano-tools/Documentation.md

@@ -0,0 +1,110 @@
+# db-analyser
+
+## About
+This tool was initially developed to help Consensus debugging some issues, while the team was still working on Shelley. Later it was recognized that db-analyser might be potentially used by other teams when benchmarking / profiling some part of the code base.
+
+## Running the tool
+
+When you run db-analyser without any arguments, it will print out a nice helper message
+
+```
+cabal build ouroboros-consensus-cardano:db-analyser
+
+Missing: --db PATH COMMAND
+
+Usage: db-analyser --db PATH
+                   [--verbose]
+                   [--only-immutable-db [--analyse-from SLOT_NUMBER]]
+                   [--validate-all-blocks | --minimum-block-validation]
+				   COMMAND
+                   [--show-slot-block-no |
+				    --count-tx-outputs |
+                    --show-block-header-size |
+					--show-block-txs-size |
+                    --show-ebbs |
+					--store-ledger SLOT_NUMBER]
+                   [--num-blocks-to-process INT]
+  Simple framework used to analyse a Chain DB
+
+```
+
+Let's now try to break each option down.
+
+### --db PATH
+
+```
+Missing: --db PATH COMMAND
+
+Usage: db-analyser --db PATH
+```
+
+The tool works on a cardano-node's ChainDB. Thus the user must provide an obligatory `--db PATH` argument pointing to the particular DB.
+
+### --verbose
+
+db-analyser will get quite noisy
+
+### --only-immutable-db
+
+By default db-analyser will process all blocks from the current chain. That is from the genesis up to the current tip. In order to do this it must first properly initialize the whole ChainDB. That means that before it even starts processing blocks it will:
+
+1. look for latest snapshot stored in DB_PATH/ledger
+2. load that snapshot into memory
+3. start replaying blocks
+   * starting from that ledger state
+   * while updating the ledger state in the process for each replayed block
+   * keeping the intermediate results (ledger states) in memory while replaying blocks that live in the volatile DB (less than k blocks from the tip)
+
+This may heavily impact any profiling that the user might be interested in doing.
+
+To counter that problem `--only-immutable-db` flag was introduced.
+
+```
+[--only-immutable-db [--analyse-from SLOT_NUMBER]]
+```
+
+When enabled, db-analyser will work only with blocks from immutableDB, thus initialization described above will not happen.
+
+This flag comes with an additional `--analyse-from` flag. It allows to start processing blocks from the requested slot number. A snapshot at that slot number must exist in `DB_PATH/ledger/SLOT_NUMBER_db-analyser` - where `SLOT_NUMBER` is the value provided by the user with the `--analyse-from` flag.
+The user can use snapshots created by the node or they can create their own snapshots via db-analyser - see the `--store-ledger` command
+
+### COMMAND
+
+There are three options: `byron`, `shelley`, `cardano`. When in doubt which one to use, use `cardano`.
+
+* `byron`
+
+User should run this if they are dealing with Byron only chain. When the command is `byron` then user must provide `--configByron PATH` pointing to a byron configuration file.
+
+* `shelley`
+
+User should run this if they are dealing with Shelley only chain (neither Byron nor Allegra or any other era that comes after). When the command is `shelley` then user must provide `--configShelley PATH` pointing to a shelley configuration file. They may also provide `--genesisHash HASH` and `--threshold THRESHOLD`
+
+* `cardano`
+User should run this if they are dealing with `cardano` chain - that is a chain that has Byron, Shelley, Allegra, Mary, Alonzo blocks in it. When the command is `cardano` user must provide configuration for both `byron` and `shelley` (as described above) with additional `--configAlonzo PATH` pointing to Alonzo configuration file
+
+### --num-blocks-to-process
+
+```
+[--num-blocks-to-process INT]
+```
+
+The user can limit the maximum number of blocks that db-analyser will process.
+
+### Analysis
+
+Lastly the user must provide the analysis they want to run on the chain. They must select one of below:
+
+* `--show-slot-block-no` Will print the slot and block number of each block it process
+
+* `--count-tx-outputs` Will print the block and slot number, tx out output for given block and the cumulative tx out output for all the blocks seen so far
+
+* `--show-block-header-size` Will show block header size for each block and also the maximum head size it has seen in the whole chain it processed
+
+* `--show-block-txs-size` Will print number of transactions and transactions size per each block
+
+* `--show-ebbs` Will print all EBB blocks including their hash, previous block hash and a boolean value whether it is a known EBB (list of known EBBs stored in module `Ouroboros.Consensus.Byron.EBBs`)
+
+* `--store-ledger SLOT_NUMBER` Will store a snapshot of a ledger state under `DB_PATH/ledger/SLOT_NUMBER_db-analyser`. If there is no block under requested slot number, it will create one on the next available slot number (and issue a warning about this fact).
+
+* `--count-blocks` Will print out the number of blocks it saw on the chain

ouroboros-consensus-cardano-tools/NOTICE

@@ -0,0 +1,13 @@
+Copyright 2022 Input Output (Hong Kong) Ltd.
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

ouroboros-consensus-cardano-tools/app/DBAnalyser/Parsers.hs

@@ -0,0 +1,214 @@
+{-# LANGUAGE ApplicativeDo   #-}
+{-# LANGUAGE RecordWildCards #-}
+
+module DBAnalyser.Parsers (parseCmdLine) where
+
+import           Cardano.Tools.DBAnalyser.Block.Byron
+import           Cardano.Tools.DBAnalyser.Block.Cardano
+import           Cardano.Tools.DBAnalyser.Block.Shelley
+import           Cardano.Tools.DBAnalyser.Types
+
+import           Data.Foldable (asum)
+import           Options.Applicative
+
+import           Cardano.Crypto (RequiresNetworkMagic (..))
+import           Ouroboros.Consensus.Block (SlotNo (..))
+import           Ouroboros.Consensus.Byron.Node (PBftSignatureThreshold (..))
+import           Ouroboros.Consensus.Shelley.Node (Nonce (..))
+import           Ouroboros.Consensus.Storage.LedgerDB.OnDisk (DiskSnapshot (..))
+
+
+{-------------------------------------------------------------------------------
+  Parsing
+-------------------------------------------------------------------------------}
+
+parseCmdLine :: Parser DBAnalyserConfig
+parseCmdLine = DBAnalyserConfig
+    <$> strOption (mconcat [
+            long "db"
+          , help "Path to the Chain DB"
+          , metavar "PATH"
+          ])
+    <*> switch (mconcat [
+            long "verbose"
+          , help "Enable verbose logging"
+          ])
+    <*> parseSelectDB
+    <*> parseValidationPolicy
+    <*> blockTypeParser
+    <*> parseAnalysis
+    <*> parseLimit
+
+parseSelectDB :: Parser SelectDB
+parseSelectDB = asum [
+    SelectImmutableDB . snd <$> ((,) <$> onlyImmutableDB <*> analyseFrom)
+  , pure SelectChainDB
+  ]
+  where
+    onlyImmutableDB = flag' () (mconcat [
+        long "only-immutable-db"
+      , help "Validate only the Immutable DB (e.g. do not do ledger validation)"
+      ])
+
+    analyseFrom :: Parser (Maybe DiskSnapshot)
+    analyseFrom = optional $ ((flip DiskSnapshot $ Just "db-analyser") . read) <$> strOption
+      (  long "analyse-from"
+      <> metavar "SLOT_NUMBER"
+      <> help "Start analysis from ledger state stored at specific slot number" )
+
+
+parseValidationPolicy :: Parser (Maybe ValidateBlocks)
+parseValidationPolicy = parseMaybe $ asum [
+      flag' ValidateAllBlocks $ mconcat [
+          long "validate-all-blocks"
+        , help "Validate all blocks of the Volatile and Immutable DB"
+        ]
+    , flag' MinimumBlockValidation $ mconcat [
+          long "minimum-block-validation"
+        , help "Validate a minimum part of the Volatile and Immutable DB"
+        ]
+    ]
+
+parseAnalysis :: Parser AnalysisName
+parseAnalysis = asum [
+      flag' ShowSlotBlockNo $ mconcat [
+          long "show-slot-block-no"
+        , help "Show slot and block number of all blocks"
+        ]
+    , flag' CountTxOutputs $ mconcat [
+          long "count-tx-outputs"
+        , help "Show number of transaction outputs per block"
+        ]
+    , flag' ShowBlockHeaderSize $ mconcat [
+          long "show-block-header-size"
+        , help "Show the header sizes of all blocks"
+        ]
+    , flag' ShowBlockTxsSize $ mconcat [
+          long "show-block-txs-size"
+        , help "Show the total transaction sizes per block"
+        ]
+    , flag' ShowEBBs $ mconcat [
+          long "show-ebbs"
+        , help "Show all EBBs and their predecessors"
+        ]
+    , storeLedgerParser
+    , flag' CountBlocks $ mconcat [
+          long "count-blocks"
+        , help "Count number of blocks processed"
+        ]
+    , checkNoThunksParser
+    , flag' TraceLedgerProcessing $ mconcat [
+          long "trace-ledger"
+        , help $ "Maintain ledger state and trace ledger phases in the GHC event"
+                <> " log. The db-analyser tool performs era-specific analysis"
+                <> " of the ledger state and inserts markers for 'significant'"
+                <> " events, such as for example epoch transitions."
+        ]
+    , fmap ReproMempoolAndForge $ option auto $ mconcat [
+          long "repro-mempool-and-forge"
+        , help $ "Maintain ledger state and mempool trafficking the"
+              <> " transactions of each block. The integer is how many"
+              <> "blocks to put in the mempool at once."
+        , metavar "INT"
+        ]
+    , pure OnlyValidation
+    ]
+
+storeLedgerParser :: Parser AnalysisName
+storeLedgerParser = (StoreLedgerStateAt . SlotNo) <$> option auto
+  (  long "store-ledger"
+  <> metavar "SLOT_NUMBER"
+  <> help "Store ledger state at specific slot number" )
+
+checkNoThunksParser :: Parser AnalysisName
+checkNoThunksParser = CheckNoThunksEvery <$> option auto
+  (  long "checkThunks"
+  <> metavar "BLOCK_COUNT"
+  <> help "Check the ledger state for thunks every n blocks" )
+
+parseLimit :: Parser Limit
+parseLimit = asum [
+    Limit <$> option auto (mconcat [
+        long "num-blocks-to-process"
+      , help "Maximum number of blocks we want to process"
+      , metavar "INT"
+      ])
+  , pure Unlimited
+  ]
+
+blockTypeParser :: Parser BlockType
+blockTypeParser = subparser $ mconcat
+  [ command "byron"
+      (info (parseByronType   <**> helper) (progDesc "Analyse a Byron-only DB"))
+  , command "shelley"
+      (info (parseShelleyType <**> helper) (progDesc "Analyse a Shelley-only DB"))
+  , command "cardano"
+      (info (parseCardanoType <**> helper) (progDesc "Analyse a Cardano DB"))
+  ]
+
+parseByronType :: Parser BlockType
+parseByronType = ByronBlock <$> parseByronArgs
+
+parseShelleyType :: Parser BlockType
+parseShelleyType = ShelleyBlock <$> parseShelleyArgs
+
+parseCardanoType :: Parser BlockType
+parseCardanoType = CardanoBlock <$> parseCardanoArgs
+
+parseMaybe ::  Parser a -> Parser (Maybe a)
+parseMaybe parser = asum [Just <$> parser, pure Nothing]
+
+
+{-------------------------------------------------------------------------------
+  Parse BlockType-specific arguments
+-------------------------------------------------------------------------------}
+
+parseCardanoArgs :: Parser CardanoBlockArgs
+parseCardanoArgs = do
+    configFile           <- parseConfigFile
+    threshold            <- parsePBftSignatureThreshold
+    pure CardanoBlockArgs {..}
+
+parseShelleyArgs :: Parser ShelleyBlockArgs
+parseShelleyArgs = ShelleyBlockArgs
+    <$> strOption (mconcat [
+            long "configShelley"
+          , help "Path to config file"
+          , metavar "PATH"
+          ])
+    <*> asum [ Nonce  <$> parseNonce
+             , pure NeutralNonce]
+  where
+    parseNonce = strOption (mconcat [
+            long "nonce"
+          , help "Initial nonce, i.e., hash of the genesis config file"
+          , metavar "NONCE"
+          ])
+
+parseConfigFile :: Parser FilePath
+parseConfigFile = strOption $ mconcat [
+      long "config"
+    , help "Path to config file"
+    , metavar "PATH"
+    ]
+
+parsePBftSignatureThreshold :: Parser (Maybe PBftSignatureThreshold)
+parsePBftSignatureThreshold = optional $ fmap PBftSignatureThreshold $ option auto $ mconcat [
+      long "threshold"
+    , help "PBftSignatureThreshold"
+    , metavar "THRESHOLD"
+    ]
+
+parseByronArgs :: Parser ByronBlockArgs
+parseByronArgs = ByronBlockArgs
+    <$> parseConfigFile
+    <*> flag RequiresNoMagic RequiresMagic (mconcat [
+            long "requires-magic"
+          , help "The DB contains blocks from a testnet, requiring network magic, rather than mainnet"
+          ])
+    <*> optional (option auto (mconcat [
+            long "genesisHash"
+          , help "Expected genesis hash"
+          , metavar "HASH"
+          ]))
+    <*> parsePBftSignatureThreshold