tools: run block generator using a preloaded db #5384
Conversation
Codecov Report
@@ Coverage Diff @@
## master #5384 +/- ##
==========================================
- Coverage 55.41% 55.36% -0.06%
==========================================
Files 452 452
Lines 63643 63672 +29
==========================================
- Hits 35270 35250 -20
- Misses 25972 26009 +37
- Partials 2401 2413 +12
... and 9 files with indirect coverage changes 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
| @@ -54,4 +54,6 @@ $(dirname "$0")/block-generator runner \ | |||
| --test-duration 30s \ | |||
| --log-level trace \ | |||
| --postgres-connection-string "host=localhost user=algorand password=algorand dbname=generator_db port=15432 sslmode=disable" \ | |||
| --scenario ${CONFIG} | |||
| --scenario ${CONFIG} \ | |||
| --db-round ${2:-0} \ | |||
There was a problem hiding this comment.
Could this be read from the db at startup time to avoid needing to provide?
There was a problem hiding this comment.
yes, but it seems expensive to create a connection just to read 1 number.
There was a problem hiding this comment.
Ok. What's the exact round number I should supply? I can add that to the Makefile I'm introducing in my followup #5394
There was a problem hiding this comment.
RunnerCmd.Flags().Uint64VarP(&runnerArgs.NextDBRound, "next-db-round", "n", 0, "next round on db. It is the next_account_round from Indexer's metastate table.")
There was a problem hiding this comment.
I wonder if something like this is going to become more important when running multiple tests back-to-back.
There was a problem hiding this comment.
yes, it's not working with I provided a scenario directory instead of a scenario file path. I'll fix this in a follow up PR.
| } else { | ||
| prefix = g.genesisID | ||
| } | ||
| l, err := ledger.OpenLedger(logging.Base(), prefix, true, ledgercore.InitState{ |
There was a problem hiding this comment.
just wanted to point out that supplying GetDefaultLocal as configuration to OpenLedger may open you up to underlying changes you don't want/expect. For instance, StorageEngine is being added in this PR, which currently specifies sql, but may specify another engine in the future.
There was a problem hiding this comment.
Do all storage engines support in memory mode?
| timestamp: 0, | ||
| rewardsLevel: 0, | ||
| rewardsResidue: 0, | ||
| rewardsRate: 0, | ||
| rewardsRecalculationRound: 0, | ||
| reportData: make(map[TxTypeID]TxData), | ||
| nextdbround: dbround, |
There was a problem hiding this comment.
nit: I'd consider calling this round offset rather than nextdbround since the generator doesn't know about the DB.
There was a problem hiding this comment.
I'll address this and the other ones in a follow up PR.
| block.BlockHeader.Round = basics.Round(round) | ||
| encodedblock := rpcs.EncodedBlockCert{ | ||
| Block: block, | ||
| Certificate: cert, |
There was a problem hiding this comment.
Is this needed? agreement.Certificate{} is used below.
| @@ -451,7 +478,16 @@ func (g *generator) WriteBlock(output io.Writer, round uint64) error { | |||
|
|
|||
| // WriteDeltas generates returns the deltas for payset. | |||
| func (g *generator) WriteDeltas(output io.Writer, round uint64) error { | |||
| delta, err := g.ledger.GetStateDeltaForRound(basics.Round(round)) | |||
| // offset round for non-empty database | |||
There was a problem hiding this comment.
| // offset round for non-empty database | |
| // the first generataed round has no statedelta. |
| @@ -13,7 +13,7 @@ fi | |||
| POSTGRES_CONTAINER=generator-test-container | |||
| POSTGRES_PORT=15432 | |||
| POSTGRES_DATABASE=generator_db | |||
| CONFIG=${2:-"$(dirname $0)/test_config.yml"} | |||
| CONFIG=${4:-"$(dirname $0)/test_config.yml"} | |||
There was a problem hiding this comment.
This script needs a little extra attention if we want to keep using it. Using positional arguments means they're all required and having so many of them gets confusing. As it is, it would be easy to run the script incorrectly.
$1 - conduit binary
$2 - db-round (required if you want a config template)
$3 - genesis file (required if you want a config template)
$4 - config template (optional)
There was a problem hiding this comment.
My follow up has a python version of this that's a bit more flexible.
This PR updates the block generator to work with a db snapshot. The generator creates blocks with synthetic accounts and write to an non-empty database by using an offset value for the round.