tools: block-generator applications. Part 1: create #5450

tzaffi · 2023-06-05T03:47:37Z

Summary

Handle application creation in the block generator including local storage.

TODO in a follow up PR

create boxes and populate app_boxes
handle additional app calls, most pressingly simple app calls show the results of typical app usage on DB ingestion

Blocked by:

Test Plan

Some unit tests are added. However, most of the testing is done locally. You can try at home as follows:

NOTE: I mixed/matched some outputs from various runs so the numbers below are inconsistent

❯ cd tools/block-generator
❯ go build
❯ ln -s ~/github/algorand/conduit/cmd/conduit/conduit conduit
❯ make debug-blockgen
... this brings up the postgres docker and prints out some info in case you want to run directly or through vs-code

❯ docker ps
CONTAINER ID   IMAGE      COMMAND                  CREATED         STATUS         PORTS                     NAMES
652e8cc02075   postgres   "docker-entrypoint.s…"   8 seconds ago   Up 7 seconds   0.0.0.0:15432->5432/tcp   generator-test-container

❯ make run-runner
./block-generator runner --conduit-binary ./conduit \
        --log-level trace \
        --keep-data-dir \
        --report-directory RUN_RUNNER_OUTPUTS \
        --test-duration 30s --log-level trace \
        --postgres-connection-string "host=localhost user=algorand password=algorand dbname=generator_db port=15432 sslmode=disable" \
        --scenario scenarios/config.app.create.yml  \

Running test for configuration 'scenarios/config.app.create.yml'
generator serving on localhost:11112
Received round request 0 == g.roundOffset but already advanced g.round=1. Not finishing round.
exiting block generator runner: signal: interrupt
Done running tests!

❯ cat RUN_RUNNER_OUTPUTS/config.app.create.report
scenario:config.app.create.yml
test_duration_seconds:30
test_duration_actual_seconds:30.022316
transaction_asset_create_total:14192
transaction_app_boxes_create_total:7171
transaction_app_swap_create_total:7083
early_average_import_time_sec:0.07
early_cumulative_import_time_sec:6.23
early_average_imported_tx_per_block:100.00
early_cumulative_imported_tx_per_block:9500
early_imported_round:95
early_overall_transactions_per_second:1524.83
early_uptime_seconds:30.03
final_average_import_time_sec:0.07
final_cumulative_import_time_sec:20.48
final_average_imported_tx_per_block:100.00
final_cumulative_imported_tx_per_block:28400
final_imported_round:284
final_overall_transactions_per_second:1386.54
final_uptime_seconds:30.03

❯ make enter-pg
docker exec -it generator-test-container psql -U algorand -d generator_db
psql (14.5 (Debian 14.5-1.pgdg110+1))
Type "help" for help.

generator_db=# \dt
               List of relations
 Schema |       Name        | Type  |  Owner   
--------+-------------------+-------+----------
 public | account           | table | algorand
 public | account_app       | table | algorand
 public | account_asset     | table | algorand
 public | app               | table | algorand
 public | app_box           | table | algorand
 public | asset             | table | algorand
 public | block_header      | table | algorand
 public | metastate         | table | algorand
 public | txn               | table | algorand
 public | txn_participation | table | algorand
(10 rows)

generator_db=# select count(*) from account;
 count 
-------
  1002
(1 row)

generator_db=# select count(*) from account_app;
 14899
generator_db=# select count(*) from account_asset;
 16382
generator_db=# select count(*) from app;
 16318
generator_db=# select count(*) from app_box;
     0
generator_db=# select count(*) from asset;
 16382
generator_db=# select count(*) from block_header;
   632
generator_db=# select count(*) from txn;
 62600
generator_db=# select count(*) from txn_participation;
 62600

generator_db=# exit
❯ make cleanup
rm -rf reports

tools/block-generator/generator/generate.go

tools/block-generator/generator/generator_types.go

tools/block-generator/generator/generate.go

tools/block-generator/generator/make_transactions.go

tzaffi · 2023-06-09T14:44:33Z

tools/block-generator/generator/make_transactions.go

 	"github.com/algorand/go-algorand/protocol"
 )

-func (g *generator) makeTxnHeader(sender basics.Address, round, intra uint64) transactions.Header {
+// ---- header / boilerplate ----
+


this diff is confusing. I didn't actually modify makeTxnHeader except for renaming the transactions import (→ txn). But I did introduce makeTestTxn which is very similar to makeTxnHeader and confused the differ.

tzaffi · 2023-06-09T14:45:42Z

tools/block-generator/generator/make_transactions.go

@@ -20,15 +20,34 @@ import (
 	"encoding/binary"

 	"github.com/algorand/go-algorand/data/basics"
-	"github.com/algorand/go-algorand/data/transactions"
+	txn "github.com/algorand/go-algorand/data/transactions"
+	"github.com/algorand/go-algorand/data/txntest"


txntest.Txn is the modern convenience struct for testing transactions

tools/block-generator/generator/generate.go

shiqizng · 2023-06-09T19:33:54Z

tools/block-generator/generator/generate.go

+	return signTxn(txn), transactions.ApplyData{}, nil
+}
+
+func (g *generator) generateAppCallInternal(txType TxTypeID, round, intra, hintIndex uint64, hintApp *appData) (TxTypeID, transactions.Transaction, error) {


hintIndex not used.

I'd like to keep this param until my follow up PR is ready for review, at which point I commit to removing if still unused.

Let's not resolve this discussion.

it's removed in #5478

tools/block-generator/generator/generate.go

shiqizng · 2023-06-09T19:42:17Z

tools/block-generator/generator/generate.go

+	numApps := uint64(len(g.apps[kind]))
+	if numApps == 0 {
+		actualAppTx = appTxTypeCreate
+	}


is this block necessary? you've assert at line 799 that appTx must be appTxTypeCreate

This is another one I'd like to keep till the next PR is ready, and so let's not resolve this discussion.

Logic has been fixed in #5478

shiqizng · 2023-06-09T19:48:49Z

tools/block-generator/generator/make_transactions.go

 	note := make([]byte, 8)
 	binary.LittleEndian.PutUint64(note, uint64(g.txnCounter+intra))

-	return transactions.Header{
+	return txntest.Txn{


shiqizng · 2023-06-09T19:50:52Z

tools/block-generator/generator/make_transactions.go

+}
+
+// makeTestTxn creates and populates the flat txntest.Txn structure with the given values.
+func (g *generator) makeTestTxn(sender basics.Address, round, intra uint64) txntest.Txn {
 	note := make([]byte, 8)
 	binary.LittleEndian.PutUint64(note, uint64(g.txnCounter+intra))


nit: uint64 casting is not needed; same for makeTxnHeader

shiqizng · 2023-06-09T19:56:43Z

tools/block-generator/generator/generate_test.go

+	ad = *g.pendingApps[appKindSwap][0]
+	holding = *ad.holdings[0]
+	require.Equal(t, holding, *ad.holders[0])
+	require.Equal(t, uint64(1001), holding.appIndex)


I'm surprised the appID is still 1001. I expected to have incremented to 1002.

This is because I'm not passing the new intra into generateAppCallInternal(). I'm much more explicit about this in the next PR. I propose keeping this thread open and adding it as a TODO to the follow up.

Co-authored-by: shiqizng <80276844+shiqizng@users.noreply.github.com>

winder

Just a nit. LGTM

tools/block-generator/generator/generate.go

Co-authored-by: Will Winder <wwinder.unh@gmail.com>

shiqizng and others added 30 commits May 8, 2023 16:50

updating generator

bd10013

parse genesis file

58dc5ae

run test with a genesis file

085d4b9

Merge remote-tracking branch 'upstream/master' into shiqi/generator

1321084

assetID starts at 1001

9b10fdb

update generator txnCounter

3e3cd5a

updates

732eca1

more updates

dfede1d

adding tests

fc03006

lint

6ffbad1

lint

be03d62

fix failing test

8704cd7

fix skipped tests

1d8b4a6

update tests

4971671

remove partitiontest from test util

119f646

pythonize run_runner

7ee1f04

Merge remote-tracking branch 'origin/master' into tools-blockgen-25K

23d3c84

Update .gitignore

d33ff1f

arbitrary scenario

f0f90ab

25K

9e455da

Merge remote-tracking branch 'origin/master' into tools-blockgen-25K

b032ee2

updates per review

4e02cb7

lint

d498cd9

Merge branch 'shiqi/generator' into tools-blockgen-25K

e19063c

validate generator config on read + start bldg app txn setup

c234270

tweaks

c1da5d2

Merge branch 'tools-blockgen-25K' into tools-blockgen-apps

ea931e7

url

6c5c88f

query db round

6d97216

go mod tidy

516e52b

Update tools/block-generator/generator/generate.go

a419a10

tzaffi commented Jun 9, 2023

View reviewed changes

tools/block-generator/generator/generate.go Show resolved Hide resolved

tzaffi commented Jun 9, 2023

View reviewed changes

tools/block-generator/generator/generator_types.go Show resolved Hide resolved

tzaffi commented Jun 9, 2023

View reviewed changes

tools/block-generator/generator/generate.go Show resolved Hide resolved

tzaffi requested review from shiqizng and a team June 9, 2023 14:34

tzaffi changed the title ~~DRAFT tools: block-generator applications. Part 1.~~ tools: block-generator applications. Part 1. Jun 9, 2023

tzaffi changed the title ~~tools: block-generator applications. Part 1.~~ tools: block-generator applications. Part 1: create Jun 9, 2023

tzaffi commented Jun 9, 2023

View reviewed changes

tools/block-generator/generator/make_transactions.go Outdated Show resolved Hide resolved

Update tools/block-generator/generator/make_transactions.go

b2da670

tzaffi commented Jun 9, 2023

View reviewed changes

tools/block-generator/generator/make_transactions.go Show resolved Hide resolved

tzaffi commented Jun 9, 2023

View reviewed changes

shiqizng reviewed Jun 9, 2023

View reviewed changes

tools/block-generator/generator/generate.go Outdated Show resolved Hide resolved

shiqizng reviewed Jun 9, 2023

View reviewed changes

tzaffi and others added 3 commits June 9, 2023 15:07

Update tools/block-generator/generator/generate.go

f62d9a0

Co-authored-by: shiqizng <80276844+shiqizng@users.noreply.github.com>

per CR: remove unnecessary cast

028e9e8

per CR: remove redundant defer

115abe1

winder previously approved these changes Jun 12, 2023

View reviewed changes

tools/block-generator/generator/generate.go Outdated Show resolved Hide resolved

Update tools/block-generator/generator/generate.go

6d2e57f

Co-authored-by: Will Winder <wwinder.unh@gmail.com>

tzaffi dismissed winder’s stale review via 6d2e57f June 12, 2023 19:49

tzaffi requested review from shiqizng and winder June 12, 2023 19:53

winder approved these changes Jun 12, 2023

View reviewed changes

shiqizng approved these changes Jun 13, 2023

View reviewed changes

winder merged commit 4d6e50d into algorand:master Jun 15, 2023
25 checks passed

tzaffi mentioned this pull request Jun 18, 2023

tools: block generator apps. Part 2: boxes #5478

Merged

17 tasks

Algo-devops-service mentioned this pull request Jul 11, 2023

go-algorand 3.17.0-beta Release PR #5541

Merged

onetechnical mentioned this pull request Jul 24, 2023

go-algorand 3.17.0-beta Release PR #5601

Merged

Algo-devops-service mentioned this pull request Aug 3, 2023

go-algorand 3.17.0-stable Release PR #5633

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tools: block-generator applications. Part 1: create #5450

tools: block-generator applications. Part 1: create #5450

tzaffi commented Jun 5, 2023 •

edited

tzaffi Jun 9, 2023

tzaffi Jun 9, 2023 •

edited

shiqizng Jun 9, 2023

tzaffi Jun 9, 2023

tzaffi Jun 18, 2023

shiqizng Jun 9, 2023

tzaffi Jun 9, 2023

tzaffi Jun 18, 2023

shiqizng Jun 9, 2023

shiqizng Jun 9, 2023

tzaffi Jun 12, 2023

shiqizng Jun 9, 2023

tzaffi Jun 12, 2023 •

edited

winder left a comment

tools: block-generator applications. Part 1: create #5450

tools: block-generator applications. Part 1: create #5450

Conversation

tzaffi commented Jun 5, 2023 • edited

Summary

TODO in a follow up PR

Blocked by:

Test Plan

NOTE: I mixed/matched some outputs from various runs so the numbers below are inconsistent

Choose a reason for hiding this comment

tzaffi Jun 9, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tzaffi Jun 12, 2023 • edited

Choose a reason for hiding this comment

winder left a comment

Choose a reason for hiding this comment

tzaffi commented Jun 5, 2023 •

edited

tzaffi Jun 9, 2023 •

edited

tzaffi Jun 12, 2023 •

edited