tools: block generator apps. Part 2: boxes #5478

tzaffi · 2023-06-17T17:11:33Z

Summary

Enable opting in and calling box apps and populate database table app_box which was the last remaining table that wasn't populated.

Changes

block-generator arguments:
- rename --log-level to --conduit-log-level
- new argument --verbose
tools/block-generator/generator/config.go:
- make previously introduced types into Stringers
tools/block-generator/generator/generate.go:
- handle the fact that a requested transaction may end up with many multiple side effects
- keep track of pending app and committed app information via:
  - gen.appMap
  - gen.appSlice
  - gen.pendingAppMap
  - gen.pendingAppSlice
- introduce gen.startRound() mostly to keep the generator's txnCounter in sync with the Ledger's
  - in particular numTxnForBlock := g.txnForRound(g.round) is no longer the fixed block size but instead becomes a LOWER BOUND
  - NOTE: for the upcoming Part 3, this can become quite distorted as a single swap app call generates a total of 256 transacations
- handle the dependency between the three box app transaction types:
  - app call can only happen after app optin which can only happen after app create
- introduce gen.introspectLedgerVsGenerator() which is only called under --verbose and prints out debugging information comparing the generator's state with its ledger's stat.
tools/block-generator/generator/make_transactions.go:
- makeAppCreateTxn() - handles an additional pay txn to fund the app in the case of creating a boxes app
- introducing makeAppOptinTxn() which also:
  - pays 1 algo to fund the new box (MBR=0.412 Algos) via a sibling txn
  - creates the box with
    - key = sender's address
    - value = "z" * 992 (pythonese)
  - has 2 inner pay txns
- introducing makeAppCallTxn() which is a ABI call that logs the box contents
tools/block-generator/generator/teal/poap_boxes.teal
- adding commentary for better readability
- adding a box creation of size 1K during opt-in
new and modified scenarios:
- conforming to naming convention config.XYZ.(small|jumbo).yml
- new scanarios:
  - config.allmixed.jumbo.yml - includes all kinds of transactions
  - config.allmixed.small.yml
  - config.appboxes.jumbo.yml

TODO's ...

...which were inherited from PR Part 1

create boxes and populate app_boxes
handle additional app calls, most pressingly simple app calls show the results of typical app usage on DB ingestion
hintindex is not used
is this block necessary?

...which have been created in this PR

Test Plan

Locally test as follows:

❯ cd tools/block-generator

# start postgres in a docker container:
❯ make debug-blockgen
...

# run a 30 second "jumbo" experiment mixed with all kinds of transactions 50% of which are app related:
❯ make run-runner SCENARIO=scenarios/config.allmixed.jumbo.yml DURATION=30s VERBOSE=--verbose
... lots of round info play-by-play that will partially look like:
...
Received round request 0, but nextRound=1. Not finishing round.
--------------------
roundNumber (generator): 1
round (ledger): 1
g.txnCounter + intra: 101000
... etc ...

# generator/conduit's log and report location:
❯ ls ../../tmp/RUN_RUNNER_OUTPUTS
config.allmixed.jumbo.conduit-log config.allmixed.jumbo.report      config.allmixed.jumbo_data

# introspect the database
❯ make enter-pg # or you might just call `psql` directly if you prefer to not enter the pg container
generator_db=# \dt
               List of relations
...

generator_db=# select count(*) from block_header;
...

generator_db=# select count(*) from txn;
...

tools/block-generator/Makefile

tools/block-generator/README.md

tools/block-generator/generator/generator_types.go

tools/block-generator/Makefile

tools/block-generator/README.md

tools/block-generator/generator/generate.go

+var effects map[TxTypeID][]TxEffect
+
+func init() {
+	effects = map[TxTypeID][]TxEffect{


Co-authored-by: shiqizng <80276844+shiqizng@users.noreply.github.com>

…ts map

codecov · 2023-06-21T16:26:44Z

Codecov Report

Merging #5478 (676db87) into master (91185ae) will increase coverage by 0.33%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #5478      +/-   ##
==========================================
+ Coverage   55.30%   55.63%   +0.33%     
==========================================
  Files         446      446              
  Lines       63153    63153              
==========================================
+ Hits        34924    35135     +211     
+ Misses      25864    25641     -223     
- Partials     2365     2377      +12

see 25 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

winder · 2023-06-22T19:40:40Z

tools/block-generator/Makefile

+block-generator: clean
+	go build


Consider moving this to the first target so that make with no arguments builds the binary.

winder · 2023-06-22T19:45:49Z

tools/block-generator/generator/config.go

+	// Special TxTypeID's recording effects of higher level transactions
+	effectPaymentTxSibling TxTypeID = "effect_payment_sibling"
+	effectInnerTx          TxTypeID = "effect_inner_tx"


Are these configurable? I believe the TxTypeID was previously used for the different types of transactions that would be chosen according to the ratios. This seems more like something that would be in GenerationConfig like tx_per_block.

I think you're right that it's better to keep TxTypeID for types of transactions that can actually be chosen. I'll work to carve out a separate types for these effects.

However, one issue here is that I do need to record the effects and these get stored in a type Report map[TxTypeID]TxData whose key is of type TxTypeID. So it does complicates things a bit. If we relax Report to have keys of type string, I still ought to be able to separate it out. I'll keep you posted.

The original concern will be addressed in a follow up PR. Here's a commit: tzaffi@291d30f

tools/block-generator/generator/generate.go

winder · 2023-06-22T19:53:39Z

tools/block-generator/generator/generate.go

-	g.txnCounter += txnCount
+// startRound updates the generator's txnCounter based on the latest block header.
+// It is assumed that g.round has already been incremented in finishRound()
+func (g *generator) startRound() error {


Are we unable to manually track the txn counter? There should be a known number of inner transactions. Does it match how many the ledger things there are?

This is a great question and deserves a good answer. I'll investigate a little further.

After some investigation, I believe it would be difficult to achieve this without some significant downsides. This involves opening up BlockEvaluator.Eval() to get the TxnCounter directly from the evaluator. There is -in fact- a method BlockEvaluator.TestingTxnCounter() which is meant for this purpose. However, accessing it means interacting with the BlockEvaluator that's created during the ledger's call to eval.Eval(). One relatively non-invasive way I see of achieving this is to modify interface EvalTracer's AfterBlock() to take in a new parameter newTxnCounter that will let us trace this value and then use it in the new method that I'm introducing ledgerAddBlock(). This feels like overkill to me though. Notice that the same method does introduce an improvement: we calculate the number of transactions in the block by looking at all the actual transactions including inners that were evaluated.

Reading a bit more carefully the original comment, it's arguable that the follow up PR DOES address the concern as it:

gets the theoretical transaction count using effects

gets the actual transaction count from ledgerAddBlock()

compares them and errors if they're unequal

see:
https://github.com/tzaffi/go-algorand/blob/f631aec855695e0e984008bcafd1041a46fa0a15/tools/block-generator/generator/generate.go#L382-L388

In a perfect world, we wouldn't have to re-count the transactions and get them directly from the block evaluator, but I think this is good enough going forward.

winder · 2023-06-22T20:17:46Z

tools/block-generator/generator/generate.go

 	g.timestamp += consensusTimeMilli
 	g.round++

 	// Apply pending assets...
 	g.assets = append(g.assets, g.pendingAssets...)
 	g.pendingAssets = nil

-	// Apply pending apps...


A comment with some more explanation about whats going on here would be helpful.

I don't think the loops and cleverness around maps & kinds are worth it here. It hurts clarity and I'm not even sure if it's shorter code.

Did you do it this way because you're imagining lots of additional kinds?

The logic around keeping track of the state of each application in the ledger is the most complex part of this PR. It's about to get significantly more complex as being able to generate an AMM swap requires 7 setup steps prior and therefore much more ability to keep track of state. It's worth a discussion.

Let's figure out if there is some simple code organization to help with that for now. Maybe we can plan for some larger architecture changes in the future (i.e. using the ledger, it wasn't available in earlier versions of this tool).

It isn't production code, so introducing tech debt here is more acceptable than some of our other systems.

This depends on where the AMM ends up, so I would defer as a future discussion. I've created a TODO for this.

winder

I ran the generator and see boxes. Nice!

I'm OK merging this in even if there is followup work.

shiqizng · 2023-06-22T19:39:38Z

tools/block-generator/generator/generate.go

+//
+// appBoxesCreate: 1 sibling payment tx
+// appBoxesOptin: 1 sibling payment tx, 2 inner tx
+var effects map[TxTypeID][]TxEffect = map[TxTypeID][]TxEffect{


nit:

Suggested change

var effects map[TxTypeID][]TxEffect = map[TxTypeID][]TxEffect{

var effects = map[TxTypeID][]TxEffect{

shiqizng · 2023-06-22T19:48:03Z

tools/block-generator/generator/generate.go

@@ -93,8 +120,19 @@ func MakeGenerator(dbround uint64, bkGenesis bookkeeping.Genesis, config Generat
 		gen.genesisHash = bkGenesis.Hash()
 	}

-	gen.apps = make(map[appKind][]*appData)
-	gen.pendingApps = make(map[appKind][]*appData)
+	gen.resetPendingApps()


I don't think resetPendingApps is needed here. could you also remind me why we need both appSlice and appMap?

appSlice is used to pick a random app. appMap is used for $O(1)$ lookup a specific up to tell if an app exists and get its information. Otherwise it would be $O(N)$ where $N$ is the number of created apps.

resetPendingApps() also sets gen.PendingApps[of each kind] and gen.PendingAppMap[of each kind] as we're appending to nil maps of other types panics (as opposed to appending to a nil slice)

@tzaffi could we move some methods to helpers file or a file that's just for app scenario? this file is getting large and it's harder to review.

The future follow up PR will introduce the following files to break up the code a bit:

generate_apps.go

generate_ledger.go

I've also introduced TODO's for this PR's description

tools/block-generator/generator/generate.go

shiqizng · 2023-06-22T20:24:04Z

tools/block-generator/generator/generate.go

@@ -763,110 +876,242 @@ func (g *generator) generateAssetTxnInternalHint(txType TxTypeID, round uint64,
 		fmt.Printf("\n\nthe sender account does not have enough algos for the transfer. idx %d, asset transaction type %v, num %d\n\n", senderIndex, actual, g.reportData[actual].GenerationCount)
 		os.Exit(1)
 	}
+
+	if assetID == 0 {


when do you expect this to happen?

This only happens if the assetID didn't get assigned correctly. That is, if I introduced a bug.

I'll add "this should never happen"

Co-authored-by: shiqizng <80276844+shiqizng@users.noreply.github.com>

…it should do

winder · 2023-06-23T13:29:04Z

tools/block-generator/generator/make_transactions.go

+	// TODO: should check for min balance}
+	g.balances[senderIndex] -= (pstFee + pstAmt)
+
+	return txntest.Group(&createTxn, &paySibTxn)


In the description you mention that this transaction has two inner transactions. Are you referring to the transaction group containing two transactions?

Looking at blocks generated by the daemon I see pairs of appl/pay transactions but they do not create any inner transactions.

The future followup PR exposes the inner transactions and even prints out a count in verbose mode.

shiqizng · 2023-06-22T20:38:33Z

tools/block-generator/scenarios/config.appboxes.small.yml

+# transaction distribution
+# tx_pay_fraction: 0.01
+# tx_asset_fraction: 0.01
+# tx_app_fraction: 0.98


can these lines be removed?

shiqizng · 2023-06-22T20:47:54Z

tools/block-generator/generator/generate.go

 			if err != nil {
-				panic(fmt.Sprintf("failed to generate transaction: %v\n", err))
+				// return err


Suggested change

// return err

incorporated in future PR. Commit: f631aec

shiqizng · 2023-06-22T20:53:08Z

tools/block-generator/generator/generate.go

-				panic(fmt.Sprintf("failed to encode transaction: %v\n", err))
+			if len(signedTxns) == 0 {
+				return fmt.Errorf("failed to generate transaction: no transactions given")
+			}


this line can be removed? the next condition, if len(signedTxns) != len(ads), should be sufficient?

In the future PR, this line is being kept, but the other condition if len(signedTxns) != len(ads) is going away because there are no ads any more.

shiqizng · 2023-06-23T18:50:34Z

tools/block-generator/generator/generate.go

@@ -93,8 +120,19 @@ func MakeGenerator(dbround uint64, bkGenesis bookkeeping.Genesis, config Generat
 		gen.genesisHash = bkGenesis.Hash()
 	}

-	gen.apps = make(map[appKind][]*appData)
-	gen.pendingApps = make(map[appKind][]*appData)
+	gen.resetPendingApps()


@tzaffi could we move some methods to helpers file or a file that's just for app scenario? this file is getting large and it's harder to review.

shiqizng · 2023-06-26T19:56:02Z

I'm ok with having a follow up PR to address the TODOs. I believe this PR works as intended. I ran the appboxes scenario locally and saw the rows in app_box table.

tzaffi changed the title ~~tools: block gerarter apps. Part 2: boxes~~ tools: block generator apps. Part 2: boxes Jun 17, 2023

tzaffi commented Jun 17, 2023

View reviewed changes

tools/block-generator/Makefile Show resolved Hide resolved

tzaffi commented Jun 17, 2023

View reviewed changes

tools/block-generator/README.md Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

tzaffi self-assigned this Jun 18, 2023

tzaffi mentioned this pull request Jun 18, 2023

tools: block-generator applications. Part 1: create #5450

Merged

2 tasks

tzaffi commented Jun 18, 2023

View reviewed changes

tools/block-generator/generator/generator_types.go Show resolved Hide resolved

tzaffi marked this pull request as ready for review June 20, 2023 16:09

tzaffi added the Enhancement label Jun 20, 2023

tzaffi requested a review from a team June 20, 2023 16:10

squashing

fbdfafb

tzaffi force-pushed the tools-blockgen-boxes branch from 89f751b to fbdfafb Compare June 21, 2023 15:45

shiqizng reviewed Jun 21, 2023

View reviewed changes

tzaffi and others added 2 commits June 21, 2023 10:56

Update tools/block-generator/generator/generate.go

8d9d6cc

Co-authored-by: shiqizng <80276844+shiqizng@users.noreply.github.com>

per CR suggestions - remove init + more detailed commentary for effec…

6dcf417

…ts map

shiqizng requested review from winder and a team June 22, 2023 19:14

winder reviewed Jun 22, 2023

View reviewed changes

tools/block-generator/generator/generate.go Show resolved Hide resolved

winder reviewed Jun 22, 2023

View reviewed changes

winder previously approved these changes Jun 22, 2023

View reviewed changes

shiqizng reviewed Jun 22, 2023

View reviewed changes

Update tools/block-generator/generator/generate.go

aee29e3

Co-authored-by: shiqizng <80276844+shiqizng@users.noreply.github.com>

tzaffi dismissed winder’s stale review via aee29e3 June 23, 2023 00:11

Zeph Grunschlag added 3 commits June 22, 2023 19:16

per CR: don't need redefinition of type in var + better comment

ae6d910

Merge remote-tracking branch 'algorand/master' into tools-blockgen-boxes

480a00d

per CR suggestion: let the default Makefile target do what you think …

53c4960

…it should do

cleaner clean targets

676db87

winder reviewed Jun 23, 2023

View reviewed changes

shiqizng reviewed Jun 23, 2023

View reviewed changes

tzaffi requested review from winder and shiqizng June 26, 2023 14:59

tzaffi mentioned this pull request Jun 26, 2023

ApplyDataTracer tzaffi/go-algorand#24

Closed

shiqizng approved these changes Jun 26, 2023

View reviewed changes

winder approved these changes Jun 27, 2023

View reviewed changes

winder merged commit 74b5f19 into algorand:master Jun 27, 2023
21 checks passed

tzaffi mentioned this pull request Jun 28, 2023

tools: block generator inner transactions #5506

Merged

19 tasks

Algo-devops-service mentioned this pull request Jul 11, 2023

go-algorand 3.17.0-beta Release PR #5541

Merged

onetechnical mentioned this pull request Jul 24, 2023

go-algorand 3.17.0-beta Release PR #5601

Merged

Algo-devops-service mentioned this pull request Aug 3, 2023

go-algorand 3.17.0-stable Release PR #5633

Merged

	var effects map[TxTypeID][]TxEffect = map[TxTypeID][]TxEffect{
	var effects = map[TxTypeID][]TxEffect{

tools: block generator apps. Part 2: boxes #5478

tools: block generator apps. Part 2: boxes #5478

Conversation

tzaffi commented Jun 17, 2023 • edited

Summary

Changes

TODO's ...

...which were inherited from PR Part 1

...which have been created in this PR

Test Plan

This comment was marked as resolved.

This comment was marked as outdated.

codecov bot commented Jun 21, 2023 • edited

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tzaffi Jun 22, 2023 • edited

Choose a reason for hiding this comment

tzaffi Jun 26, 2023 • edited

Choose a reason for hiding this comment

tzaffi Jun 26, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

winder left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tzaffi Jun 23, 2023 • edited

Choose a reason for hiding this comment

tzaffi Jun 23, 2023 • edited

Choose a reason for hiding this comment

shiqizng Jun 23, 2023 • edited

Choose a reason for hiding this comment

tzaffi Jun 26, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shiqizng Jun 23, 2023 • edited

Choose a reason for hiding this comment

shiqizng commented Jun 26, 2023

tzaffi commented Jun 17, 2023 •

edited

codecov bot commented Jun 21, 2023 •

edited

tzaffi Jun 22, 2023 •

edited

tzaffi Jun 26, 2023 •

edited

tzaffi Jun 26, 2023 •

edited

tzaffi Jun 23, 2023 •

edited

tzaffi Jun 23, 2023 •

edited

shiqizng Jun 23, 2023 •

edited

tzaffi Jun 26, 2023 •

edited

shiqizng Jun 23, 2023 •

edited