Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simulate: Allow unnamed foreign resource access #5366

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
111 commits
Select commit Hold shift + click to select a range
272851e
Expose StateDeltas to EvalTracer
jasonpaulos Mar 28, 2023
dcbe0e9
doc
jasonpaulos Mar 28, 2023
2080a1b
Rename + doc
jasonpaulos Mar 28, 2023
bd327a4
Pass StateDelta directly, and add option to gather StateDeltas for in…
jasonpaulos Mar 28, 2023
04a8819
Commit to parent in granular eval even during errors, for idential be…
jasonpaulos Mar 28, 2023
de02048
Update docstring
jasonpaulos Mar 29, 2023
ba5368d
Merge branch 'master' into eval-tracer-state-deltas
jasonpaulos Mar 29, 2023
bfcbe93
Make updates fn private
jasonpaulos Mar 29, 2023
5af6522
Merge branch 'master' into eval-tracer-state-deltas
jasonpaulos Mar 31, 2023
c07ccd2
Reuse child cows in BlockEvaluator.TransactionGroup & remove Updates …
jasonpaulos Mar 31, 2023
2801129
WIP
jasonpaulos Apr 14, 2023
f7b7777
Merge branch 'master' into eval-tracer-state-deltas
jasonpaulos Apr 19, 2023
d7fb785
Test AfterTxnGroup deltas
jasonpaulos Apr 21, 2023
a492003
Start to test AfterTxn deltas (granular eval)
jasonpaulos Apr 21, 2023
ed8ff74
TestTransactionGroupWithTracer working
jasonpaulos Apr 21, 2023
32f43b5
Testing from simulation package
jasonpaulos Apr 24, 2023
36c5427
Merge branch 'master' into eval-tracer-state-deltas
jasonpaulos Apr 24, 2023
f57040b
Fix race and simplify test table
jasonpaulos Apr 24, 2023
106aed5
Document statedelta methods
jasonpaulos Apr 24, 2023
9cb70e1
Merge branch 'master' into eval-tracer-state-deltas
jasonpaulos May 3, 2023
7ca0a1d
Remove StateDelta from simulation.Result
jasonpaulos May 3, 2023
1652a5c
Use StateDelta.Hydrate()
jasonpaulos May 3, 2023
b2f9af1
Ledger unit tests
jasonpaulos May 4, 2023
67cb81a
Dehydrate deltas before comparison
jasonpaulos May 4, 2023
955dcea
Merge branch 'master' into eval-tracer-state-deltas
jasonpaulos May 4, 2023
08eb749
Unit tests for hydrate/dehydrate
jasonpaulos May 4, 2023
ca3f65e
Allocate new cow for granular eval in Perform
jasonpaulos May 4, 2023
0616125
Initial implementation
jasonpaulos May 8, 2023
8371075
Merge branch 'master' into eval-tracer-state-deltas
jasonpaulos May 8, 2023
f87545e
Undo individual txn deltas
jasonpaulos May 8, 2023
d7cc4b9
Use Environment struct in simulation eval tests
jasonpaulos May 8, 2023
0e3fed9
Merge branch 'eval-tracer-state-deltas-tests' into simulate-unlimited…
jasonpaulos May 9, 2023
3db0807
WIP
jasonpaulos May 9, 2023
85cffdc
Merge branch 'master' into eval-tracer-state-deltas-tests
jasonpaulos May 9, 2023
8cc7132
Merge branch 'eval-tracer-state-deltas-tests' into simulate-unlimited…
jasonpaulos May 9, 2023
f544699
Fix test expected block header
jasonpaulos May 9, 2023
7044abb
Merge branch 'eval-tracer-state-deltas-tests' into simulate-unlimited…
jasonpaulos May 9, 2023
d30f5d7
Correctly implement transactions before simulation
jasonpaulos May 9, 2023
7cf3960
Fixes and unit test for unlimited resource read & write
jasonpaulos May 10, 2023
effefce
I forgot about boxes
jasonpaulos May 10, 2023
22cb32d
Add goal flag & e2e test
jasonpaulos May 10, 2023
93a43ac
Goal e2e test
jasonpaulos May 11, 2023
6865296
Config option gates feature
jasonpaulos May 11, 2023
2c35df5
Enable config option for e2e tests
jasonpaulos May 11, 2023
1dfb552
Merge branch 'master' into simulate-unlimited-indirection
jasonpaulos May 15, 2023
51ab296
Increase overspend amount to reduce test flakiness
jasonpaulos May 15, 2023
0ccde29
Logic unit test for unlimited resource access
jasonpaulos May 15, 2023
cea90e5
Remove unneeded param
jasonpaulos May 15, 2023
d78433c
Remove unneeded code
jasonpaulos May 16, 2023
324c8ec
Merge branch 'master' into simulate-unlimited-indirection
jasonpaulos Jun 1, 2023
7c6e874
Incorporate feedback and remove unneeded code
jasonpaulos Jun 1, 2023
2b54df9
Rename unlimited to unnamed
jasonpaulos Jun 1, 2023
ec5eb21
Reorder params and remove e2e network modification
jasonpaulos Jun 2, 2023
c94b6a0
New implementation of unnamed resource access with limits
jasonpaulos Jun 13, 2023
02b6a76
Merge branch 'master' into simulate-unlimited-indirection
jasonpaulos Jun 14, 2023
4d254d4
Fix tests
jasonpaulos Jun 14, 2023
295fd16
Fix build error
jasonpaulos Jun 14, 2023
d379430
Docstrings for reviewdog
jasonpaulos Jun 14, 2023
536b24c
spelling
jasonpaulos Jun 14, 2023
6b43eca
Minor rename and add additional comments
jasonpaulos Jun 14, 2023
3f60c40
Add REST API response
jasonpaulos Jun 14, 2023
7ee71c1
Move txn local resources into txn response
jasonpaulos Jun 15, 2023
03d5f6f
reviewdog strikes again
jasonpaulos Jun 15, 2023
e4ac73f
Unnamed resource limits account for potential additional app calls
jasonpaulos Jun 15, 2023
fe0b8eb
Merge branch 'master' into simulate-unlimited-indirection
jasonpaulos Jun 15, 2023
031c7c8
incorporate PR feedback
jasonpaulos Jun 15, 2023
0861eec
Use logic.BoxRef
jasonpaulos Jun 15, 2023
a2e2c4b
logic.EvalParams.GetApplicationAddress is public
jasonpaulos Jun 16, 2023
7de6599
Merge branch 'master' into simulate-unlimited-indirection
jasonpaulos Jun 16, 2023
4428369
Merge branch 'master' into simulate-unlimited-indirection
jasonpaulos Jun 20, 2023
820d186
Report needed empty box refs to satisfy IO budget
jasonpaulos Jun 22, 2023
3e685b9
comments
jasonpaulos Jun 22, 2023
79c4032
Merge branch 'master' into simulate-unlimited-indirection
jasonpaulos Jun 22, 2023
3af24b3
report empty box refs to REST endpoint
jasonpaulos Jun 22, 2023
f6e57e3
Merge branch 'master' into simulate-unlimited-indirection
jasonpaulos Jun 22, 2023
4a05718
fixup tests
jasonpaulos Jun 22, 2023
f340fc6
generic helper func
jasonpaulos Jun 22, 2023
9717315
Merge branch 'master' into simulate-unlimited-indirection
jasonpaulos Jun 29, 2023
b79ef27
Merge fix
jasonpaulos Jun 29, 2023
50fa7ef
tone down AddSaturate and friends
jasonpaulos Jun 29, 2023
e4a265d
Add box IO limit tests & fix read/write extra refs bug
jasonpaulos Jul 5, 2023
7305043
Merge branch 'master' into simulate-unlimited-indirection
jasonpaulos Jul 5, 2023
92daa43
Individual resource limit tests (but not cross-products)
jasonpaulos Jul 6, 2023
85f449d
Expand resource limits testing down to v5, address other bugs
jasonpaulos Jul 6, 2023
aeeb255
Referencing local unnamed resources removes slots from global pool as…
jasonpaulos Jul 6, 2023
6ea4463
Initial implementation and testing of cross product limits
jasonpaulos Jul 11, 2023
e4f8c53
Merge branch 'master' into simulate-unlimited-indirection
jasonpaulos Jul 11, 2023
94d5d80
Merge branch 'master' into simulate-unlimited-indirection
jasonpaulos Jul 12, 2023
da32c03
Tests for exceeding cross-product limits
jasonpaulos Jul 12, 2023
3fa219d
Cross-product limit tests and increase account limits to account for …
jasonpaulos Jul 18, 2023
2011376
Test to ensure newly created resources don't incur cross-product refe…
jasonpaulos Jul 18, 2023
acbf54e
Merge branch 'master' into simulate-unlimited-indirection
jasonpaulos Jul 18, 2023
0db9d30
Merge branch 'master' into simulate-unlimited-indirection
jasonpaulos Jul 20, 2023
a51ac6c
Merge branch 'master' into simulate-unlimited-indirection
jasonpaulos Jul 24, 2023
f343aa4
First round of feedback
jasonpaulos Jul 24, 2023
79b3818
more feedback changes
jasonpaulos Jul 24, 2023
5a494eb
Merge branch 'master' into simulate-unlimited-indirection
jasonpaulos Jul 28, 2023
db28190
Remove resource limits from API response
jasonpaulos Jul 28, 2023
032e7d8
Use x-algorand-format: uint64 for relevent fields
jasonpaulos Jul 28, 2023
2b64938
Merge branch 'master' into simulate-unlimited-indirection
jasonpaulos Jul 31, 2023
9b112ee
Rename structs, consolidate into SimulateUnnamedResourcesAccessed
jasonpaulos Aug 1, 2023
61f310d
DivCeil & address emptyRefs overflow problem
jasonpaulos Aug 1, 2023
03c6d52
Merge branch 'master' into simulate-unlimited-indirection
jasonpaulos Aug 1, 2023
e423c7d
address more feedback
jasonpaulos Aug 3, 2023
81e1888
Merge branch 'master' into simulate-unlimited-indirection
jasonpaulos Aug 3, 2023
edc7925
UnnamedResourcesAccessed description change
jasonpaulos Aug 3, 2023
25d6cb1
Merge branch 'master' into simulate-unlimited-indirection
jasonpaulos Aug 3, 2023
55fd6f7
DivCeil positive numbers only
jasonpaulos Aug 4, 2023
128725e
Merge branch 'master' into simulate-unlimited-indirection
jasonpaulos Aug 4, 2023
a0468d5
Merge branch 'master' into simulate-unlimited-indirection
jasonpaulos Aug 8, 2023
6892ffc
Merge branch 'master' into simulate-unlimited-indirection
jasonpaulos Aug 9, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
28 changes: 16 additions & 12 deletions cmd/goal/clerk.go
Original file line number Diff line number Diff line change
Expand Up @@ -74,10 +74,11 @@
simulateAllowMoreOpcodeBudget bool
simulateExtraOpcodeBudget uint64

simulateFullTrace bool
simulateEnableRequestTrace bool
simulateStackChange bool
simulateScratchChange bool
simulateFullTrace bool
simulateEnableRequestTrace bool
simulateStackChange bool
simulateScratchChange bool
simulateAllowUnnamedResources bool
)

func init() {
Expand Down Expand Up @@ -171,6 +172,7 @@
simulateCmd.Flags().BoolVar(&simulateEnableRequestTrace, "trace", false, "Enable simulation time execution trace of app calls")
simulateCmd.Flags().BoolVar(&simulateStackChange, "stack", false, "Report stack change during simulation time")
simulateCmd.Flags().BoolVar(&simulateScratchChange, "scratch", false, "Report scratch change during simulation time")
simulateCmd.Flags().BoolVar(&simulateAllowUnnamedResources, "allow-unnamed-resources", false, "Allow access to unnamed resources during simulation")
}

var clerkCmd = &cobra.Command{
Expand Down Expand Up @@ -1281,10 +1283,11 @@
Txns: txgroup,
},
},
AllowEmptySignatures: simulateAllowEmptySignatures,
AllowMoreLogging: simulateAllowMoreLogging,
ExtraOpcodeBudget: simulateExtraOpcodeBudget,
ExecTraceConfig: traceCmdOptionToSimulateTraceConfigModel(),
AllowEmptySignatures: simulateAllowEmptySignatures,
AllowMoreLogging: simulateAllowMoreLogging,
AllowUnnamedResources: simulateAllowUnnamedResources,
ExtraOpcodeBudget: simulateExtraOpcodeBudget,
ExecTraceConfig: traceCmdOptionToSimulateTraceConfigModel(),

Check warning on line 1290 in cmd/goal/clerk.go

View check run for this annotation

Codecov / codecov/patch

cmd/goal/clerk.go#L1286-L1290

Added lines #L1286 - L1290 were not covered by tests
}
err := writeFile(requestOutFilename, protocol.EncodeJSON(simulateRequest), 0600)
if err != nil {
Expand All @@ -1305,10 +1308,11 @@
Txns: txgroup,
},
},
AllowEmptySignatures: simulateAllowEmptySignatures,
AllowMoreLogging: simulateAllowMoreLogging,
ExtraOpcodeBudget: simulateExtraOpcodeBudget,
ExecTraceConfig: traceCmdOptionToSimulateTraceConfigModel(),
AllowEmptySignatures: simulateAllowEmptySignatures,
AllowMoreLogging: simulateAllowMoreLogging,
AllowUnnamedResources: simulateAllowUnnamedResources,
ExtraOpcodeBudget: simulateExtraOpcodeBudget,
ExecTraceConfig: traceCmdOptionToSimulateTraceConfigModel(),

Check warning on line 1315 in cmd/goal/clerk.go

View check run for this annotation

Codecov / codecov/patch

cmd/goal/clerk.go#L1311-L1315

Added lines #L1311 - L1315 were not covered by tests
}
simulateResponse, responseErr = client.SimulateTransactions(simulateRequest)
} else {
Expand Down
130 changes: 129 additions & 1 deletion daemon/algod/api/algod.oas2.json
ahangsu marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
Expand Up @@ -3093,6 +3093,46 @@
}
}
},
"AssetHoldingReference": {
"description": "References an asset held by an account.",
"type": "object",
"required": [
"account",
"asset"
],
"properties": {
"account": {
"description": "Address of the account holding the asset.",
"type": "string",
"x-algorand-format": "Address"
},
"asset": {
"description": "Asset ID of the holding.",
"type": "integer",
"x-algorand-format": "uint64"
}
}
},
"ApplicationLocalReference": {
"description": "References an account's local state for an application.",
"type": "object",
"required": [
"account",
"app"
],
"properties": {
"account": {
"description": "Address of the account with the local state.",
"type": "string",
"x-algorand-format": "Address"
},
"app": {
"description": "Application ID of the local state application.",
"type": "integer",
"x-algorand-format": "uint64"
}
}
},
"ApplicationStateSchema": {
"description": "Specifies maximums on the number of each type that may be stored.",
"type": "object",
Expand Down Expand Up @@ -3590,13 +3630,17 @@
}
},
"allow-empty-signatures": {
"description": "Allow transactions without signatures to be simulated as if they had correct signatures.",
"description": "Allows transactions without signatures to be simulated as if they had correct signatures.",
"type": "boolean"
},
"allow-more-logging": {
"description": "Lifts limits on log opcode usage during simulation.",
"type": "boolean"
},
"allow-unnamed-resources": {
"description": "Allows access to unnamed resources during simulation.",
"type": "boolean"
},
"extra-opcode-budget": {
"description": "Applies extra opcode budget during simulation for each transaction group.",
"type": "integer"
Expand Down Expand Up @@ -3682,6 +3726,25 @@
}
}
},
"BoxReference": {
"description": "References a box of an application.",
"type": "object",
"required": [
"app",
"name"
],
"properties": {
"app": {
"description": "Application ID which this box belongs to",
"type": "integer"
},
"name": {
"description": "Base64 encoded box name",
"type": "string",
"format": "byte"
}
}
},
"KvDelta": {
"description": "A single Delta containing the key, the previous value and the current value for a single round.",
"type": "object",
Expand Down Expand Up @@ -3874,6 +3937,9 @@
"app-budget-consumed": {
"description": "Total budget consumed during execution of app calls in the transaction group.",
"type": "integer"
},
"unnamed-resources-accessed": {
"$ref": "#/definitions/SimulateUnnamedResourcesAccessed"
}
}
},
Expand All @@ -3897,6 +3963,9 @@
},
"exec-trace": {
"$ref": "#/definitions/SimulationTransactionExecTrace"
},
"unnamed-resources-accessed": {
"$ref": "#/definitions/SimulateUnnamedResourcesAccessed"
}
}
},
Expand Down Expand Up @@ -3988,6 +4057,10 @@
"description": "If true, transactions without signatures are allowed and simulated as if they were properly signed.",
"type": "boolean"
},
"allow-unnamed-resources": {
"description": "If true, allows access to unnamed resources during simulation.",
"type": "boolean"
},
"max-log-calls": {
"description": "The maximum log calls one can make during simulation",
"type": "integer"
Expand Down Expand Up @@ -4090,6 +4163,61 @@
}
}
}
},
"SimulateUnnamedResourcesAccessed": {
"description": "These are resources that were accessed by this group that would normally have caused failure, but were allowed in simulation. Depending on where this object is in the response, the unnamed resources it contains may or may not qualify for group resource sharing. If this is a field in SimulateTransactionGroupResult, the resources do qualify, but if this is a field in SimulateTransactionResult, they do not qualify. In order to make this group valid for actual submission, resources that qualify for group sharing can be made available by any transaction of the group; otherwise, resources must be placed in the same transaction which accessed them.",
"type": "object",
"properties": {
"accounts": {
"description": "The unnamed accounts that were referenced. The order of this array is arbitrary.",
"type": "array",
"items": {
"type": "string",
"x-algorand-format": "Address"
}
},
"assets": {
"description": "The unnamed assets that were referenced. The order of this array is arbitrary.",
"type": "array",
"items": {
"type": "integer",
"x-algorand-format": "uint64"
}
},
"apps": {
"description": "The unnamed applications that were referenced. The order of this array is arbitrary.",
"type": "array",
"items": {
"type": "integer",
"x-algorand-format": "uint64"
}
},
"boxes": {
jasonpaulos marked this conversation as resolved.
Show resolved Hide resolved
"description": "The unnamed boxes that were referenced. The order of this array is arbitrary.",
"type": "array",
"items": {
"$ref": "#/definitions/BoxReference"
}
},
"extra-box-refs": {
"description": "The number of extra box references used to increase the IO budget. This is in addition to the references defined in the input transaction group and any referenced to unnamed boxes.",
"type": "integer"
},
"asset-holdings": {
"description": "The unnamed asset holdings that were referenced. The order of this array is arbitrary.",
"type": "array",
"items": {
"$ref": "#/definitions/AssetHoldingReference"
}
},
"app-locals": {
"description": "The unnamed application local states that were referenced. The order of this array is arbitrary.",
"type": "array",
"items": {
"$ref": "#/definitions/ApplicationLocalReference"
}
}
}
}
},
"parameters": {
Expand Down