CI: use libboost-math-dev instead of libboost-all-dev #2717

cce · 2021-08-09T14:14:16Z

Summary

Small change: libboost-math-dev requires just 4 packages to install, while libboost-all-dev requires > 100. Only Debian/Ubuntu distributions provide fine-grained boost packages like this, but should shave a little time off the CI builds. (Our only boost include is boost/math/distributions/binomial.hpp.)

Test Plan

CI tests should pass as before.

tsachiherman · 2021-08-09T16:52:54Z

take a look on the travis failure. it looks like on arm64 it's now missing make. ( which I guess was a dependency )

codecov-commenter · 2021-08-11T14:35:59Z

Codecov Report

Merging #2717 (31b4dc6) into master (358a2e1) will increase coverage by 0.00%.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #2717   +/-   ##
=======================================
  Coverage   47.06%   47.07%           
=======================================
  Files         349      349           
  Lines       55858    55858           
=======================================
+ Hits        26289    26294    +5     
+ Misses      26617    26616    -1     
+ Partials     2952     2948    -4

Impacted Files	Coverage Δ
network/requestTracker.go	`70.25% <0.00%> (-1.30%)`	⬇️
cmd/tealdbg/debugger.go	`73.86% <0.00%> (+1.00%)`	⬆️
catchup/peerSelector.go	`100.00% <0.00%> (+1.04%)`	⬆️
util/metrics/counter.go	`80.95% <0.00%> (+2.38%)`	⬆️
util/metrics/gauge.go	`70.66% <0.00%> (+2.66%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 358a2e1...31b4dc6. Read the comment docs.

tsachiherman · 2021-08-11T16:26:26Z

@cce - you might want to ensure that you have the make installed in scripts/install_linux_deps.sh rather the fixing it in the travis file.

## Summary Small change: libboost-math-dev requires just 4 packages to install, while libboost-all-dev requires > 100. Only Debian/Ubuntu distributions provide fine-grained boost packages like this, but should shave a little time off the CI builds. (Our only boost include is boost/math/distributions/binomial.hpp.) ## Test Plan Builds should pass as before. Now that we are no longer using Travis for Linux builds, the side effect of libboost-all-dev installing make and other missing build tools on Travis encountered in #2717 is no longer a concern.

* Update the Version, BuildNumber, genesistimestamp.data * Three new globals for to help contract-to-contract usability * detritis * Check error * doc comments * Support transaction arguments for `goal app method` (#3233) * Implement transactions as arguments * Fix indexing and dryrun issue * Add docstring * Satisfy review dog * Fix pointer issue * Fix group command * Rename e2e test * Fix filename variable * Add e2e test * Use tab * CI: use libboost-math-dev instead of libboost-all-dev (#3223) ## Summary Small change: libboost-math-dev requires just 4 packages to install, while libboost-all-dev requires > 100. Only Debian/Ubuntu distributions provide fine-grained boost packages like this, but should shave a little time off the CI builds. (Our only boost include is boost/math/distributions/binomial.hpp.) ## Test Plan Builds should pass as before. Now that we are no longer using Travis for Linux builds, the side effect of libboost-all-dev installing make and other missing build tools on Travis encountered in #2717 is no longer a concern. * testing: fixes to rest-participation-key e2e test (#3238) ## Summary - Test to make sure RES has the right input before counting line numbers for result size. - Rest RES to empty so that the same output is not recycled in case of an error. - exit 1 in case of an error - Reduce LAST_ROUND from 1200000 to 120 - "Get List of Keys" before getting NUM_IDS_3 otherwise it will recycle old RES value. * testing: interactive mode for e2e testing (#3227) ## Summary Some e2e tests require a python environment for testing. Unfortunately, setting up that environment adequately similar to the testing environment may not be trivial. This change introduces an interactive mode to the e2e.sh script which stops at the point of running the tests, and allows the user to run the tests from the same testing environment. ## Test Plan No tests needed. Tested the script locally. * Make dev-mode tests less flaky. (#3252) ## Summary Fix a couple flaws in the new go-e2e tests built ontop of DevMode: * Shutdown the fixture when finished. * Don't run in parallel. * Longer delays / better algorithms to wait for data flushing to complete. * Check for "out of order" keys. ## Test Plan N/A, this is a test. * adding libtool to ubuntu deps (#3251) ## Summary The sandbox is not building with dev config using master branch algorand/sandbox#85, complains about libtool not being installed Guessing from this change #3223 Adding libtool to UBUNTU_DEPS in install scripts ## Test Plan Set config in sandbox to my branch `sandbox up dev` It built * Fix error shadowing in Eval (#3258) ## Summary Error from account preloading was shadowed by returning a wrong err variable. This caused subsequent problems in account updates and masked the original failure. ## Test Plan Use existing tests * Disable flaky test. (#3256) ## Summary This test doesn't work properly, disable it until #3255 addresses any underlying problems. * Update the Version, BuildNumber, genesistimestamp.data * Fix a data race in app tests (#3269) ## Summary A test helper function `commitRound` accessed `l.trackers.lastFlushTime` without taking a lock. Fixed. ## Test Plan ``` go test ./ledger -run TestAppEmpty -race -count=50 ok github.com/algorand/go-algorand/ledger 4.078s ``` * Fix e2e.sh mixed indent characters. (#3266) ## Summary Fix e2e.sh mixed indent characters. * Fix ParticipationKeyResponse type. (#3264) ## Summary Fix a small type discrepancy in the OpenAPI spec ahead of some other work that's about to happen. * disable parallelism for e2e-go tests (#3242) ## Summary This sets `-p 1` for the e2e-go tests, intended to make them more deterministic when running on a VM with relatively constrained resources. Since each e2e-go test might spin up a few nodes, it seems like it would help to avoid resource contention. ## Test Plan Tests should run as before. Desired effect can be verified by looking at the test output where the value of PARALLEL_FLAG is printed out before tests are run. * Updating Readme.md with circleci status badges (#3245) * Fix formatting for CircleCI badges (#3272) * Add Custom Scenario for Performance Testing (#3278) Add Custom Scenario for Performance Testing. Add README on how to run custom scenario and modify create_and_deploy_recipe.sh to accept a network template that will generate a new recipe. * Impose limits on the entire "tree" of inner calls. This also increases the realism of testing of multiple app calls in a group by creating the EvalParams with the real constructor, thus getting the pooling stuff tested here without playing games manipulating the ep after construction. * ParticipationRegistry - StateProof loading methods (#3261) ## Summary Add ParticipationRegistry methods for setting and retrieving state proof keys. Since they aren't in master yet there is a `type StateProofKey []byte` stub which will need to be updated later. ## Test Plan New unit tests. * Op base64 decode (#3220) b64 opcode, tests, and specs * Bump Version, Remove buildnumber.dat and genesistimestamp.dat files. * Change golang version to 1.16.11 in go-algorand (#2825) Upgrading to 1.16 to help alleviate issues with working on different go versions, and update to a supported, more secure version. Release notes for Go 1.15 and 1.16: https://tip.golang.org/doc/go1.16 https://tip.golang.org/doc/go1.15 * Compatibility mode for partkeyinfo. (#3291) ## Summary Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running Workarounds: 1) use an older `goal` binary. 2) use `algokey part info --keyfile <file>` ## Test Plan Tested manually: ``` ~$ goal account partkeyinfo -d /tmp/private_network/Node/ Dumping participation key info from /tmp/private_network/Node/... Participation ID: CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA Parent address: DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU Last vote round: 1 Last block proposal round: 2 Effective first round: 1 Effective last round: 3000000 First round: 0 Last round: 3000000 Key dilution: 10000 Selection key: 5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ= Voting key: PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs= ~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c Dumping participation key info from /tmp/private_network/Node/... ------------------------------------------------------------------ File: Wallet2.0.3000000.partkey { "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU", "last": 3000000, "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=", "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=", "voteKD": 10000 } ``` * TestEcdsa: fix flaky "tampering" of public key (#3282) ## Summary This test (TestEcdsa) tests the ecdsa_pk_decompress opcode and intentionally "tampers" with the public key by setting the first byte to zero. Occasionally this test is failing, likely because the first byte was already zero. (The test failures are for the cases where failure is expected, `pass=false`) ## Test Plan Existing test should pass, occasional flakiness should go away. * Move appID tracking into EvalContext, out of LedgerForLogic This change increases the seperation between AVM execution and the ledger being used to lookup resources. Previously, the ledger kept track of the appID being executed, to offer a narrower interface to those resources. But now, with app-to-app calls, the appID being executed must change, and the AVM needs to maintain the current appID. * Stupid linter * Support reference types in `goal app method` (#3275) * Fix method signature parse bug * Support reference types * Review dog fixes * Fix comments * Add a hash prefix for ARCs-related hashes (#3298) ## Summary This is to allow ARCs (github.com/algorandfoundation/ARCs) to have their own hash prefix without risk of collision. ## Test Plan It is purely informational. There is no real code change. * catchup: suspend the catchup session once the agreement service kicks in (#3299) The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on. The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it. In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete. When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup. A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch. However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete. This change fixes the return value from true to false. * Compatibility mode for partkeyinfo. (#3291) ## Summary Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running Workarounds: 1) use an older `goal` binary. 2) use `algokey part info --keyfile <file>` ## Test Plan Tested manually: ``` ~$ goal account partkeyinfo -d /tmp/private_network/Node/ Dumping participation key info from /tmp/private_network/Node/... Participation ID: CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA Parent address: DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU Last vote round: 1 Last block proposal round: 2 Effective first round: 1 Effective last round: 3000000 First round: 0 Last round: 3000000 Key dilution: 10000 Selection key: 5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ= Voting key: PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs= ~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c Dumping participation key info from /tmp/private_network/Node/... ------------------------------------------------------------------ File: Wallet2.0.3000000.partkey { "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU", "last": 3000000, "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=", "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=", "voteKD": 10000 } ``` * catchup: suspend the catchup session once the agreement service kicks in (#3299) The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on. The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it. In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete. When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup. A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch. However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete. This change fixes the return value from true to false. * Bump buildnumber.dat * testing: disable flaky test (#3268) Disable a flaky test, to be re-enabled later with #3267. * enumerate conditions that might cause this fetchAndWrite to return false (#3301) ## Summary The fetchAndWrite function contains some complex logic to ultimately determine if we should continue trying to catch up. The conditions that might cause it to return false should be more explicitly enumerated. ## Test Plan Just comments * Fix unit tests error messages * make sure the block service is not attempting to access the ledger after being stopped. (#3303) ## Summary The block service was attempting to serve block via the http handler even after it has been stopped. This lead to undesired downstream failures in the ledger, which was shutdown as well. ## Test Plan unit test added. * Avoid creating algod process for the sole purpose of retrieving the genesis-id. (#3308) ## Summary Avoid creating algod process for the sole purpose of retrieving the genesis-id. Existing code was calling `algod -G -d <data dir>` in order to obtain the genesis version string. The genesis version string can be easily retrieved by loading the genesis file. ## Test Plan Use existing e2e tests. * documentation: fix algorand specs link (#3309) ## Summary This PR fixes a link in a README. ## Testing I clicked on the new link. * testing: reword partitiontest lint message. (#3297) ## Summary The wording on this was tripping me, maybe I was having an off day. I think it would be slightly easier if the message were to tell exactly what you need to do (and not use the angle brackets). * testing: fix random data race in TestAppAccountDataStorage (#3315) fix random data race in unit test * Allow access to resources created in the same transaction group The method will be reworked, but the tests are correct and want to get them visible to team. Co-authored-by: DevOps Service <devops-service@algorand.com> Co-authored-by: Jason Paulos <jasonpaulos@users.noreply.github.com> Co-authored-by: chris erway <51567+cce@users.noreply.github.com> Co-authored-by: Shant Karakashian <55754073+algonautshant@users.noreply.github.com> Co-authored-by: John Lee <64482439+algojohnlee@users.noreply.github.com> Co-authored-by: Will Winder <wwinder.unh@gmail.com> Co-authored-by: Ben Guidarelli <ben.guidarelli@gmail.com> Co-authored-by: Pavel Zbitskiy <65323360+algorandskiy@users.noreply.github.com> Co-authored-by: Jack <87339414+algojack@users.noreply.github.com> Co-authored-by: John Lee <john.lee@algorand.com> Co-authored-by: algobarb <78746954+algobarb@users.noreply.github.com> Co-authored-by: Zeph Grunschlag <tzaffi@users.noreply.github.com> Co-authored-by: Fabrice Benhamouda <fabrice.benhamouda@normalesup.org> Co-authored-by: Tsachi Herman <tsachi.herman@algorand.com> Co-authored-by: Tolik Zinovyev <tolik@algorand.com> Co-authored-by: egieseke <eric_gieseke@yahoo.com>

* Update the Version, BuildNumber, genesistimestamp.data * Three new globals for to help contract-to-contract usability * detritis * Check error * doc comments * Support transaction arguments for `goal app method` (#3233) * Implement transactions as arguments * Fix indexing and dryrun issue * Add docstring * Satisfy review dog * Fix pointer issue * Fix group command * Rename e2e test * Fix filename variable * Add e2e test * Use tab * CI: use libboost-math-dev instead of libboost-all-dev (#3223) ## Summary Small change: libboost-math-dev requires just 4 packages to install, while libboost-all-dev requires > 100. Only Debian/Ubuntu distributions provide fine-grained boost packages like this, but should shave a little time off the CI builds. (Our only boost include is boost/math/distributions/binomial.hpp.) ## Test Plan Builds should pass as before. Now that we are no longer using Travis for Linux builds, the side effect of libboost-all-dev installing make and other missing build tools on Travis encountered in #2717 is no longer a concern. * testing: fixes to rest-participation-key e2e test (#3238) ## Summary - Test to make sure RES has the right input before counting line numbers for result size. - Rest RES to empty so that the same output is not recycled in case of an error. - exit 1 in case of an error - Reduce LAST_ROUND from 1200000 to 120 - "Get List of Keys" before getting NUM_IDS_3 otherwise it will recycle old RES value. * testing: interactive mode for e2e testing (#3227) ## Summary Some e2e tests require a python environment for testing. Unfortunately, setting up that environment adequately similar to the testing environment may not be trivial. This change introduces an interactive mode to the e2e.sh script which stops at the point of running the tests, and allows the user to run the tests from the same testing environment. ## Test Plan No tests needed. Tested the script locally. * Make dev-mode tests less flaky. (#3252) ## Summary Fix a couple flaws in the new go-e2e tests built ontop of DevMode: * Shutdown the fixture when finished. * Don't run in parallel. * Longer delays / better algorithms to wait for data flushing to complete. * Check for "out of order" keys. ## Test Plan N/A, this is a test. * adding libtool to ubuntu deps (#3251) ## Summary The sandbox is not building with dev config using master branch algorand/sandbox#85, complains about libtool not being installed Guessing from this change #3223 Adding libtool to UBUNTU_DEPS in install scripts ## Test Plan Set config in sandbox to my branch `sandbox up dev` It built * Fix error shadowing in Eval (#3258) ## Summary Error from account preloading was shadowed by returning a wrong err variable. This caused subsequent problems in account updates and masked the original failure. ## Test Plan Use existing tests * Disable flaky test. (#3256) ## Summary This test doesn't work properly, disable it until #3255 addresses any underlying problems. * Update the Version, BuildNumber, genesistimestamp.data * Fix a data race in app tests (#3269) ## Summary A test helper function `commitRound` accessed `l.trackers.lastFlushTime` without taking a lock. Fixed. ## Test Plan ``` go test ./ledger -run TestAppEmpty -race -count=50 ok github.com/algorand/go-algorand/ledger 4.078s ``` * Fix e2e.sh mixed indent characters. (#3266) ## Summary Fix e2e.sh mixed indent characters. * Fix ParticipationKeyResponse type. (#3264) ## Summary Fix a small type discrepancy in the OpenAPI spec ahead of some other work that's about to happen. * disable parallelism for e2e-go tests (#3242) ## Summary This sets `-p 1` for the e2e-go tests, intended to make them more deterministic when running on a VM with relatively constrained resources. Since each e2e-go test might spin up a few nodes, it seems like it would help to avoid resource contention. ## Test Plan Tests should run as before. Desired effect can be verified by looking at the test output where the value of PARALLEL_FLAG is printed out before tests are run. * Updating Readme.md with circleci status badges (#3245) * Fix formatting for CircleCI badges (#3272) * Add Custom Scenario for Performance Testing (#3278) Add Custom Scenario for Performance Testing. Add README on how to run custom scenario and modify create_and_deploy_recipe.sh to accept a network template that will generate a new recipe. * Impose limits on the entire "tree" of inner calls. This also increases the realism of testing of multiple app calls in a group by creating the EvalParams with the real constructor, thus getting the pooling stuff tested here without playing games manipulating the ep after construction. * ParticipationRegistry - StateProof loading methods (#3261) ## Summary Add ParticipationRegistry methods for setting and retrieving state proof keys. Since they aren't in master yet there is a `type StateProofKey []byte` stub which will need to be updated later. ## Test Plan New unit tests. * Op base64 decode (#3220) b64 opcode, tests, and specs * Bump Version, Remove buildnumber.dat and genesistimestamp.dat files. * Change golang version to 1.16.11 in go-algorand (#2825) Upgrading to 1.16 to help alleviate issues with working on different go versions, and update to a supported, more secure version. Release notes for Go 1.15 and 1.16: https://tip.golang.org/doc/go1.16 https://tip.golang.org/doc/go1.15 * Compatibility mode for partkeyinfo. (#3291) ## Summary Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running Workarounds: 1) use an older `goal` binary. 2) use `algokey part info --keyfile <file>` ## Test Plan Tested manually: ``` ~$ goal account partkeyinfo -d /tmp/private_network/Node/ Dumping participation key info from /tmp/private_network/Node/... Participation ID: CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA Parent address: DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU Last vote round: 1 Last block proposal round: 2 Effective first round: 1 Effective last round: 3000000 First round: 0 Last round: 3000000 Key dilution: 10000 Selection key: 5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ= Voting key: PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs= ~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c Dumping participation key info from /tmp/private_network/Node/... ------------------------------------------------------------------ File: Wallet2.0.3000000.partkey { "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU", "last": 3000000, "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=", "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=", "voteKD": 10000 } ``` * TestEcdsa: fix flaky "tampering" of public key (#3282) ## Summary This test (TestEcdsa) tests the ecdsa_pk_decompress opcode and intentionally "tampers" with the public key by setting the first byte to zero. Occasionally this test is failing, likely because the first byte was already zero. (The test failures are for the cases where failure is expected, `pass=false`) ## Test Plan Existing test should pass, occasional flakiness should go away. * Move appID tracking into EvalContext, out of LedgerForLogic This change increases the seperation between AVM execution and the ledger being used to lookup resources. Previously, the ledger kept track of the appID being executed, to offer a narrower interface to those resources. But now, with app-to-app calls, the appID being executed must change, and the AVM needs to maintain the current appID. * Stupid linter * Support reference types in `goal app method` (#3275) * Fix method signature parse bug * Support reference types * Review dog fixes * Fix comments * Add a hash prefix for ARCs-related hashes (#3298) ## Summary This is to allow ARCs (github.com/algorandfoundation/ARCs) to have their own hash prefix without risk of collision. ## Test Plan It is purely informational. There is no real code change. * catchup: suspend the catchup session once the agreement service kicks in (#3299) The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on. The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it. In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete. When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup. A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch. However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete. This change fixes the return value from true to false. * Compatibility mode for partkeyinfo. (#3291) ## Summary Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running Workarounds: 1) use an older `goal` binary. 2) use `algokey part info --keyfile <file>` ## Test Plan Tested manually: ``` ~$ goal account partkeyinfo -d /tmp/private_network/Node/ Dumping participation key info from /tmp/private_network/Node/... Participation ID: CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA Parent address: DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU Last vote round: 1 Last block proposal round: 2 Effective first round: 1 Effective last round: 3000000 First round: 0 Last round: 3000000 Key dilution: 10000 Selection key: 5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ= Voting key: PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs= ~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c Dumping participation key info from /tmp/private_network/Node/... ------------------------------------------------------------------ File: Wallet2.0.3000000.partkey { "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU", "last": 3000000, "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=", "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=", "voteKD": 10000 } ``` * catchup: suspend the catchup session once the agreement service kicks in (#3299) The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on. The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it. In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete. When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup. A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch. However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete. This change fixes the return value from true to false. * Bump buildnumber.dat * testing: disable flaky test (#3268) Disable a flaky test, to be re-enabled later with #3267. * enumerate conditions that might cause this fetchAndWrite to return false (#3301) ## Summary The fetchAndWrite function contains some complex logic to ultimately determine if we should continue trying to catch up. The conditions that might cause it to return false should be more explicitly enumerated. ## Test Plan Just comments * Fix unit tests error messages * make sure the block service is not attempting to access the ledger after being stopped. (#3303) ## Summary The block service was attempting to serve block via the http handler even after it has been stopped. This lead to undesired downstream failures in the ledger, which was shutdown as well. ## Test Plan unit test added. * Avoid creating algod process for the sole purpose of retrieving the genesis-id. (#3308) ## Summary Avoid creating algod process for the sole purpose of retrieving the genesis-id. Existing code was calling `algod -G -d <data dir>` in order to obtain the genesis version string. The genesis version string can be easily retrieved by loading the genesis file. ## Test Plan Use existing e2e tests. * documentation: fix algorand specs link (#3309) ## Summary This PR fixes a link in a README. ## Testing I clicked on the new link. * testing: reword partitiontest lint message. (#3297) ## Summary The wording on this was tripping me, maybe I was having an off day. I think it would be slightly easier if the message were to tell exactly what you need to do (and not use the angle brackets). * testing: fix random data race in TestAppAccountDataStorage (#3315) fix random data race in unit test * Allow access to resources created in the same transaction group The method will be reworked, but the tests are correct and want to get them visible to team. * ledger: perform the catchpoint writing outside the trackers lock. (#3311) ## Summary This PR moves the catchpoint file writing to be performed outside of the trackers lock. This resolves the issue where a long catchpoint file writing blocks the agreement from validating and propagating votes. ## Test Plan * [x] Test manually & use existing tests. * [x] Implement a unit test * [x] Deploy a local network where the catchpoint writing takes a long time and verify it doesn't get blocked during catchpoint writing. * Separate tx and key validity for `goal account renewpartkey` (#3286) Always use currentRound+proto.MaxTxnLife as last valid round for the transaction when renewing instead of using the partkey validity period. This fixes #3283 * Add qkniep to THANKS.md (#3320) ## Summary Add qkniep to THANKS.md * Followup to opcode base64_decode (#3288) * alphabet begone in favor of encoding * unit test various padding and whitespace scenarios * padding permutations also fail * "Slicing" --> "Manipulation" * fix the codegen fail? * Documenting padding, whitespace, other character behavior * Add help and fish mode to e2e interactive mode. (#3313) ## Summary Minor improvements to e2e.sh interactive mode: * add to -h output * do not run start stop test in interactive mode * support fish shell ## Test Plan Manual testing: ``` ~$ ./e2e.sh -i ... lots of output removed ... ********** READY ********** The test environment is now set. You can now run tests in another terminal. Configure the environment: set -g VIRTUAL_ENV "/home/will/go/src/github.com/algorand/go-algorand/tmp/out/e2e/130013-1639576513257/ve" set -g PATH "$VIRTUAL_ENV/bin:$PATH" python3 "/home/will/go/src/github.com/algorand/go-algorand/test/scripts"/e2e_client_runner.py "/home/will/go/src/github.com/algorand/go-algorand/test/scripts"/e2e_subs/SCRIPT_FILE_NAME Press enter to shut down the test environment... ``` * Minimum Account Balance in Algod (#3287) * Access to apps created in group Also adds some tests that are currently skipped for testing - access to addresses of newly created apps - use of gaid in inner transactions Both require some work to implement the thing being tested. * Remove tracked created mechanism in favor of examining applydata. * Add convertAddress tool. (#3304) ## Summary New tool: convertAddress I share this tool with someone every few months, putting it in the repo along with some documentation should make it easier to share and encourage people to share it amongst themselves if it's useful. Merge `debug` into `tools` to make it easier to organize these miscellaneous tools. * tealdbg: increase intermediate reading/writing buffers (#3335) ## Summary Some large teal source files cause the tealdbg/cdt session to choke. Upping the buffer size to allow for larger source files. closes #3100 ## Test Plan Run tealdbg with a large source teal file, ensure the source file makes it to cdt without choking. * Adding method pseudo op to readme (#3338) * Allow v6 AVM code to use in-group created asas, apps (& their accts) One exception - apps can not mutate (put or del) keys from the app accounts, because EvalDelta cannot encode such changes. * lint docs * typo * The review dog needs obedience training. * add config.DeadlockDetectionThreshold (#3339) Summary This allows for the deadlock detection threshold to be set by configuration. Test Plan Existing tests should pass. * Use one EvalParams for logic evals, another for apps in dry run We used to use one ep per transaction, shared between sig and and app. But the new model of ep usage is to keep using one while evaluating an entire group. The app ep is now built logic.NewAppEvalParams which, hopefully, will prevent some bugs when we change something in the EvalParams and don't reflect it in what was a "raw" EvalParams construction in debugger and dry run. * Use logic.NewAppEvalParams to decrease copying and bugs in debugger * Simplify use of NewEvalParams. No more nil return when no apps. This way, NewEvalParams can be used for all creations of EvalParams, whether they are intended for logicsig or app use, greatly simplifying the way we make them for use by dry run or debugger (where they serve double duty). * Remove explicit PastSideEffects handling in tealdbg * Fix flaky test in randomized ABI encoding test (#3346) * update abi encoding test random testcase generator, scale down parameters to avoid flaky test * parameterized test script * add notes to explain why flaky test is eliminated * show more information from self-roundtrip testing * fully utilize require, remove fmt * Always create EvalParams to evaluate a transaction group. We used to have an optimization to avoid creating EvalParams unless there was an app call in the transaction group. But the interface to allow transaction processing to communicate changes into the EvalParams is complicated by that (we must only do it if there is one!) This also allows us to use the same construction function for eps created for app and logic evaluation, simplifying dry-run and debugger. The optimization is less needed now anyway: 1) The ep is now shared for the whole group, so it's only one. 2) The ep is smaller now, as we only store nil pointers instead of larger scratch space objects for non-app calls. * Correct mistaken commit * Update `goal app method` handling of args and return values (#3352) * Fix method call arg overflow handling * Only check last log for return value * Address feedback * Add comment explaining ABI return prefix * Support app creation in `goal app method` (#3353) * Support app creation in `goal app method` * Don't use nonprintable tab character * Link to specific gist version * Fix error messages * Rename `methodCreation` to `methodCreatesApp` * Spec improvments * Update license to 2022 (#3360) Update license on all source files to 2022. * Add totals checks into acct updates tests (#3367) ## Summary After #2922 there is some leftover unused code for totals calculations. Turned this code into actual asserts. ## Test Plan This is tests update * More spec improvments, including resource "availability" * Recursively return inner transaction tree * Lint * No need for ConfirmedRound, so don't deref a nil pointer! * license check * Shut up, dawg. Co-authored-by: DevOps Service <devops-service@algorand.com> Co-authored-by: Jason Paulos <jasonpaulos@users.noreply.github.com> Co-authored-by: chris erway <51567+cce@users.noreply.github.com> Co-authored-by: Shant Karakashian <55754073+algonautshant@users.noreply.github.com> Co-authored-by: John Lee <64482439+algojohnlee@users.noreply.github.com> Co-authored-by: Will Winder <wwinder.unh@gmail.com> Co-authored-by: Ben Guidarelli <ben.guidarelli@gmail.com> Co-authored-by: Pavel Zbitskiy <65323360+algorandskiy@users.noreply.github.com> Co-authored-by: Jack <87339414+algojack@users.noreply.github.com> Co-authored-by: John Lee <john.lee@algorand.com> Co-authored-by: algobarb <78746954+algobarb@users.noreply.github.com> Co-authored-by: Zeph Grunschlag <tzaffi@users.noreply.github.com> Co-authored-by: Fabrice Benhamouda <fabrice.benhamouda@normalesup.org> Co-authored-by: Tsachi Herman <tsachi.herman@algorand.com> Co-authored-by: Tolik Zinovyev <tolik@algorand.com> Co-authored-by: egieseke <eric_gieseke@yahoo.com> Co-authored-by: Quentin Kniep <kniepque@hu-berlin.de> Co-authored-by: Hang Su <87964331+ahangsu@users.noreply.github.com> Co-authored-by: Or Aharonee <or.aharonee@algorand.com>

* Update the Version, BuildNumber, genesistimestamp.data * Three new globals for to help contract-to-contract usability * detritis * Check error * doc comments * Support transaction arguments for `goal app method` (#3233) * Implement transactions as arguments * Fix indexing and dryrun issue * Add docstring * Satisfy review dog * Fix pointer issue * Fix group command * Rename e2e test * Fix filename variable * Add e2e test * Use tab * CI: use libboost-math-dev instead of libboost-all-dev (#3223) ## Summary Small change: libboost-math-dev requires just 4 packages to install, while libboost-all-dev requires > 100. Only Debian/Ubuntu distributions provide fine-grained boost packages like this, but should shave a little time off the CI builds. (Our only boost include is boost/math/distributions/binomial.hpp.) ## Test Plan Builds should pass as before. Now that we are no longer using Travis for Linux builds, the side effect of libboost-all-dev installing make and other missing build tools on Travis encountered in #2717 is no longer a concern. * testing: fixes to rest-participation-key e2e test (#3238) ## Summary - Test to make sure RES has the right input before counting line numbers for result size. - Rest RES to empty so that the same output is not recycled in case of an error. - exit 1 in case of an error - Reduce LAST_ROUND from 1200000 to 120 - "Get List of Keys" before getting NUM_IDS_3 otherwise it will recycle old RES value. * testing: interactive mode for e2e testing (#3227) ## Summary Some e2e tests require a python environment for testing. Unfortunately, setting up that environment adequately similar to the testing environment may not be trivial. This change introduces an interactive mode to the e2e.sh script which stops at the point of running the tests, and allows the user to run the tests from the same testing environment. ## Test Plan No tests needed. Tested the script locally. * Make dev-mode tests less flaky. (#3252) ## Summary Fix a couple flaws in the new go-e2e tests built ontop of DevMode: * Shutdown the fixture when finished. * Don't run in parallel. * Longer delays / better algorithms to wait for data flushing to complete. * Check for "out of order" keys. ## Test Plan N/A, this is a test. * adding libtool to ubuntu deps (#3251) ## Summary The sandbox is not building with dev config using master branch algorand/sandbox#85, complains about libtool not being installed Guessing from this change #3223 Adding libtool to UBUNTU_DEPS in install scripts ## Test Plan Set config in sandbox to my branch `sandbox up dev` It built * Fix error shadowing in Eval (#3258) ## Summary Error from account preloading was shadowed by returning a wrong err variable. This caused subsequent problems in account updates and masked the original failure. ## Test Plan Use existing tests * Disable flaky test. (#3256) ## Summary This test doesn't work properly, disable it until #3255 addresses any underlying problems. * Update the Version, BuildNumber, genesistimestamp.data * Fix a data race in app tests (#3269) ## Summary A test helper function `commitRound` accessed `l.trackers.lastFlushTime` without taking a lock. Fixed. ## Test Plan ``` go test ./ledger -run TestAppEmpty -race -count=50 ok github.com/algorand/go-algorand/ledger 4.078s ``` * Fix e2e.sh mixed indent characters. (#3266) ## Summary Fix e2e.sh mixed indent characters. * Fix ParticipationKeyResponse type. (#3264) ## Summary Fix a small type discrepancy in the OpenAPI spec ahead of some other work that's about to happen. * disable parallelism for e2e-go tests (#3242) ## Summary This sets `-p 1` for the e2e-go tests, intended to make them more deterministic when running on a VM with relatively constrained resources. Since each e2e-go test might spin up a few nodes, it seems like it would help to avoid resource contention. ## Test Plan Tests should run as before. Desired effect can be verified by looking at the test output where the value of PARALLEL_FLAG is printed out before tests are run. * Updating Readme.md with circleci status badges (#3245) * Fix formatting for CircleCI badges (#3272) * Add Custom Scenario for Performance Testing (#3278) Add Custom Scenario for Performance Testing. Add README on how to run custom scenario and modify create_and_deploy_recipe.sh to accept a network template that will generate a new recipe. * Impose limits on the entire "tree" of inner calls. This also increases the realism of testing of multiple app calls in a group by creating the EvalParams with the real constructor, thus getting the pooling stuff tested here without playing games manipulating the ep after construction. * ParticipationRegistry - StateProof loading methods (#3261) ## Summary Add ParticipationRegistry methods for setting and retrieving state proof keys. Since they aren't in master yet there is a `type StateProofKey []byte` stub which will need to be updated later. ## Test Plan New unit tests. * Op base64 decode (#3220) b64 opcode, tests, and specs * Bump Version, Remove buildnumber.dat and genesistimestamp.dat files. * Change golang version to 1.16.11 in go-algorand (#2825) Upgrading to 1.16 to help alleviate issues with working on different go versions, and update to a supported, more secure version. Release notes for Go 1.15 and 1.16: https://tip.golang.org/doc/go1.16 https://tip.golang.org/doc/go1.15 * Compatibility mode for partkeyinfo. (#3291) ## Summary Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running Workarounds: 1) use an older `goal` binary. 2) use `algokey part info --keyfile <file>` ## Test Plan Tested manually: ``` ~$ goal account partkeyinfo -d /tmp/private_network/Node/ Dumping participation key info from /tmp/private_network/Node/... Participation ID: CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA Parent address: DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU Last vote round: 1 Last block proposal round: 2 Effective first round: 1 Effective last round: 3000000 First round: 0 Last round: 3000000 Key dilution: 10000 Selection key: 5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ= Voting key: PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs= ~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c Dumping participation key info from /tmp/private_network/Node/... ------------------------------------------------------------------ File: Wallet2.0.3000000.partkey { "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU", "last": 3000000, "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=", "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=", "voteKD": 10000 } ``` * TestEcdsa: fix flaky "tampering" of public key (#3282) ## Summary This test (TestEcdsa) tests the ecdsa_pk_decompress opcode and intentionally "tampers" with the public key by setting the first byte to zero. Occasionally this test is failing, likely because the first byte was already zero. (The test failures are for the cases where failure is expected, `pass=false`) ## Test Plan Existing test should pass, occasional flakiness should go away. * Move appID tracking into EvalContext, out of LedgerForLogic This change increases the seperation between AVM execution and the ledger being used to lookup resources. Previously, the ledger kept track of the appID being executed, to offer a narrower interface to those resources. But now, with app-to-app calls, the appID being executed must change, and the AVM needs to maintain the current appID. * Stupid linter * Support reference types in `goal app method` (#3275) * Fix method signature parse bug * Support reference types * Review dog fixes * Fix comments * Add a hash prefix for ARCs-related hashes (#3298) ## Summary This is to allow ARCs (github.com/algorandfoundation/ARCs) to have their own hash prefix without risk of collision. ## Test Plan It is purely informational. There is no real code change. * catchup: suspend the catchup session once the agreement service kicks in (#3299) The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on. The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it. In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete. When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup. A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch. However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete. This change fixes the return value from true to false. * Compatibility mode for partkeyinfo. (#3291) ## Summary Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running Workarounds: 1) use an older `goal` binary. 2) use `algokey part info --keyfile <file>` ## Test Plan Tested manually: ``` ~$ goal account partkeyinfo -d /tmp/private_network/Node/ Dumping participation key info from /tmp/private_network/Node/... Participation ID: CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA Parent address: DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU Last vote round: 1 Last block proposal round: 2 Effective first round: 1 Effective last round: 3000000 First round: 0 Last round: 3000000 Key dilution: 10000 Selection key: 5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ= Voting key: PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs= ~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c Dumping participation key info from /tmp/private_network/Node/... ------------------------------------------------------------------ File: Wallet2.0.3000000.partkey { "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU", "last": 3000000, "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=", "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=", "voteKD": 10000 } ``` * catchup: suspend the catchup session once the agreement service kicks in (#3299) The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on. The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it. In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete. When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup. A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch. However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete. This change fixes the return value from true to false. * Bump buildnumber.dat * testing: disable flaky test (#3268) Disable a flaky test, to be re-enabled later with #3267. * enumerate conditions that might cause this fetchAndWrite to return false (#3301) ## Summary The fetchAndWrite function contains some complex logic to ultimately determine if we should continue trying to catch up. The conditions that might cause it to return false should be more explicitly enumerated. ## Test Plan Just comments * Fix unit tests error messages * make sure the block service is not attempting to access the ledger after being stopped. (#3303) ## Summary The block service was attempting to serve block via the http handler even after it has been stopped. This lead to undesired downstream failures in the ledger, which was shutdown as well. ## Test Plan unit test added. * Avoid creating algod process for the sole purpose of retrieving the genesis-id. (#3308) ## Summary Avoid creating algod process for the sole purpose of retrieving the genesis-id. Existing code was calling `algod -G -d <data dir>` in order to obtain the genesis version string. The genesis version string can be easily retrieved by loading the genesis file. ## Test Plan Use existing e2e tests. * documentation: fix algorand specs link (#3309) ## Summary This PR fixes a link in a README. ## Testing I clicked on the new link. * testing: reword partitiontest lint message. (#3297) ## Summary The wording on this was tripping me, maybe I was having an off day. I think it would be slightly easier if the message were to tell exactly what you need to do (and not use the angle brackets). * testing: fix random data race in TestAppAccountDataStorage (#3315) fix random data race in unit test * Allow access to resources created in the same transaction group The method will be reworked, but the tests are correct and want to get them visible to team. * ledger: perform the catchpoint writing outside the trackers lock. (#3311) ## Summary This PR moves the catchpoint file writing to be performed outside of the trackers lock. This resolves the issue where a long catchpoint file writing blocks the agreement from validating and propagating votes. ## Test Plan * [x] Test manually & use existing tests. * [x] Implement a unit test * [x] Deploy a local network where the catchpoint writing takes a long time and verify it doesn't get blocked during catchpoint writing. * Separate tx and key validity for `goal account renewpartkey` (#3286) Always use currentRound+proto.MaxTxnLife as last valid round for the transaction when renewing instead of using the partkey validity period. This fixes #3283 * Add qkniep to THANKS.md (#3320) ## Summary Add qkniep to THANKS.md * Followup to opcode base64_decode (#3288) * alphabet begone in favor of encoding * unit test various padding and whitespace scenarios * padding permutations also fail * "Slicing" --> "Manipulation" * fix the codegen fail? * Documenting padding, whitespace, other character behavior * Add help and fish mode to e2e interactive mode. (#3313) ## Summary Minor improvements to e2e.sh interactive mode: * add to -h output * do not run start stop test in interactive mode * support fish shell ## Test Plan Manual testing: ``` ~$ ./e2e.sh -i ... lots of output removed ... ********** READY ********** The test environment is now set. You can now run tests in another terminal. Configure the environment: set -g VIRTUAL_ENV "/home/will/go/src/github.com/algorand/go-algorand/tmp/out/e2e/130013-1639576513257/ve" set -g PATH "$VIRTUAL_ENV/bin:$PATH" python3 "/home/will/go/src/github.com/algorand/go-algorand/test/scripts"/e2e_client_runner.py "/home/will/go/src/github.com/algorand/go-algorand/test/scripts"/e2e_subs/SCRIPT_FILE_NAME Press enter to shut down the test environment... ``` * Minimum Account Balance in Algod (#3287) * Access to apps created in group Also adds some tests that are currently skipped for testing - access to addresses of newly created apps - use of gaid in inner transactions Both require some work to implement the thing being tested. * Remove tracked created mechanism in favor of examining applydata. * Add convertAddress tool. (#3304) ## Summary New tool: convertAddress I share this tool with someone every few months, putting it in the repo along with some documentation should make it easier to share and encourage people to share it amongst themselves if it's useful. Merge `debug` into `tools` to make it easier to organize these miscellaneous tools. * tealdbg: increase intermediate reading/writing buffers (#3335) ## Summary Some large teal source files cause the tealdbg/cdt session to choke. Upping the buffer size to allow for larger source files. closes #3100 ## Test Plan Run tealdbg with a large source teal file, ensure the source file makes it to cdt without choking. * Adding method pseudo op to readme (#3338) * Allow v6 AVM code to use in-group created asas, apps (& their accts) One exception - apps can not mutate (put or del) keys from the app accounts, because EvalDelta cannot encode such changes. * lint docs * typo * The review dog needs obedience training. * add config.DeadlockDetectionThreshold (#3339) Summary This allows for the deadlock detection threshold to be set by configuration. Test Plan Existing tests should pass. * Use one EvalParams for logic evals, another for apps in dry run We used to use one ep per transaction, shared between sig and and app. But the new model of ep usage is to keep using one while evaluating an entire group. The app ep is now built logic.NewAppEvalParams which, hopefully, will prevent some bugs when we change something in the EvalParams and don't reflect it in what was a "raw" EvalParams construction in debugger and dry run. * Use logic.NewAppEvalParams to decrease copying and bugs in debugger * Simplify use of NewEvalParams. No more nil return when no apps. This way, NewEvalParams can be used for all creations of EvalParams, whether they are intended for logicsig or app use, greatly simplifying the way we make them for use by dry run or debugger (where they serve double duty). * Remove explicit PastSideEffects handling in tealdbg * Fix flaky test in randomized ABI encoding test (#3346) * update abi encoding test random testcase generator, scale down parameters to avoid flaky test * parameterized test script * add notes to explain why flaky test is eliminated * show more information from self-roundtrip testing * fully utilize require, remove fmt * Always create EvalParams to evaluate a transaction group. We used to have an optimization to avoid creating EvalParams unless there was an app call in the transaction group. But the interface to allow transaction processing to communicate changes into the EvalParams is complicated by that (we must only do it if there is one!) This also allows us to use the same construction function for eps created for app and logic evaluation, simplifying dry-run and debugger. The optimization is less needed now anyway: 1) The ep is now shared for the whole group, so it's only one. 2) The ep is smaller now, as we only store nil pointers instead of larger scratch space objects for non-app calls. * Correct mistaken commit * ledger: perform the catchpoint writing outside the trackers lock. (#3311) ## Summary This PR moves the catchpoint file writing to be performed outside of the trackers lock. This resolves the issue where a long catchpoint file writing blocks the agreement from validating and propagating votes. ## Test Plan * [x] Test manually & use existing tests. * [x] Implement a unit test * [x] Deploy a local network where the catchpoint writing takes a long time and verify it doesn't get blocked during catchpoint writing. * Bump version number * Update `goal app method` handling of args and return values (#3352) * Fix method call arg overflow handling * Only check last log for return value * Address feedback * Add comment explaining ABI return prefix * Support app creation in `goal app method` (#3353) * Support app creation in `goal app method` * Don't use nonprintable tab character * Link to specific gist version * Fix error messages * Rename `methodCreation` to `methodCreatesApp` * Spec improvments * Update license to 2022 (#3360) Update license on all source files to 2022. * Add totals checks into acct updates tests (#3367) ## Summary After #2922 there is some leftover unused code for totals calculations. Turned this code into actual asserts. ## Test Plan This is tests update * More spec improvments, including resource "availability" * Recursively return inner transaction tree * Lint * No need for ConfirmedRound, so don't deref a nil pointer! * remove buildnumber.dat * license check * Shut up, dawg. * PKI State Proof Incremental Key Loading (#3281) ## Summary Followup to #3261 (contained in diff). Use the new key loading routine from the REST API. ## Test Plan New unit tests. * Limit number of simultaneous REST connections (#3326) ## Summary This PR limits the number of simultaneous REST connections we process to prevent the exhaustion of resources and ultimately a crash. Two limits are introduced: soft and hard. When the soft limit is exceeded, new connections are returned the 429 Too Many Requests http code. When the hard limit is exceeded, new connections are accepted and immediately closed. Partially resolves algorand/go-algorand-internal#1814. ## Test Plan Added unit tests. * Use rejecting limit listener in WebsocketNetwork. (#3380) ## Summary Replace the standard limit listener with the new rejecting limit listener in `WebsocketNetwork`. This will let the dialing node know that connection is impossible faster. ## Test Plan Probably not necessary. * Delete unused constant. (#3379) ## Summary This PR deletes an unused constant. ## Test Plan None. * Add test to exercise lookup corner cases (#3376) ## Summary This test attempts to cover the case when an accountUpdates.lookupX method can't find the requested address, falls through looking at deltas and the LRU accounts cache, then hits the database — only to discover that the round stored in the database (committed in `accountUpdates.commitRound`) is out of sync with `accountUpdates.cachedDBRound` (updated a little bit later in `accountUpdates.postCommit`). In this case, the lookup method waits and tries again, iterating the `for { }` it is in. We did not have coverage for this code path before. ## Test Plan Adds new test. * Test for catchup stop on completion (#3306) Adding a test for the fix in #3299 ## Test Plan This is a test * Delete unused AtomicCommitWriteLock(). (#3383) ## Summary This PR deletes unused `AtomicCommitWriteLock()` and simplifies code. ## Test Plan None. Co-authored-by: DevOps Service <devops-service@algorand.com> Co-authored-by: Jason Paulos <jasonpaulos@users.noreply.github.com> Co-authored-by: chris erway <51567+cce@users.noreply.github.com> Co-authored-by: Shant Karakashian <55754073+algonautshant@users.noreply.github.com> Co-authored-by: John Lee <64482439+algojohnlee@users.noreply.github.com> Co-authored-by: Will Winder <wwinder.unh@gmail.com> Co-authored-by: Ben Guidarelli <ben.guidarelli@gmail.com> Co-authored-by: Pavel Zbitskiy <65323360+algorandskiy@users.noreply.github.com> Co-authored-by: Jack <87339414+algojack@users.noreply.github.com> Co-authored-by: John Lee <john.lee@algorand.com> Co-authored-by: algobarb <78746954+algobarb@users.noreply.github.com> Co-authored-by: Zeph Grunschlag <tzaffi@users.noreply.github.com> Co-authored-by: Fabrice Benhamouda <fabrice.benhamouda@normalesup.org> Co-authored-by: Tsachi Herman <tsachi.herman@algorand.com> Co-authored-by: Tolik Zinovyev <tolik@algorand.com> Co-authored-by: egieseke <eric_gieseke@yahoo.com> Co-authored-by: Quentin Kniep <kniepque@hu-berlin.de> Co-authored-by: Hang Su <87964331+ahangsu@users.noreply.github.com> Co-authored-by: Or Aharonee <or.aharonee@algorand.com> Co-authored-by: Barbara Poon <barbara.poon@algorand.com>

* Update the Version, BuildNumber, genesistimestamp.data * Support transaction arguments for `goal app method` (#3233) * Implement transactions as arguments * Fix indexing and dryrun issue * Add docstring * Satisfy review dog * Fix pointer issue * Fix group command * Rename e2e test * Fix filename variable * Add e2e test * Use tab * CI: use libboost-math-dev instead of libboost-all-dev (#3223) ## Summary Small change: libboost-math-dev requires just 4 packages to install, while libboost-all-dev requires > 100. Only Debian/Ubuntu distributions provide fine-grained boost packages like this, but should shave a little time off the CI builds. (Our only boost include is boost/math/distributions/binomial.hpp.) ## Test Plan Builds should pass as before. Now that we are no longer using Travis for Linux builds, the side effect of libboost-all-dev installing make and other missing build tools on Travis encountered in #2717 is no longer a concern. * testing: fixes to rest-participation-key e2e test (#3238) ## Summary - Test to make sure RES has the right input before counting line numbers for result size. - Rest RES to empty so that the same output is not recycled in case of an error. - exit 1 in case of an error - Reduce LAST_ROUND from 1200000 to 120 - "Get List of Keys" before getting NUM_IDS_3 otherwise it will recycle old RES value. * testing: interactive mode for e2e testing (#3227) ## Summary Some e2e tests require a python environment for testing. Unfortunately, setting up that environment adequately similar to the testing environment may not be trivial. This change introduces an interactive mode to the e2e.sh script which stops at the point of running the tests, and allows the user to run the tests from the same testing environment. ## Test Plan No tests needed. Tested the script locally. * Make dev-mode tests less flaky. (#3252) ## Summary Fix a couple flaws in the new go-e2e tests built ontop of DevMode: * Shutdown the fixture when finished. * Don't run in parallel. * Longer delays / better algorithms to wait for data flushing to complete. * Check for "out of order" keys. ## Test Plan N/A, this is a test. * adding libtool to ubuntu deps (#3251) ## Summary The sandbox is not building with dev config using master branch algorand/sandbox#85, complains about libtool not being installed Guessing from this change #3223 Adding libtool to UBUNTU_DEPS in install scripts ## Test Plan Set config in sandbox to my branch `sandbox up dev` It built * Fix error shadowing in Eval (#3258) ## Summary Error from account preloading was shadowed by returning a wrong err variable. This caused subsequent problems in account updates and masked the original failure. ## Test Plan Use existing tests * Disable flaky test. (#3256) ## Summary This test doesn't work properly, disable it until #3255 addresses any underlying problems. * Update the Version, BuildNumber, genesistimestamp.data * Fix a data race in app tests (#3269) ## Summary A test helper function `commitRound` accessed `l.trackers.lastFlushTime` without taking a lock. Fixed. ## Test Plan ``` go test ./ledger -run TestAppEmpty -race -count=50 ok github.com/algorand/go-algorand/ledger 4.078s ``` * Fix e2e.sh mixed indent characters. (#3266) ## Summary Fix e2e.sh mixed indent characters. * Fix ParticipationKeyResponse type. (#3264) ## Summary Fix a small type discrepancy in the OpenAPI spec ahead of some other work that's about to happen. * disable parallelism for e2e-go tests (#3242) ## Summary This sets `-p 1` for the e2e-go tests, intended to make them more deterministic when running on a VM with relatively constrained resources. Since each e2e-go test might spin up a few nodes, it seems like it would help to avoid resource contention. ## Test Plan Tests should run as before. Desired effect can be verified by looking at the test output where the value of PARALLEL_FLAG is printed out before tests are run. * Updating Readme.md with circleci status badges (#3245) * Fix formatting for CircleCI badges (#3272) * Add Custom Scenario for Performance Testing (#3278) Add Custom Scenario for Performance Testing. Add README on how to run custom scenario and modify create_and_deploy_recipe.sh to accept a network template that will generate a new recipe. * ParticipationRegistry - StateProof loading methods (#3261) ## Summary Add ParticipationRegistry methods for setting and retrieving state proof keys. Since they aren't in master yet there is a `type StateProofKey []byte` stub which will need to be updated later. ## Test Plan New unit tests. * Op base64 decode (#3220) b64 opcode, tests, and specs * Bump Version, Remove buildnumber.dat and genesistimestamp.dat files. * Change golang version to 1.16.11 in go-algorand (#2825) Upgrading to 1.16 to help alleviate issues with working on different go versions, and update to a supported, more secure version. Release notes for Go 1.15 and 1.16: https://tip.golang.org/doc/go1.16 https://tip.golang.org/doc/go1.15 * Compatibility mode for partkeyinfo. (#3291) ## Summary Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running Workarounds: 1) use an older `goal` binary. 2) use `algokey part info --keyfile <file>` ## Test Plan Tested manually: ``` ~$ goal account partkeyinfo -d /tmp/private_network/Node/ Dumping participation key info from /tmp/private_network/Node/... Participation ID: CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA Parent address: DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU Last vote round: 1 Last block proposal round: 2 Effective first round: 1 Effective last round: 3000000 First round: 0 Last round: 3000000 Key dilution: 10000 Selection key: 5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ= Voting key: PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs= ~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c Dumping participation key info from /tmp/private_network/Node/... ------------------------------------------------------------------ File: Wallet2.0.3000000.partkey { "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU", "last": 3000000, "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=", "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=", "voteKD": 10000 } ``` * TestEcdsa: fix flaky "tampering" of public key (#3282) ## Summary This test (TestEcdsa) tests the ecdsa_pk_decompress opcode and intentionally "tampers" with the public key by setting the first byte to zero. Occasionally this test is failing, likely because the first byte was already zero. (The test failures are for the cases where failure is expected, `pass=false`) ## Test Plan Existing test should pass, occasional flakiness should go away. * Support reference types in `goal app method` (#3275) * Fix method signature parse bug * Support reference types * Review dog fixes * Fix comments * Add a hash prefix for ARCs-related hashes (#3298) ## Summary This is to allow ARCs (github.com/algorandfoundation/ARCs) to have their own hash prefix without risk of collision. ## Test Plan It is purely informational. There is no real code change. * catchup: suspend the catchup session once the agreement service kicks in (#3299) The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on. The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it. In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete. When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup. A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch. However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete. This change fixes the return value from true to false. * Compatibility mode for partkeyinfo. (#3291) ## Summary Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running Workarounds: 1) use an older `goal` binary. 2) use `algokey part info --keyfile <file>` ## Test Plan Tested manually: ``` ~$ goal account partkeyinfo -d /tmp/private_network/Node/ Dumping participation key info from /tmp/private_network/Node/... Participation ID: CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA Parent address: DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU Last vote round: 1 Last block proposal round: 2 Effective first round: 1 Effective last round: 3000000 First round: 0 Last round: 3000000 Key dilution: 10000 Selection key: 5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ= Voting key: PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs= ~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c Dumping participation key info from /tmp/private_network/Node/... ------------------------------------------------------------------ File: Wallet2.0.3000000.partkey { "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU", "last": 3000000, "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=", "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=", "voteKD": 10000 } ``` * catchup: suspend the catchup session once the agreement service kicks in (#3299) The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on. The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it. In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete. When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup. A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch. However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete. This change fixes the return value from true to false. * Bump buildnumber.dat * testing: disable flaky test (#3268) Disable a flaky test, to be re-enabled later with #3267. * enumerate conditions that might cause this fetchAndWrite to return false (#3301) ## Summary The fetchAndWrite function contains some complex logic to ultimately determine if we should continue trying to catch up. The conditions that might cause it to return false should be more explicitly enumerated. ## Test Plan Just comments * make sure the block service is not attempting to access the ledger after being stopped. (#3303) ## Summary The block service was attempting to serve block via the http handler even after it has been stopped. This lead to undesired downstream failures in the ledger, which was shutdown as well. ## Test Plan unit test added. * Avoid creating algod process for the sole purpose of retrieving the genesis-id. (#3308) ## Summary Avoid creating algod process for the sole purpose of retrieving the genesis-id. Existing code was calling `algod -G -d <data dir>` in order to obtain the genesis version string. The genesis version string can be easily retrieved by loading the genesis file. ## Test Plan Use existing e2e tests. * documentation: fix algorand specs link (#3309) ## Summary This PR fixes a link in a README. ## Testing I clicked on the new link. * testing: reword partitiontest lint message. (#3297) ## Summary The wording on this was tripping me, maybe I was having an off day. I think it would be slightly easier if the message were to tell exactly what you need to do (and not use the angle brackets). * testing: fix random data race in TestAppAccountDataStorage (#3315) fix random data race in unit test * ledger: perform the catchpoint writing outside the trackers lock. (#3311) ## Summary This PR moves the catchpoint file writing to be performed outside of the trackers lock. This resolves the issue where a long catchpoint file writing blocks the agreement from validating and propagating votes. ## Test Plan * [x] Test manually & use existing tests. * [x] Implement a unit test * [x] Deploy a local network where the catchpoint writing takes a long time and verify it doesn't get blocked during catchpoint writing. * Separate tx and key validity for `goal account renewpartkey` (#3286) Always use currentRound+proto.MaxTxnLife as last valid round for the transaction when renewing instead of using the partkey validity period. This fixes #3283 * Add qkniep to THANKS.md (#3320) ## Summary Add qkniep to THANKS.md * Followup to opcode base64_decode (#3288) * alphabet begone in favor of encoding * unit test various padding and whitespace scenarios * padding permutations also fail * "Slicing" --> "Manipulation" * fix the codegen fail? * Documenting padding, whitespace, other character behavior * Add help and fish mode to e2e interactive mode. (#3313) ## Summary Minor improvements to e2e.sh interactive mode: * add to -h output * do not run start stop test in interactive mode * support fish shell ## Test Plan Manual testing: ``` ~$ ./e2e.sh -i ... lots of output removed ... ********** READY ********** The test environment is now set. You can now run tests in another terminal. Configure the environment: set -g VIRTUAL_ENV "/home/will/go/src/github.com/algorand/go-algorand/tmp/out/e2e/130013-1639576513257/ve" set -g PATH "$VIRTUAL_ENV/bin:$PATH" python3 "/home/will/go/src/github.com/algorand/go-algorand/test/scripts"/e2e_client_runner.py "/home/will/go/src/github.com/algorand/go-algorand/test/scripts"/e2e_subs/SCRIPT_FILE_NAME Press enter to shut down the test environment... ``` * Minimum Account Balance in Algod (#3287) * Add convertAddress tool. (#3304) ## Summary New tool: convertAddress I share this tool with someone every few months, putting it in the repo along with some documentation should make it easier to share and encourage people to share it amongst themselves if it's useful. Merge `debug` into `tools` to make it easier to organize these miscellaneous tools. * tealdbg: increase intermediate reading/writing buffers (#3335) ## Summary Some large teal source files cause the tealdbg/cdt session to choke. Upping the buffer size to allow for larger source files. closes #3100 ## Test Plan Run tealdbg with a large source teal file, ensure the source file makes it to cdt without choking. * Adding method pseudo op to readme (#3338) * add config.DeadlockDetectionThreshold (#3339) Summary This allows for the deadlock detection threshold to be set by configuration. Test Plan Existing tests should pass. * Fix flaky test in randomized ABI encoding test (#3346) * update abi encoding test random testcase generator, scale down parameters to avoid flaky test * parameterized test script * add notes to explain why flaky test is eliminated * show more information from self-roundtrip testing * fully utilize require, remove fmt * ledger: perform the catchpoint writing outside the trackers lock. (#3311) ## Summary This PR moves the catchpoint file writing to be performed outside of the trackers lock. This resolves the issue where a long catchpoint file writing blocks the agreement from validating and propagating votes. ## Test Plan * [x] Test manually & use existing tests. * [x] Implement a unit test * [x] Deploy a local network where the catchpoint writing takes a long time and verify it doesn't get blocked during catchpoint writing. * Bump version number * Update `goal app method` handling of args and return values (#3352) * Fix method call arg overflow handling * Only check last log for return value * Address feedback * Add comment explaining ABI return prefix * Support app creation in `goal app method` (#3353) * Support app creation in `goal app method` * Don't use nonprintable tab character * Link to specific gist version * Fix error messages * Rename `methodCreation` to `methodCreatesApp` * Update license to 2022 (#3360) Update license on all source files to 2022. * Add totals checks into acct updates tests (#3367) ## Summary After #2922 there is some leftover unused code for totals calculations. Turned this code into actual asserts. ## Test Plan This is tests update * remove buildnumber.dat * PKI State Proof Incremental Key Loading (#3281) ## Summary Followup to #3261 (contained in diff). Use the new key loading routine from the REST API. ## Test Plan New unit tests. * Limit number of simultaneous REST connections (#3326) ## Summary This PR limits the number of simultaneous REST connections we process to prevent the exhaustion of resources and ultimately a crash. Two limits are introduced: soft and hard. When the soft limit is exceeded, new connections are returned the 429 Too Many Requests http code. When the hard limit is exceeded, new connections are accepted and immediately closed. Partially resolves algorand/go-algorand-internal#1814. ## Test Plan Added unit tests. * Use rejecting limit listener in WebsocketNetwork. (#3380) ## Summary Replace the standard limit listener with the new rejecting limit listener in `WebsocketNetwork`. This will let the dialing node know that connection is impossible faster. ## Test Plan Probably not necessary. * Delete unused constant. (#3379) ## Summary This PR deletes an unused constant. ## Test Plan None. * Add test to exercise lookup corner cases (#3376) ## Summary This test attempts to cover the case when an accountUpdates.lookupX method can't find the requested address, falls through looking at deltas and the LRU accounts cache, then hits the database — only to discover that the round stored in the database (committed in `accountUpdates.commitRound`) is out of sync with `accountUpdates.cachedDBRound` (updated a little bit later in `accountUpdates.postCommit`). In this case, the lookup method waits and tries again, iterating the `for { }` it is in. We did not have coverage for this code path before. ## Test Plan Adds new test. * Test for catchup stop on completion (#3306) Adding a test for the fix in #3299 ## Test Plan This is a test * Delete unused AtomicCommitWriteLock(). (#3383) ## Summary This PR deletes unused `AtomicCommitWriteLock()` and simplifies code. ## Test Plan None. Co-authored-by: DevOps Service <devops-service@algorand.com> Co-authored-by: Jason Paulos <jasonpaulos@users.noreply.github.com> Co-authored-by: chris erway <51567+cce@users.noreply.github.com> Co-authored-by: Shant Karakashian <55754073+algonautshant@users.noreply.github.com> Co-authored-by: John Lee <64482439+algojohnlee@users.noreply.github.com> Co-authored-by: Will Winder <wwinder.unh@gmail.com> Co-authored-by: Ben Guidarelli <ben.guidarelli@gmail.com> Co-authored-by: Pavel Zbitskiy <65323360+algorandskiy@users.noreply.github.com> Co-authored-by: Jack <87339414+algojack@users.noreply.github.com> Co-authored-by: John Lee <john.lee@algorand.com> Co-authored-by: algobarb <78746954+algobarb@users.noreply.github.com> Co-authored-by: Zeph Grunschlag <tzaffi@users.noreply.github.com> Co-authored-by: Fabrice Benhamouda <fabrice.benhamouda@normalesup.org> Co-authored-by: Tsachi Herman <tsachi.herman@algorand.com> Co-authored-by: Tolik Zinovyev <tolik@algorand.com> Co-authored-by: egieseke <eric_gieseke@yahoo.com> Co-authored-by: Quentin Kniep <kniepque@hu-berlin.de> Co-authored-by: Hang Su <87964331+ahangsu@users.noreply.github.com> Co-authored-by: Or Aharonee <or.aharonee@algorand.com> Co-authored-by: Barbara Poon <barbara.poon@algorand.com>

* Update the Version, BuildNumber, genesistimestamp.data * Three new globals for to help contract-to-contract usability * detritis * Check error * doc comments * Support transaction arguments for `goal app method` (#3233) * Implement transactions as arguments * Fix indexing and dryrun issue * Add docstring * Satisfy review dog * Fix pointer issue * Fix group command * Rename e2e test * Fix filename variable * Add e2e test * Use tab * CI: use libboost-math-dev instead of libboost-all-dev (#3223) ## Summary Small change: libboost-math-dev requires just 4 packages to install, while libboost-all-dev requires > 100. Only Debian/Ubuntu distributions provide fine-grained boost packages like this, but should shave a little time off the CI builds. (Our only boost include is boost/math/distributions/binomial.hpp.) ## Test Plan Builds should pass as before. Now that we are no longer using Travis for Linux builds, the side effect of libboost-all-dev installing make and other missing build tools on Travis encountered in #2717 is no longer a concern. * testing: fixes to rest-participation-key e2e test (#3238) ## Summary - Test to make sure RES has the right input before counting line numbers for result size. - Rest RES to empty so that the same output is not recycled in case of an error. - exit 1 in case of an error - Reduce LAST_ROUND from 1200000 to 120 - "Get List of Keys" before getting NUM_IDS_3 otherwise it will recycle old RES value. * testing: interactive mode for e2e testing (#3227) ## Summary Some e2e tests require a python environment for testing. Unfortunately, setting up that environment adequately similar to the testing environment may not be trivial. This change introduces an interactive mode to the e2e.sh script which stops at the point of running the tests, and allows the user to run the tests from the same testing environment. ## Test Plan No tests needed. Tested the script locally. * Make dev-mode tests less flaky. (#3252) ## Summary Fix a couple flaws in the new go-e2e tests built ontop of DevMode: * Shutdown the fixture when finished. * Don't run in parallel. * Longer delays / better algorithms to wait for data flushing to complete. * Check for "out of order" keys. ## Test Plan N/A, this is a test. * adding libtool to ubuntu deps (#3251) ## Summary The sandbox is not building with dev config using master branch algorand/sandbox#85, complains about libtool not being installed Guessing from this change #3223 Adding libtool to UBUNTU_DEPS in install scripts ## Test Plan Set config in sandbox to my branch `sandbox up dev` It built * Fix error shadowing in Eval (#3258) ## Summary Error from account preloading was shadowed by returning a wrong err variable. This caused subsequent problems in account updates and masked the original failure. ## Test Plan Use existing tests * Disable flaky test. (#3256) ## Summary This test doesn't work properly, disable it until #3255 addresses any underlying problems. * Update the Version, BuildNumber, genesistimestamp.data * Fix a data race in app tests (#3269) ## Summary A test helper function `commitRound` accessed `l.trackers.lastFlushTime` without taking a lock. Fixed. ## Test Plan ``` go test ./ledger -run TestAppEmpty -race -count=50 ok github.com/algorand/go-algorand/ledger 4.078s ``` * Fix e2e.sh mixed indent characters. (#3266) ## Summary Fix e2e.sh mixed indent characters. * Fix ParticipationKeyResponse type. (#3264) ## Summary Fix a small type discrepancy in the OpenAPI spec ahead of some other work that's about to happen. * disable parallelism for e2e-go tests (#3242) ## Summary This sets `-p 1` for the e2e-go tests, intended to make them more deterministic when running on a VM with relatively constrained resources. Since each e2e-go test might spin up a few nodes, it seems like it would help to avoid resource contention. ## Test Plan Tests should run as before. Desired effect can be verified by looking at the test output where the value of PARALLEL_FLAG is printed out before tests are run. * Updating Readme.md with circleci status badges (#3245) * Fix formatting for CircleCI badges (#3272) * Add Custom Scenario for Performance Testing (#3278) Add Custom Scenario for Performance Testing. Add README on how to run custom scenario and modify create_and_deploy_recipe.sh to accept a network template that will generate a new recipe. * Impose limits on the entire "tree" of inner calls. This also increases the realism of testing of multiple app calls in a group by creating the EvalParams with the real constructor, thus getting the pooling stuff tested here without playing games manipulating the ep after construction. * ParticipationRegistry - StateProof loading methods (#3261) ## Summary Add ParticipationRegistry methods for setting and retrieving state proof keys. Since they aren't in master yet there is a `type StateProofKey []byte` stub which will need to be updated later. ## Test Plan New unit tests. * Op base64 decode (#3220) b64 opcode, tests, and specs * Bump Version, Remove buildnumber.dat and genesistimestamp.dat files. * Change golang version to 1.16.11 in go-algorand (#2825) Upgrading to 1.16 to help alleviate issues with working on different go versions, and update to a supported, more secure version. Release notes for Go 1.15 and 1.16: https://tip.golang.org/doc/go1.16 https://tip.golang.org/doc/go1.15 * Compatibility mode for partkeyinfo. (#3291) ## Summary Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running Workarounds: 1) use an older `goal` binary. 2) use `algokey part info --keyfile <file>` ## Test Plan Tested manually: ``` ~$ goal account partkeyinfo -d /tmp/private_network/Node/ Dumping participation key info from /tmp/private_network/Node/... Participation ID: CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA Parent address: DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU Last vote round: 1 Last block proposal round: 2 Effective first round: 1 Effective last round: 3000000 First round: 0 Last round: 3000000 Key dilution: 10000 Selection key: 5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ= Voting key: PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs= ~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c Dumping participation key info from /tmp/private_network/Node/... ------------------------------------------------------------------ File: Wallet2.0.3000000.partkey { "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU", "last": 3000000, "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=", "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=", "voteKD": 10000 } ``` * TestEcdsa: fix flaky "tampering" of public key (#3282) ## Summary This test (TestEcdsa) tests the ecdsa_pk_decompress opcode and intentionally "tampers" with the public key by setting the first byte to zero. Occasionally this test is failing, likely because the first byte was already zero. (The test failures are for the cases where failure is expected, `pass=false`) ## Test Plan Existing test should pass, occasional flakiness should go away. * Move appID tracking into EvalContext, out of LedgerForLogic This change increases the seperation between AVM execution and the ledger being used to lookup resources. Previously, the ledger kept track of the appID being executed, to offer a narrower interface to those resources. But now, with app-to-app calls, the appID being executed must change, and the AVM needs to maintain the current appID. * Stupid linter * Support reference types in `goal app method` (#3275) * Fix method signature parse bug * Support reference types * Review dog fixes * Fix comments * Add a hash prefix for ARCs-related hashes (#3298) ## Summary This is to allow ARCs (github.com/algorandfoundation/ARCs) to have their own hash prefix without risk of collision. ## Test Plan It is purely informational. There is no real code change. * catchup: suspend the catchup session once the agreement service kicks in (#3299) The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on. The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it. In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete. When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup. A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch. However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete. This change fixes the return value from true to false. * Compatibility mode for partkeyinfo. (#3291) ## Summary Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running Workarounds: 1) use an older `goal` binary. 2) use `algokey part info --keyfile <file>` ## Test Plan Tested manually: ``` ~$ goal account partkeyinfo -d /tmp/private_network/Node/ Dumping participation key info from /tmp/private_network/Node/... Participation ID: CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA Parent address: DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU Last vote round: 1 Last block proposal round: 2 Effective first round: 1 Effective last round: 3000000 First round: 0 Last round: 3000000 Key dilution: 10000 Selection key: 5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ= Voting key: PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs= ~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c Dumping participation key info from /tmp/private_network/Node/... ------------------------------------------------------------------ File: Wallet2.0.3000000.partkey { "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU", "last": 3000000, "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=", "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=", "voteKD": 10000 } ``` * catchup: suspend the catchup session once the agreement service kicks in (#3299) The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on. The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it. In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete. When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup. A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch. However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete. This change fixes the return value from true to false. * Bump buildnumber.dat * testing: disable flaky test (#3268) Disable a flaky test, to be re-enabled later with #3267. * enumerate conditions that might cause this fetchAndWrite to return false (#3301) ## Summary The fetchAndWrite function contains some complex logic to ultimately determine if we should continue trying to catch up. The conditions that might cause it to return false should be more explicitly enumerated. ## Test Plan Just comments * Fix unit tests error messages * make sure the block service is not attempting to access the ledger after being stopped. (#3303) ## Summary The block service was attempting to serve block via the http handler even after it has been stopped. This lead to undesired downstream failures in the ledger, which was shutdown as well. ## Test Plan unit test added. * Avoid creating algod process for the sole purpose of retrieving the genesis-id. (#3308) ## Summary Avoid creating algod process for the sole purpose of retrieving the genesis-id. Existing code was calling `algod -G -d <data dir>` in order to obtain the genesis version string. The genesis version string can be easily retrieved by loading the genesis file. ## Test Plan Use existing e2e tests. * documentation: fix algorand specs link (#3309) ## Summary This PR fixes a link in a README. ## Testing I clicked on the new link. * testing: reword partitiontest lint message. (#3297) ## Summary The wording on this was tripping me, maybe I was having an off day. I think it would be slightly easier if the message were to tell exactly what you need to do (and not use the angle brackets). * testing: fix random data race in TestAppAccountDataStorage (#3315) fix random data race in unit test * Allow access to resources created in the same transaction group The method will be reworked, but the tests are correct and want to get them visible to team. * ledger: perform the catchpoint writing outside the trackers lock. (#3311) ## Summary This PR moves the catchpoint file writing to be performed outside of the trackers lock. This resolves the issue where a long catchpoint file writing blocks the agreement from validating and propagating votes. ## Test Plan * [x] Test manually & use existing tests. * [x] Implement a unit test * [x] Deploy a local network where the catchpoint writing takes a long time and verify it doesn't get blocked during catchpoint writing. * Separate tx and key validity for `goal account renewpartkey` (#3286) Always use currentRound+proto.MaxTxnLife as last valid round for the transaction when renewing instead of using the partkey validity period. This fixes #3283 * Add qkniep to THANKS.md (#3320) ## Summary Add qkniep to THANKS.md * Followup to opcode base64_decode (#3288) * alphabet begone in favor of encoding * unit test various padding and whitespace scenarios * padding permutations also fail * "Slicing" --> "Manipulation" * fix the codegen fail? * Documenting padding, whitespace, other character behavior * Add help and fish mode to e2e interactive mode. (#3313) ## Summary Minor improvements to e2e.sh interactive mode: * add to -h output * do not run start stop test in interactive mode * support fish shell ## Test Plan Manual testing: ``` ~$ ./e2e.sh -i ... lots of output removed ... ********** READY ********** The test environment is now set. You can now run tests in another terminal. Configure the environment: set -g VIRTUAL_ENV "/home/will/go/src/github.com/algorand/go-algorand/tmp/out/e2e/130013-1639576513257/ve" set -g PATH "$VIRTUAL_ENV/bin:$PATH" python3 "/home/will/go/src/github.com/algorand/go-algorand/test/scripts"/e2e_client_runner.py "/home/will/go/src/github.com/algorand/go-algorand/test/scripts"/e2e_subs/SCRIPT_FILE_NAME Press enter to shut down the test environment... ``` * Minimum Account Balance in Algod (#3287) * Access to apps created in group Also adds some tests that are currently skipped for testing - access to addresses of newly created apps - use of gaid in inner transactions Both require some work to implement the thing being tested. * Remove tracked created mechanism in favor of examining applydata. * Add convertAddress tool. (#3304) ## Summary New tool: convertAddress I share this tool with someone every few months, putting it in the repo along with some documentation should make it easier to share and encourage people to share it amongst themselves if it's useful. Merge `debug` into `tools` to make it easier to organize these miscellaneous tools. * tealdbg: increase intermediate reading/writing buffers (#3335) ## Summary Some large teal source files cause the tealdbg/cdt session to choke. Upping the buffer size to allow for larger source files. closes #3100 ## Test Plan Run tealdbg with a large source teal file, ensure the source file makes it to cdt without choking. * Adding method pseudo op to readme (#3338) * Allow v6 AVM code to use in-group created asas, apps (& their accts) One exception - apps can not mutate (put or del) keys from the app accounts, because EvalDelta cannot encode such changes. * lint docs * typo * The review dog needs obedience training. * add config.DeadlockDetectionThreshold (#3339) Summary This allows for the deadlock detection threshold to be set by configuration. Test Plan Existing tests should pass. * Use one EvalParams for logic evals, another for apps in dry run We used to use one ep per transaction, shared between sig and and app. But the new model of ep usage is to keep using one while evaluating an entire group. The app ep is now built logic.NewAppEvalParams which, hopefully, will prevent some bugs when we change something in the EvalParams and don't reflect it in what was a "raw" EvalParams construction in debugger and dry run. * Use logic.NewAppEvalParams to decrease copying and bugs in debugger * Simplify use of NewEvalParams. No more nil return when no apps. This way, NewEvalParams can be used for all creations of EvalParams, whether they are intended for logicsig or app use, greatly simplifying the way we make them for use by dry run or debugger (where they serve double duty). * Remove explicit PastSideEffects handling in tealdbg * Fix flaky test in randomized ABI encoding test (#3346) * update abi encoding test random testcase generator, scale down parameters to avoid flaky test * parameterized test script * add notes to explain why flaky test is eliminated * show more information from self-roundtrip testing * fully utilize require, remove fmt * Always create EvalParams to evaluate a transaction group. We used to have an optimization to avoid creating EvalParams unless there was an app call in the transaction group. But the interface to allow transaction processing to communicate changes into the EvalParams is complicated by that (we must only do it if there is one!) This also allows us to use the same construction function for eps created for app and logic evaluation, simplifying dry-run and debugger. The optimization is less needed now anyway: 1) The ep is now shared for the whole group, so it's only one. 2) The ep is smaller now, as we only store nil pointers instead of larger scratch space objects for non-app calls. * Correct mistaken commit * ledger: perform the catchpoint writing outside the trackers lock. (#3311) ## Summary This PR moves the catchpoint file writing to be performed outside of the trackers lock. This resolves the issue where a long catchpoint file writing blocks the agreement from validating and propagating votes. ## Test Plan * [x] Test manually & use existing tests. * [x] Implement a unit test * [x] Deploy a local network where the catchpoint writing takes a long time and verify it doesn't get blocked during catchpoint writing. * Bump version number * Update `goal app method` handling of args and return values (#3352) * Fix method call arg overflow handling * Only check last log for return value * Address feedback * Add comment explaining ABI return prefix * Support app creation in `goal app method` (#3353) * Support app creation in `goal app method` * Don't use nonprintable tab character * Link to specific gist version * Fix error messages * Rename `methodCreation` to `methodCreatesApp` * Spec improvments * Update license to 2022 (#3360) Update license on all source files to 2022. * Add totals checks into acct updates tests (#3367) ## Summary After #2922 there is some leftover unused code for totals calculations. Turned this code into actual asserts. ## Test Plan This is tests update * More spec improvments, including resource "availability" * Recursively return inner transaction tree * Lint * No need for ConfirmedRound, so don't deref a nil pointer! * remove buildnumber.dat * license check * Shut up, dawg. * PKI State Proof Incremental Key Loading (#3281) ## Summary Followup to #3261 (contained in diff). Use the new key loading routine from the REST API. ## Test Plan New unit tests. * Limit number of simultaneous REST connections (#3326) ## Summary This PR limits the number of simultaneous REST connections we process to prevent the exhaustion of resources and ultimately a crash. Two limits are introduced: soft and hard. When the soft limit is exceeded, new connections are returned the 429 Too Many Requests http code. When the hard limit is exceeded, new connections are accepted and immediately closed. Partially resolves algorand/go-algorand-internal#1814. ## Test Plan Added unit tests. * Use rejecting limit listener in WebsocketNetwork. (#3380) ## Summary Replace the standard limit listener with the new rejecting limit listener in `WebsocketNetwork`. This will let the dialing node know that connection is impossible faster. ## Test Plan Probably not necessary. * Delete unused constant. (#3379) ## Summary This PR deletes an unused constant. ## Test Plan None. * Add test to exercise lookup corner cases (#3376) ## Summary This test attempts to cover the case when an accountUpdates.lookupX method can't find the requested address, falls through looking at deltas and the LRU accounts cache, then hits the database — only to discover that the round stored in the database (committed in `accountUpdates.commitRound`) is out of sync with `accountUpdates.cachedDBRound` (updated a little bit later in `accountUpdates.postCommit`). In this case, the lookup method waits and tries again, iterating the `for { }` it is in. We did not have coverage for this code path before. ## Test Plan Adds new test. * Test for catchup stop on completion (#3306) Adding a test for the fix in #3299 ## Test Plan This is a test * Delete unused AtomicCommitWriteLock(). (#3383) ## Summary This PR deletes unused `AtomicCommitWriteLock()` and simplifies code. ## Test Plan None. Co-authored-by: DevOps Service <devops-service@algorand.com> Co-authored-by: Jason Paulos <jasonpaulos@users.noreply.github.com> Co-authored-by: chris erway <51567+cce@users.noreply.github.com> Co-authored-by: Shant Karakashian <55754073+algonautshant@users.noreply.github.com> Co-authored-by: John Lee <64482439+algojohnlee@users.noreply.github.com> Co-authored-by: Will Winder <wwinder.unh@gmail.com> Co-authored-by: Ben Guidarelli <ben.guidarelli@gmail.com> Co-authored-by: Pavel Zbitskiy <65323360+algorandskiy@users.noreply.github.com> Co-authored-by: Jack <87339414+algojack@users.noreply.github.com> Co-authored-by: John Lee <john.lee@algorand.com> Co-authored-by: algobarb <78746954+algobarb@users.noreply.github.com> Co-authored-by: Zeph Grunschlag <tzaffi@users.noreply.github.com> Co-authored-by: Fabrice Benhamouda <fabrice.benhamouda@normalesup.org> Co-authored-by: Tsachi Herman <tsachi.herman@algorand.com> Co-authored-by: Tolik Zinovyev <tolik@algorand.com> Co-authored-by: egieseke <eric_gieseke@yahoo.com> Co-authored-by: Quentin Kniep <kniepque@hu-berlin.de> Co-authored-by: Hang Su <87964331+ahangsu@users.noreply.github.com> Co-authored-by: Or Aharonee <or.aharonee@algorand.com> Co-authored-by: Barbara Poon <barbara.poon@algorand.com>

* The new inner appl fields * Unit tests for field setting on appls * Construct EvalDelta in AVM rather than by inspecting ledger * Obey the linter! * more LedgerForEvaluation accomodation * Test inner evaldeltas * Checks on calling old AVM apps, or re-entrancy * Allow opcode budget to be added to by executing inner apps. * TxID and GroupID for inner transactions * gitxn/gitxna * Lint, spec generate * txn simplifications * Encode "arrayness" in the txn field spec * Pavel's CR comments * Update tests to distinguish assembly / eval errors * Test itxn_field assembly separate from eval * factor out the array index parsing of all the txn assembly * Consistent errors and parsign for many opcodes * Cleanup immediate parsing, prep txn effects testing * EvalParams is now a single object used to evaluate each txn in turn. * Simplifications for the Dawg (the Review Dog) * Use a copy for the EvalParams.TxnGroup * Set the logicsig on txns in the GroupContext, so check() can see it * Update test for explicit empty check * Three new globals for to help contract-to-contract usability (#3237) * Three new globals for to help contract-to-contract usability * detritis * Check error * doc comments * Gloadss (#3248) * Three new globals for to help contract-to-contract usability * detritis * Check error * doc comments * opcode, docs, and tests * specs update * Feature/contract to contract (#3285) * Update the Version, BuildNumber, genesistimestamp.data * Three new globals for to help contract-to-contract usability * detritis * Check error * doc comments * Support transaction arguments for `goal app method` (#3233) * Implement transactions as arguments * Fix indexing and dryrun issue * Add docstring * Satisfy review dog * Fix pointer issue * Fix group command * Rename e2e test * Fix filename variable * Add e2e test * Use tab * CI: use libboost-math-dev instead of libboost-all-dev (#3223) ## Summary Small change: libboost-math-dev requires just 4 packages to install, while libboost-all-dev requires > 100. Only Debian/Ubuntu distributions provide fine-grained boost packages like this, but should shave a little time off the CI builds. (Our only boost include is boost/math/distributions/binomial.hpp.) ## Test Plan Builds should pass as before. Now that we are no longer using Travis for Linux builds, the side effect of libboost-all-dev installing make and other missing build tools on Travis encountered in #2717 is no longer a concern. * testing: fixes to rest-participation-key e2e test (#3238) ## Summary - Test to make sure RES has the right input before counting line numbers for result size. - Rest RES to empty so that the same output is not recycled in case of an error. - exit 1 in case of an error - Reduce LAST_ROUND from 1200000 to 120 - "Get List of Keys" before getting NUM_IDS_3 otherwise it will recycle old RES value. * testing: interactive mode for e2e testing (#3227) ## Summary Some e2e tests require a python environment for testing. Unfortunately, setting up that environment adequately similar to the testing environment may not be trivial. This change introduces an interactive mode to the e2e.sh script which stops at the point of running the tests, and allows the user to run the tests from the same testing environment. ## Test Plan No tests needed. Tested the script locally. * Make dev-mode tests less flaky. (#3252) ## Summary Fix a couple flaws in the new go-e2e tests built ontop of DevMode: * Shutdown the fixture when finished. * Don't run in parallel. * Longer delays / better algorithms to wait for data flushing to complete. * Check for "out of order" keys. ## Test Plan N/A, this is a test. * adding libtool to ubuntu deps (#3251) ## Summary The sandbox is not building with dev config using master branch https://github.com/algorand/sandbox/issues/85, complains about libtool not being installed Guessing from this change https://github.com/algorand/go-algorand/pull/3223 Adding libtool to UBUNTU_DEPS in install scripts ## Test Plan Set config in sandbox to my branch `sandbox up dev` It built * Fix error shadowing in Eval (#3258) ## Summary Error from account preloading was shadowed by returning a wrong err variable. This caused subsequent problems in account updates and masked the original failure. ## Test Plan Use existing tests * Disable flaky test. (#3256) ## Summary This test doesn't work properly, disable it until #3255 addresses any underlying problems. * Update the Version, BuildNumber, genesistimestamp.data * Fix a data race in app tests (#3269) ## Summary A test helper function `commitRound` accessed `l.trackers.lastFlushTime` without taking a lock. Fixed. ## Test Plan ``` go test ./ledger -run TestAppEmpty -race -count=50 ok github.com/algorand/go-algorand/ledger 4.078s ``` * Fix e2e.sh mixed indent characters. (#3266) ## Summary Fix e2e.sh mixed indent characters. * Fix ParticipationKeyResponse type. (#3264) ## Summary Fix a small type discrepancy in the OpenAPI spec ahead of some other work that's about to happen. * disable parallelism for e2e-go tests (#3242) ## Summary This sets `-p 1` for the e2e-go tests, intended to make them more deterministic when running on a VM with relatively constrained resources. Since each e2e-go test might spin up a few nodes, it seems like it would help to avoid resource contention. ## Test Plan Tests should run as before. Desired effect can be verified by looking at the test output where the value of PARALLEL_FLAG is printed out before tests are run. * Updating Readme.md with circleci status badges (#3245) * Fix formatting for CircleCI badges (#3272) * Add Custom Scenario for Performance Testing (#3278) Add Custom Scenario for Performance Testing. Add README on how to run custom scenario and modify create_and_deploy_recipe.sh to accept a network template that will generate a new recipe. * Impose limits on the entire "tree" of inner calls. This also increases the realism of testing of multiple app calls in a group by creating the EvalParams with the real constructor, thus getting the pooling stuff tested here without playing games manipulating the ep after construction. * ParticipationRegistry - StateProof loading methods (#3261) ## Summary Add ParticipationRegistry methods for setting and retrieving state proof keys. Since they aren't in master yet there is a `type StateProofKey []byte` stub which will need to be updated later. ## Test Plan New unit tests. * Op base64 decode (#3220) b64 opcode, tests, and specs * Bump Version, Remove buildnumber.dat and genesistimestamp.dat files. * Change golang version to 1.16.11 in go-algorand (#2825) Upgrading to 1.16 to help alleviate issues with working on different go versions, and update to a supported, more secure version. Release notes for Go 1.15 and 1.16: https://tip.golang.org/doc/go1.16 https://tip.golang.org/doc/go1.15 * Compatibility mode for partkeyinfo. (#3291) ## Summary Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running Workarounds: 1) use an older `goal` binary. 2) use `algokey part info --keyfile <file>` ## Test Plan Tested manually: ``` ~$ goal account partkeyinfo -d /tmp/private_network/Node/ Dumping participation key info from /tmp/private_network/Node/... Participation ID: CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA Parent address: DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU Last vote round: 1 Last block proposal round: 2 Effective first round: 1 Effective last round: 3000000 First round: 0 Last round: 3000000 Key dilution: 10000 Selection key: 5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ= Voting key: PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs= ~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c Dumping participation key info from /tmp/private_network/Node/... ------------------------------------------------------------------ File: Wallet2.0.3000000.partkey { "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU", "last": 3000000, "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=", "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=", "voteKD": 10000 } ``` * TestEcdsa: fix flaky "tampering" of public key (#3282) ## Summary This test (TestEcdsa) tests the ecdsa_pk_decompress opcode and intentionally "tampers" with the public key by setting the first byte to zero. Occasionally this test is failing, likely because the first byte was already zero. (The test failures are for the cases where failure is expected, `pass=false`) ## Test Plan Existing test should pass, occasional flakiness should go away. * Move appID tracking into EvalContext, out of LedgerForLogic This change increases the seperation between AVM execution and the ledger being used to lookup resources. Previously, the ledger kept track of the appID being executed, to offer a narrower interface to those resources. But now, with app-to-app calls, the appID being executed must change, and the AVM needs to maintain the current appID. * Stupid linter * Support reference types in `goal app method` (#3275) * Fix method signature parse bug * Support reference types * Review dog fixes * Fix comments * Add a hash prefix for ARCs-related hashes (#3298) ## Summary This is to allow ARCs (github.com/algorandfoundation/ARCs) to have their own hash prefix without risk of collision. ## Test Plan It is purely informational. There is no real code change. * catchup: suspend the catchup session once the agreement service kicks in (#3299) The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on. The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it. In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete. When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup. A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch. However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete. This change fixes the return value from true to false. * Compatibility mode for partkeyinfo. (#3291) ## Summary Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running Workarounds: 1) use an older `goal` binary. 2) use `algokey part info --keyfile <file>` ## Test Plan Tested manually: ``` ~$ goal account partkeyinfo -d /tmp/private_network/Node/ Dumping participation key info from /tmp/private_network/Node/... Participation ID: CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA Parent address: DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU Last vote round: 1 Last block proposal round: 2 Effective first round: 1 Effective last round: 3000000 First round: 0 Last round: 3000000 Key dilution: 10000 Selection key: 5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ= Voting key: PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs= ~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c Dumping participation key info from /tmp/private_network/Node/... ------------------------------------------------------------------ File: Wallet2.0.3000000.partkey { "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU", "last": 3000000, "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=", "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=", "voteKD": 10000 } ``` * catchup: suspend the catchup session once the agreement service kicks in (#3299) The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on. The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it. In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete. When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup. A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch. However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete. This change fixes the return value from true to false. * Bump buildnumber.dat * testing: disable flaky test (#3268) Disable a flaky test, to be re-enabled later with #3267. * enumerate conditions that might cause this fetchAndWrite to return false (#3301) ## Summary The fetchAndWrite function contains some complex logic to ultimately determine if we should continue trying to catch up. The conditions that might cause it to return false should be more explicitly enumerated. ## Test Plan Just comments * Fix unit tests error messages * make sure the block service is not attempting to access the ledger after being stopped. (#3303) ## Summary The block service was attempting to serve block via the http handler even after it has been stopped. This lead to undesired downstream failures in the ledger, which was shutdown as well. ## Test Plan unit test added. * Avoid creating algod process for the sole purpose of retrieving the genesis-id. (#3308) ## Summary Avoid creating algod process for the sole purpose of retrieving the genesis-id. Existing code was calling `algod -G -d <data dir>` in order to obtain the genesis version string. The genesis version string can be easily retrieved by loading the genesis file. ## Test Plan Use existing e2e tests. * documentation: fix algorand specs link (#3309) ## Summary This PR fixes a link in a README. ## Testing I clicked on the new link. * testing: reword partitiontest lint message. (#3297) ## Summary The wording on this was tripping me, maybe I was having an off day. I think it would be slightly easier if the message were to tell exactly what you need to do (and not use the angle brackets). * testing: fix random data race in TestAppAccountDataStorage (#3315) fix random data race in unit test * Allow access to resources created in the same transaction group The method will be reworked, but the tests are correct and want to get them visible to team. Co-authored-by: DevOps Service <devops-service@algorand.com> Co-authored-by: Jason Paulos <jasonpaulos@users.noreply.github.com> Co-authored-by: chris erway <51567+cce@users.noreply.github.com> Co-authored-by: Shant Karakashian <55754073+algonautshant@users.noreply.github.com> Co-authored-by: John Lee <64482439+algojohnlee@users.noreply.github.com> Co-authored-by: Will Winder <wwinder.unh@gmail.com> Co-authored-by: Ben Guidarelli <ben.guidarelli@gmail.com> Co-authored-by: Pavel Zbitskiy <65323360+algorandskiy@users.noreply.github.com> Co-authored-by: Jack <87339414+algojack@users.noreply.github.com> Co-authored-by: John Lee <john.lee@algorand.com> Co-authored-by: algobarb <78746954+algobarb@users.noreply.github.com> Co-authored-by: Zeph Grunschlag <tzaffi@users.noreply.github.com> Co-authored-by: Fabrice Benhamouda <fabrice.benhamouda@normalesup.org> Co-authored-by: Tsachi Herman <tsachi.herman@algorand.com> Co-authored-by: Tolik Zinovyev <tolik@algorand.com> Co-authored-by: egieseke <eric_gieseke@yahoo.com> * add access to resources created in the current group (#3340) * Feature/contract to contract (#3357) * Update the Version, BuildNumber, genesistimestamp.data * Three new globals for to help contract-to-contract usability * detritis * Check error * doc comments * Support transaction arguments for `goal app method` (#3233) * Implement transactions as arguments * Fix indexing and dryrun issue * Add docstring * Satisfy review dog * Fix pointer issue * Fix group command * Rename e2e test * Fix filename variable * Add e2e test * Use tab * CI: use libboost-math-dev instead of libboost-all-dev (#3223) ## Summary Small change: libboost-math-dev requires just 4 packages to install, while libboost-all-dev requires > 100. Only Debian/Ubuntu distributions provide fine-grained boost packages like this, but should shave a little time off the CI builds. (Our only boost include is boost/math/distributions/binomial.hpp.) ## Test Plan Builds should pass as before. Now that we are no longer using Travis for Linux builds, the side effect of libboost-all-dev installing make and other missing build tools on Travis encountered in #2717 is no longer a concern. * testing: fixes to rest-participation-key e2e test (#3238) ## Summary - Test to make sure RES has the right input before counting line numbers for result size. - Rest RES to empty so that the same output is not recycled in case of an error. - exit 1 in case of an error - Reduce LAST_ROUND from 1200000 to 120 - "Get List of Keys" before getting NUM_IDS_3 otherwise it will recycle old RES value. * testing: interactive mode for e2e testing (#3227) ## Summary Some e2e tests require a python environment for testing. Unfortunately, setting up that environment adequately similar to the testing environment may not be trivial. This change introduces an interactive mode to the e2e.sh script which stops at the point of running the tests, and allows the user to run the tests from the same testing environment. ## Test Plan No tests needed. Tested the script locally. * Make dev-mode tests less flaky. (#3252) ## Summary Fix a couple flaws in the new go-e2e tests built ontop of DevMode: * Shutdown the fixture when finished. * Don't run in parallel. * Longer delays / better algorithms to wait for data flushing to complete. * Check for "out of order" keys. ## Test Plan N/A, this is a test. * adding libtool to ubuntu deps (#3251) ## Summary The sandbox is not building with dev config using master branch https://github.com/algorand/sandbox/issues/85, complains about libtool not being installed Guessing from this change https://github.com/algorand/go-algorand/pull/3223 Adding libtool to UBUNTU_DEPS in install scripts ## Test Plan Set config in sandbox to my branch `sandbox up dev` It built * Fix error shadowing in Eval (#3258) ## Summary Error from account preloading was shadowed by returning a wrong err variable. This caused subsequent problems in account updates and masked the original failure. ## Test Plan Use existing tests * Disable flaky test. (#3256) ## Summary This test doesn't work properly, disable it until #3255 addresses any underlying problems. * Update the Version, BuildNumber, genesistimestamp.data * Fix a data race in app tests (#3269) ## Summary A test helper function `commitRound` accessed `l.trackers.lastFlushTime` without taking a lock. Fixed. ## Test Plan ``` go test ./ledger -run TestAppEmpty -race -count=50 ok github.com/algorand/go-algorand/ledger 4.078s ``` * Fix e2e.sh mixed indent characters. (#3266) ## Summary Fix e2e.sh mixed indent characters. * Fix ParticipationKeyResponse type. (#3264) ## Summary Fix a small type discrepancy in the OpenAPI spec ahead of some other work that's about to happen. * disable parallelism for e2e-go tests (#3242) ## Summary This sets `-p 1` for the e2e-go tests, intended to make them more deterministic when running on a VM with relatively constrained resources. Since each e2e-go test might spin up a few nodes, it seems like it would help to avoid resource contention. ## Test Plan Tests should run as before. Desired effect can be verified by looking at the test output where the value of PARALLEL_FLAG is printed out before tests are run. * Updating Readme.md with circleci status badges (#3245) * Fix formatting for CircleCI badges (#3272) * Add Custom Scenario for Performance Testing (#3278) Add Custom Scenario for Performance Testing. Add README on how to run custom scenario and modify create_and_deploy_recipe.sh to accept a network template that will generate a new recipe. * Impose limits on the entire "tree" of inner calls. This also increases the realism of testing of multiple app calls in a group by creating the EvalParams with the real constructor, thus getting the pooling stuff tested here without playing games manipulating the ep after construction. * ParticipationRegistry - StateProof loading methods (#3261) ## Summary Add ParticipationRegistry methods for setting and retrieving state proof keys. Since they aren't in master yet there is a `type StateProofKey []byte` stub which will need to be updated later. ## Test Plan New unit tests. * Op base64 decode (#3220) b64 opcode, tests, and specs * Bump Version, Remove buildnumber.dat and genesistimestamp.dat files. * Change golang version to 1.16.11 in go-algorand (#2825) Upgrading to 1.16 to help alleviate issues with working on different go versions, and update to a supported, more secure version. Release notes for Go 1.15 and 1.16: https://tip.golang.org/doc/go1.16 https://tip.golang.org/doc/go1.15 * Compatibility mode for partkeyinfo. (#3291) ## Summary Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running Workarounds: 1) use an older `goal` binary. 2) use `algokey part info --keyfile <file>` ## Test Plan Tested manually: ``` ~$ goal account partkeyinfo -d /tmp/private_network/Node/ Dumping participation key info from /tmp/private_network/Node/... Participation ID: CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA Parent address: DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU Last vote round: 1 Last block proposal round: 2 Effective first round: 1 Effective last round: 3000000 First round: 0 Last round: 3000000 Key dilution: 10000 Selection key: 5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ= Voting key: PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs= ~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c Dumping participation key info from /tmp/private_network/Node/... ------------------------------------------------------------------ File: Wallet2.0.3000000.partkey { "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU", "last": 3000000, "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=", "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=", "voteKD": 10000 } ``` * TestEcdsa: fix flaky "tampering" of public key (#3282) ## Summary This test (TestEcdsa) tests the ecdsa_pk_decompress opcode and intentionally "tampers" with the public key by setting the first byte to zero. Occasionally this test is failing, likely because the first byte was already zero. (The test failures are for the cases where failure is expected, `pass=false`) ## Test Plan Existing test should pass, occasional flakiness should go away. * Move appID tracking into EvalContext, out of LedgerForLogic This change increases the seperation between AVM execution and the ledger being used to lookup resources. Previously, the ledger kept track of the appID being executed, to offer a narrower interface to those resources. But now, with app-to-app calls, the appID being executed must change, and the AVM needs to maintain the current appID. * Stupid linter * Support reference types in `goal app method` (#3275) * Fix method signature parse bug * Support reference types * Review dog fixes * Fix comments * Add a hash prefix for ARCs-related hashes (#3298) ## Summary This is to allow ARCs (github.com/algorandfoundation/ARCs) to have their own hash prefix without risk of collision. ## Test Plan It is purely informational. There is no real code change. * catchup: suspend the catchup session once the agreement service kicks in (#3299) The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on. The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it. In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete. When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup. A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch. However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete. This change fixes the return value from true to false. * Compatibility mode for partkeyinfo. (#3291) ## Summary Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running Workarounds: 1) use an older `goal` binary. 2) use `algokey part info --keyfile <file>` ## Test Plan Tested manually: ``` ~$ goal account partkeyinfo -d /tmp/private_network/Node/ Dumping participation key info from /tmp/private_network/Node/... Participation ID: CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA Parent address: DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU Last vote round: 1 Last block proposal round: 2 Effective first round: 1 Effective last round: 3000000 First round: 0 Last round: 3000000 Key dilution: 10000 Selection key: 5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ= Voting key: PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs= ~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c Dumping participation key info from /tmp/private_network/Node/... ------------------------------------------------------------------ File: Wallet2.0.3000000.partkey { "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU", "last": 3000000, "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=", "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=", "voteKD": 10000 } ``` * catchup: suspend the catchup session once the agreement service kicks in (#3299) The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on. The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it. In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete. When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup. A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch. However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete. This change fixes the return value from true to false. * Bump buildnumber.dat * testing: disable flaky test (#3268) Disable a flaky test, to be re-enabled later with #3267. * enumerate conditions that might cause this fetchAndWrite to return false (#3301) ## Summary The fetchAndWrite function contains some complex logic to ultimately determine if we should continue trying to catch up. The conditions that might cause it to return false should be more explicitly enumerated. ## Test Plan Just comments * Fix unit tests error messages * make sure the block service is not attempting to access the ledger after being stopped. (#3303) ## Summary The block service was attempting to serve block via the http handler even after it has been stopped. This lead to undesired downstream failures in the ledger, which was shutdown as well. ## Test Plan unit test added. * Avoid creating algod process for the sole purpose of retrieving the genesis-id. (#3308) ## Summary Avoid creating algod process for the sole purpose of retrieving the genesis-id. Existing code was calling `algod -G -d <data dir>` in order to obtain the genesis version string. The genesis version string can be easily retrieved by loading the genesis file. ## Test Plan Use existing e2e tests. * documentation: fix algorand specs link (#3309) ## Summary This PR fixes a link in a README. ## Testing I clicked on the new link. * testing: reword partitiontest lint message. (#3297) ## Summary The wording on this was tripping me, maybe I was having an off day. I think it would be slightly easier if the message were to tell exactly what you need to do (and not use the angle brackets). * testing: fix random data race in TestAppAccountDataStorage (#3315) fix random data race in unit test * Allow access to resources created in the same transaction group The method will be reworked, but the tests are correct and want to get them visible to team. * ledger: perform the catchpoint writing outside the trackers lock. (#3311) ## Summary This PR moves the catchpoint file writing to be performed outside of the trackers lock. This resolves the issue where a long catchpoint file writing blocks the agreement from validating and propagating votes. ## Test Plan * [x] Test manually & use existing tests. * [x] Implement a unit test * [x] Deploy a local network where the catchpoint writing takes a long time and verify it doesn't get blocked during catchpoint writing. * Separate tx and key validity for `goal account renewpartkey` (#3286) Always use currentRound+proto.MaxTxnLife as last valid round for the transaction when renewing instead of using the partkey validity period. This fixes #3283 * Add qkniep to THANKS.md (#3320) ## Summary Add qkniep to THANKS.md * Followup to opcode base64_decode (#3288) * alphabet begone in favor of encoding * unit test various padding and whitespace scenarios * padding permutations also fail * "Slicing" --> "Manipulation" * fix the codegen fail? * Documenting padding, whitespace, other character behavior * Add help and fish mode to e2e interactive mode. (#3313) ## Summary Minor improvements to e2e.sh interactive mode: * add to -h output * do not run start stop test in interactive mode * support fish shell ## Test Plan Manual testing: ``` ~$ ./e2e.sh -i ... lots of output removed ... ********** READY ********** The test environment is now set. You can now run tests in another terminal. Configure the environment: set -g VIRTUAL_ENV "/home/will/go/src/github.com/algorand/go-algorand/tmp/out/e2e/130013-1639576513257/ve" set -g PATH "$VIRTUAL_ENV/bin:$PATH" python3 "/home/will/go/src/github.com/algorand/go-algorand/test/scripts"/e2e_client_runner.py "/home/will/go/src/github.com/algorand/go-algorand/test/scripts"/e2e_subs/SCRIPT_FILE_NAME Press enter to shut down the test environment... ``` * Minimum Account Balance in Algod (#3287) * Access to apps created in group Also adds some tests that are currently skipped for testing - access to addresses of newly created apps - use of gaid in inner transactions Both require some work to implement the thing being tested. * Remove tracked created mechanism in favor of examining applydata. * Add convertAddress tool. (#3304) ## Summary New tool: convertAddress I share this tool with someone every few months, putting it in the repo along with some documentation should make it easier to share and encourage people to share it amongst themselves if it's useful. Merge `debug` into `tools` to make it easier to organize these miscellaneous tools. * tealdbg: increase intermediate reading/writing buffers (#3335) ## Summary Some large teal source files cause the tealdbg/cdt session to choke. Upping the buffer size to allow for larger source files. closes #3100 ## Test Plan Run tealdbg with a large source teal file, ensure the source file makes it to cdt without choking. * Adding method pseudo op to readme (#3338) * Allow v6 AVM code to use in-group created asas, apps (& their accts) One exception - apps can not mutate (put or del) keys from the app accounts, because EvalDelta cannot encode such changes. * lint docs * typo * The review dog needs obedience training. * add config.DeadlockDetectionThreshold (#3339) Summary This allows for the deadlock detection threshold to be set by configuration. Test Plan Existing tests should pass. * Use one EvalParams for logic evals, another for apps in dry run We used to use one ep per transaction, shared between sig and and app. But the new model of ep usage is to keep using one while evaluating an entire group. The app ep is now built logic.NewAppEvalParams which, hopefully, will prevent some bugs when we change something in the EvalParams and don't reflect it in what was a "raw" EvalParams construction in debugger and dry run. * Use logic.NewAppEvalParams to decrease copying and bugs in debugger * Simplify use of NewEvalParams. No more nil return when no apps. This way, NewEvalParams can be used for all creations of EvalParams, whether they are intended for logicsig or app use, greatly simplifying the way we make them for use by dry run or debugger (where they serve double duty). * Remove explicit PastSideEffects handling in tealdbg * Fix flaky test in randomized ABI encoding test (#3346) * update abi encoding test random testcase generator, scale down parameters to avoid flaky test * parameterized test script * add notes to explain why flaky test is eliminated * show more information from self-roundtrip testing * fully utilize require, remove fmt * Always create EvalParams to evaluate a transaction group. We used to have an optimization to avoid creating EvalParams unless there was an app call in the transaction group. But the interface to allow transaction processing to communicate changes into the EvalParams is complicated by that (we must only do it if there is one!) This also allows us to use the same construction function for eps created for app and logic evaluation, simplifying dry-run and debugger. The optimization is less needed now anyway: 1) The ep is now shared for the whole group, so it's only one. 2) The ep is smaller now, as we only store nil pointers instead of larger scratch space objects for non-app calls. * Correct mistaken commit * Update `goal app method` handling of args and return values (#3352) * Fix method call arg overflow handling * Only check last log for return value * Address feedback * Add comment explaining ABI return prefix * Support app creation in `goal app method` (#3353) * Support app creation in `goal app method` * Don't use nonprintable tab character * Link to specific gist version * Fix error messages * Rename `methodCreation` to `methodCreatesApp` * Spec improvments * Update license to 2022 (#3360) Update license on all source files to 2022. * Add totals checks into acct updates tests (#3367) ## Summary After #2922 there is some leftover unused code for totals calculations. Turned this code into actual asserts. ## Test Plan This is tests update * More spec improvments, including resource "availability" * Recursively return inner transaction tree * Lint * No need for ConfirmedRound, so don't deref a nil pointer! * license check * Shut up, dawg. Co-authored-by: DevOps Service <devops-service@algorand.com> Co-authored-by: Jason Paulos <jasonpaulos@users.noreply.github.com> Co-authored-by: chris erway <51567+cce@users.noreply.github.com> Co-authored-by: Shant Karakashian <55754073+algonautshant@users.noreply.github.com> Co-authored-by: John Lee <64482439+algojohnlee@users.noreply.github.com> Co-authored-by: Will Winder <wwinder.unh@gmail.com> Co-authored-by: Ben Guidarelli <ben.guidarelli@gmail.com> Co-authored-by: Pavel Zbitskiy <65323360+algorandskiy@users.noreply.github.com> Co-authored-by: Jack <87339414+algojack@users.noreply.github.com> Co-authored-by: John Lee <john.lee@algorand.com> Co-authored-by: algobarb <78746954+algobarb@users.noreply.github.com> Co-authored-by: Zeph Grunschlag <tzaffi@users.noreply.github.com> Co-authored-by: Fabrice Benhamouda <fabrice.benhamouda@normalesup.org> Co-authored-by: Tsachi Herman <tsachi.herman@algorand.com> Co-authored-by: Tolik Zinovyev <tolik@algorand.com> Co-authored-by: egieseke <eric_gieseke@yahoo.com> Co-authored-by: Quentin Kniep <kniepque@hu-berlin.de> Co-authored-by: Hang Su <87964331+ahangsu@users.noreply.github.com> Co-authored-by: Or Aharonee <or.aharonee@algorand.com> * Feature/contract to contract (#3389) * Update the Version, BuildNumber, genesistimestamp.data * Three new globals for to help contract-to-contract usability * detritis * Check error * doc comments * Support transaction arguments for `goal app method` (#3233) * Implement transactions as arguments * Fix indexing and dryrun issue * Add docstring * Satisfy review dog * Fix pointer issue * Fix group command * Rename e2e test * Fix filename variable * Add e2e test * Use tab * CI: use libboost-math-dev instead of libboost-all-dev (#3223) ## Summary Small change: libboost-math-dev requires just 4 packages to install, while libboost-all-dev requires > 100. Only Debian/Ubuntu distributions provide fine-grained boost packages like this, but should shave a little time off the CI builds. (Our only boost include is boost/math/distributions/binomial.hpp.) ## Test Plan Builds should pass as before. Now that we are no longer using Travis for Linux builds, the side effect of libboost-all-dev installing make and other missing build tools on Travis encountered in #2717 is no longer a concern. * testing: fixes to rest-participation-key e2e test (#3238) ## Summary - Test to make sure RES has the right input before counting line numbers for result size. - Rest RES to empty so that the same output is not recycled in case of an error. - exit 1 in case of an error - Reduce LAST_ROUND from 1200000 to 120 - "Get List of Keys" before getting NUM_IDS_3 otherwise it will recycle old RES value. * testing: interactive mode for e2e testing (#3227) ## Summary Some e2e tests require a python environment for testing. Unfortunately, setting up that environment adequately similar to the testing environment may not be trivial. This change introduces an interactive mode to the e2e.sh script which stops at the point of running the tests, and allows the user to run the tests from the same testing environment. ## Test Plan No tests needed. Tested the script locally. * Make dev-mode tests less flaky. (#3252) ## Summary Fix a couple flaws in the new go-e2e tests built ontop of DevMode: * Shutdown the fixture when finished. * Don't run in parallel. * Longer delays / better algorithms to wait for data flushing to complete. * Check for "out of order" keys. ## Test Plan N/A, this is a test. * adding libtool to ubuntu deps (#3251) ## Summary The sandbox is not building with dev config using master branch https://github.com/algorand/sandbox/issues/85, complains about libtool not being installed Guessing from this change https://github.com/algorand/go-algorand/pull/3223 Adding libtool to UBUNTU_DEPS in install scripts ## Test Plan Set config in sandbox to my branch `sandbox up dev` It built * Fix error shadowing in Eval (#3258) ## Summary Error from account preloading was shadowed by returning a wrong err variable. This caused subsequent problems in account updates and masked the original failure. ## Test Plan Use existing tests * Disable flaky test. (#3256) ## Summary This test doesn't work properly, disable it until #3255 addresses any underlying problems. * Update the Version, BuildNumber, genesistimestamp.data * Fix a data race in app tests (#3269) ## Summary A test helper function `commitRound` accessed `l.trackers.lastFlushTime` without taking a lock. Fixed. ## Test Plan ``` go test ./ledger -run TestAppEmpty -race -count=50 ok github.com/algorand/go-algorand/ledger 4.078s ``` * Fix e2e.sh mixed indent characters. (#3266) ## Summary Fix e2e.sh mixed indent characters. * Fix ParticipationKeyResponse type. (#3264) ## Summary Fix a small type discrepancy in the OpenAPI spec ahead of some other work that's about to happen. * disable parallelism for e2e-go tests (#3242) ## Summary This sets `-p 1` for the e2e-go tests, intended to make them more deterministic when running on a VM with relatively constrained resources. Since each e2e-go test might spin up a few nodes, it seems like it would help to avoid resource contention. ## Test Plan Tests should run as before. Desired effect can be verified by looking at the test output where the value of PARALLEL_FLAG is printed out before tests are run. * Updating Readme.md with circleci status badges (#3245) * Fix formatting for CircleCI badges (#3272) * Add Custom Scenario for Performance Testing (#3278) Add Custom Scenario for Performance Testing. Add README on how to run custom scenario and modify create_and_deploy_recipe.sh to accept a network template that will generate a new recipe. * Impose limits on the entire "tree" of inner calls. This also increases the realism of testing of multiple app calls in a group by creating the EvalParams with the real constructor, thus getting the pooling stuff tested here without playing games manipulating the ep after construction. * ParticipationRegistry - StateProof loading methods (#3261) ## Summary Add ParticipationRegistry methods for setting and retrieving state proof keys. Since they aren't in master yet there is a `type StateProofKey []byte` stub which will need to be updated later. ## Test Plan New unit tests. * Op base64 decode (#3220) b64 opcode, tests, and specs * Bump Version, Remove buildnumber.dat and genesistimestamp.dat files. * Change golang version to 1.16.11 in go-algorand (#2825) Upgrading to 1.16 to help alleviate issues with working on different go versions, and update to a supported, more secure version. Release notes for Go 1.15 and 1.16: https://tip.golang.org/doc/go1.16 https://tip.golang.org/doc/go1.15 * Compatibility mode for partkeyinfo. (#3291) ## Summary Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running Workarounds: 1) use an older `goal` binary. 2) use `algokey part info --keyfile <file>` ## Test Plan Tested manually: ``` ~$ goal account partkeyinfo -d /tmp/private_network/Node/ Dumping participation key info from /tmp/private_network/Node/... Participation ID: CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA Parent address: DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU Last vote round: 1 Last block proposal round: 2 Effective first round: 1 Effective last round: 3000000 First round: 0 Last round: 3000000 Key dilution: 10000 Selection key: 5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ= Voting key: PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs= ~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c Dumping participation key info from /tmp/private_network/Node/... ------------------------------------------------------------------ File: Wallet2.0.3000000.partkey { "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU", "last": 3000000, "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=", "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=", "voteKD": 10000 } ``` * TestEcdsa: fix flaky "tampering" of public key (#3282) ## Summary This test (TestEcdsa) tests the ecdsa_pk_decompress opcode and intentionally "tampers" with the public key by setting the first byte to zero. Occasionally this test is failing, likely because the first byte was already zero. (The test failures are for the cases where failure is expected, `pass=false`) ## Test Plan Existing test should pass, occasional flakiness should go away. * Move appID tracking into EvalContext, out of LedgerForLogic This change increases the seperation between AVM execution and the ledger being used to lookup resources. Previously, the ledger kept track of the appID being executed, to offer a narrower interface to those resources. But now, with app-to-app calls, the appID being executed must change, and the AVM needs to maintain the current appID. * Stupid linter * Support reference types in `goal app method` (#3275) * Fix method signature parse bug * Support reference types * Review dog fixes * Fix comments * Add a hash prefix for ARCs-related hashes (#3298) ## Summary This is to allow ARCs (github.com/algorandfoundation/ARCs) to have their own hash prefix without risk of collision. ## Test Plan It is purely informational. There is no real code change. * catchup: suspend the catchup session once the agreement service kicks in (#3299) The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on. The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it. In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete. When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup. A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch. However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete. This change fixes the return value from true to false. * Compatibility mode for partkeyinfo. (#3291) ## Summary Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running Workarounds: 1) use an older `goal` binary. 2) use `algokey part info --keyfile <file>` ## Test Plan Tested manually: ``` ~$ goal account partkeyinfo -d /tmp/private_network/Node/ Dumping participation key info from /tmp/private_network/Node/... Participation ID: CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA Parent address: DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU Last vote round: 1 Last block proposal round: 2 Effective first round: 1 Effective last round: 3000000 First round: 0 Last round: 3000000 Key dilution: 10000 Selection key: 5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ= Voting key: PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs= ~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c Dumping participation key info from /tmp/private_network/Node/... ------------------------------------------------------------------ File: Wallet2.0.3000000.partkey { "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU", "last": 3000000, "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=", "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=", "voteKD": 10000 } ``` * catchup: suspend the catchup session once the agreement service kicks in (#3299) The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on. The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it. In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete. When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup. A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch. However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete. This change fixes the return value from true to false. * Bump buildnumber.dat * testing: disable flaky test (#3268) Disable a flaky test, to be re-enabled later with #3267. * enumerate conditions that might cause this fetchAndWrite to return false (#3301) ## Summary The fetchAndWrite function contains some complex logic to ultimately determine if we should continue trying to catch up. The conditions that might cause it to return false should be more explicitly enumerated. ## Test Plan Just comments * Fix unit tests error messages * make sure the block service is not attempting to access the ledger after being stopped. (#3303) ## Summary The block service was attempting to serve block via the http handler even after it has been stopped. This lead to undesired downstream failures in the ledger, which was shutdown as well. ## Test Plan unit test added. * Avoid creating algod process for the sole purpose of retrieving the genesis-id. (#3308) ## Summary Avoid creating algod process for the sole purpose of retrieving the genesis-id. Existing code was calling `algod -G -d <data dir>` in order to obtain the genesis version string. The genesis version string can be easily retrieved by loading the genesis file. ## Test Plan Use existing e2e tests. * documentation: fix algorand specs link (#3309) ## Summary This PR fixes a link in a README. ## Testing I clicked on the new link. * testing: reword partitiontest lint message. (#3297) ## Summary The wording on this was tripping me, maybe I was having an off day. I think it would be slightly easier if the message were to tell exactly what you need to do (and not use the angle brackets). * testing: fix random data race in TestAppAccountDataStorage (#3315) fix random data race in unit test * Allow access to resources created in the same transaction group The method will be reworked, but the tests are correct and want to get them visible to team. * ledger: perform the catchpoint writing outside the trackers lock. (#3311) ## Summary This PR moves the catchpoint file writing to be performed outside of the trackers lock. This resolves the issue where a long catchpoint file writing blocks the agreement from validating and propagating votes. ## Test Plan * [x] Test manually & use existing tests. * [x] Implement a unit test * [x] Deploy a local network where the catchpoint writing takes a long time and verify it doesn't get blocked during catchpoint writing. * Separate tx and key validity for `goal account renewpartkey` (#3286) Always use currentRound+proto.MaxTxnLife as last valid round for the transaction when renewing instead of using the partkey validity period. This fixes #3283 * Add qkniep to THANKS.md (#3320) ## Summary Add qkniep to THANKS.md * Followup to opcode base64_decode (#3288) * alphabet begone in favor of encoding * unit test various padding and whitespace scenarios * padding permutations also fail * "Slicing" --> "Manipulation" * fix the codegen fail? * Documenting padding, whitespace, other character behavior * Add help and fish mode to e2e interactive mode. (#3313) ## Summary Minor improvements to e2e.sh interactive mode: * add to -h output * do not run start stop test in interactive mode * support fish shell ## Test Plan Manual testing: ``` ~$ ./e2e.sh -i ... lots of output removed ... ********** READY ********** The test environment is now set. You can now run tests in another terminal. Configure the environment: set -g VIRTUAL_ENV "/home/will/go/src/github.com/algorand/go-algorand/tmp/out/e2e/130013-1639576513257/ve" set -g PATH "$VIRTUAL_ENV/bin:$PATH" python3 "/home/will/go/src/github.com/algorand/go-algorand/test/scripts"/e2e_client_runner.py "/home/will/go/src/github.com/algorand/go-algorand/test/scripts"/e2e_subs/SCRIPT_FILE_NAME Press enter to shut down the test environment... ``` * Minimum Account Balance in Algod (#3287) * Access to apps created in group Also adds some tests that are currently skipped for testing - access to addresses of newly created apps - use of gaid in inner transactions Both require some work to implement the thing being tested. * Remove tracked created mechanism in favor of examining applydata. * Add convertAddress tool. (#3304) ## Summary New tool: convertAddress I share this tool with someone every few months, putting it in the repo along with some documentation should make it easier to share and encourage people to share it amongst themselves if it's useful. Merge `debug` into `tools` to make it easier to organize these miscellaneous tools. * tealdbg: increase intermediate reading/writing buffers (#3335) ## Summary Some large teal source files cause the tealdbg/cdt session to choke. Upping the buffer size to allow for larger source files. closes #3100 ## Test Plan Run tealdbg with a large source teal file, ensure the source file makes it to cdt without choking. * Adding method pseudo op to readme (#3338) * Allow v6 AVM code to use in-group created asas, apps (& their accts) One exception - apps can not mutate (put or del) keys from the app accounts, because EvalDelta cannot encode such changes. * lint docs * typo * The review dog needs obedience training. * add config.DeadlockDetectionThreshold (#3339) Summary This allows for the deadlock detection threshold to be set by configuration. Test Plan Existing tests should pass. * Use one EvalParams for logic evals, another for apps in dry run We used to use one ep per transaction, shared between sig and and app. But the new model of ep usage is to keep using one while evaluating an entire group. The app ep is now built logic.NewAppEvalParams which, hopefully, will prevent some bugs when we change something in the EvalParams and don't reflect it in what was a "raw" EvalParams construction in debugger and dry run. * Use logic.NewAppEvalParams to decrease copying and bugs in debugger * Simplify use of NewEvalParams. No more nil return when no apps. This way, NewEvalParams can be used for all creations of EvalParams, whether they are intended for logicsig or app use, greatly simplifying the way we make them for use by dry run or debugger (where they serve double duty). * Remove explicit PastSideEffects handling in tealdbg * Fix flaky test in randomized ABI encoding test (#3346) * update abi encoding test random testcase generator, scale down parameters to avoid flaky test * parameterized test script * add notes to explain why flaky test is eliminated * show more information from self-roundtrip testing * fully utilize require, remove fmt * Always create EvalParams to evaluate a transaction group. We used to have an optimization to avoid creating EvalParams unless there was an app call in the transaction group. But the interface to allow transaction processing to communicate changes into the EvalParams is complicated by that (we must only do it if there is one!) This also allows us to use the same construction function for eps created for app and logic evaluation, simplifying dry-run and debugger. The optimization is less needed now anyway: 1) The ep is now shared for the whole group, so it's only one. 2) The ep is smaller now, as we only store nil pointers instead of larger scratch space objects for non-app calls. * Correct mistaken commit * ledger: perform the catchpoint writing outside the trackers lock. (#3311) ## Summary This PR moves the catchpoint file writing to be performed outside of the trackers lock. This resolves the issue where a long catchpoint file writing blocks the agreement from validating and propagating votes. ## Test Plan * [x] Test manually & use existing tests. * [x] Implement a unit test * [x] Deploy a local network where the catchpoint writing takes a long time and verify it doesn't get blocked during catchpoint writing. * Bump version number * Update `goal app method` handling of args and return values (#3352) * Fix method call arg overflow handling * Only check last log for return value * Address feedback * Add comment explaining ABI return prefix * Support app creation in `goal app method` (#3353) * Support app creation in `goal app method` * Don't use nonprintable tab character * Link to specific gist version * Fix error messages * Rename `methodCreation` to `methodCreatesApp` * Spec improvments * Update license to 2022 (#3360) Update license on all source files to 2022. * Add totals checks into acct updates tests (#3367) ## Summary After #2922 there is some leftover unused code for totals calculations. Turned this code into actual asserts. ## Test Plan This is tests update * More spec improvments, including resource "availability" * Recursively return inner transaction tree * Lint * No need for ConfirmedRound, so don't deref a nil pointer! * remove buildnumber.dat * license check * Shut up, dawg. * PKI State Proof Incremental Key Loading (#3281) ## Summary Followup to #3261 (contained in diff). Use the new key loading routine from the REST API. ## Test Plan New unit tests. * Limit number of simultaneous REST connections (#3326) ## Summary This PR limits the number of simultaneous REST connections we process to prevent the exhaustion of resources and ultimately a crash. Two limits are introduced: soft and hard. When the soft limit is exceeded, new connections are returned the 429 Too Many Requests http code. When the hard limit is exceeded, new connections are accepted and immediately closed. Partially resolves https://github.com/algorand/go-algorand-internal/issues/1814. ## Test Plan Added unit tests. * Use rejecting limit listener in WebsocketNetwork. (#3380) ## Summary Replace the standard limit listener with the new rejecting limit listener in `WebsocketNetwork`. This will let the dialing node know that connection is impossible faster. ## Test Plan Probably not necessary. * Delete unused constant. (#3379) ## Summary This PR deletes an unused constant. ## Test Plan None. * Add test to exercise lookup corner cases (#3376) ## Summary This test attempts to cover the case when an accountUpdates.lookupX method can't find the requested address, falls through looking at deltas and the LRU accounts cache, then hits the database — only to discover that the round stored in the database (committed in `accountUpdates.commitRound`) is out of sync with `accountUpdates.cachedDBRound` (updated a little bit later in `accountUpdates.postCommit`). In this case, the lookup method waits and tries again, iterating the `for { }` it is in. We did not have coverage for this code path before. ## Test Plan Adds new test. * Test for catchup stop on completion (#3306) Adding a test for the fix in #3299 ## Test Plan This is a test * Delete unused AtomicCommitWriteLock(). (#3383) ## Summary This PR deletes unused `AtomicCommitWriteLock()` and simplifies code. ## Test Plan None. Co-authored-by: DevOps Service <devops-service@algorand.com> Co-authored-by: Jason Paulos <jasonpaulos@users.noreply.github.com> Co-authored-by: chris erway <51567+cce@users.noreply.github.com> Co-authored-by: Shant Karakashian <55754073+algonautshant@users.noreply.github.com> Co-authored-by: John Lee <64482439+algojohnlee@users.noreply.github.com> Co-authored-by: Will Winder <wwinder.unh@gmail.com> Co-authored-by: Ben Guidarelli <ben.guidarelli@gmail.com> Co-authored-by: Pavel Zbitskiy <65323360+algorandskiy@users.noreply.github.com> Co-authored-by: Jack <87339414+algojack@users.noreply.github.com> Co-authored-by: John Lee <john.lee@algorand.com> Co-authored-by: algobarb <78746954+algobarb@users.noreply.github.com> Co-authored-by: Zeph Grunschlag <tzaffi@users.noreply.github.com> Co-authored-by: Fabrice Benhamouda <fabrice.benhamouda@normalesup.org> Co-authored-by: Tsachi Herman <tsachi.herman@algorand.com> Co-authored-by: Tolik Zinovyev <tolik@algorand.com> Co-authored-by: egieseke <eric_gieseke@yahoo.com> Co-authored-by: Quentin Kniep <kniepque@hu-berlin.de> Co-authored-by: Hang Su <87964331+ahangsu@users.noreply.github.com> Co-authored-by: Or Aharonee <or.aharonee@algorand.com> Co-authored-by: Barbara Poon <barbara.poon@algorand.com> * Feature/contract to contract (#3390) * Update the Version, BuildNumber, genesistimestamp.data * Thr…

* The new inner appl fields * Unit tests for field setting on appls * Construct EvalDelta in AVM rather than by inspecting ledger * Obey the linter! * more LedgerForEvaluation accomodation * Test inner evaldeltas * Checks on calling old AVM apps, or re-entrancy * Allow opcode budget to be added to by executing inner apps. * TxID and GroupID for inner transactions * gitxn/gitxna * Lint, spec generate * txn simplifications * Encode "arrayness" in the txn field spec * Pavel's CR comments * Update tests to distinguish assembly / eval errors * Test itxn_field assembly separate from eval * factor out the array index parsing of all the txn assembly * Consistent errors and parsign for many opcodes * Cleanup immediate parsing, prep txn effects testing * EvalParams is now a single object used to evaluate each txn in turn. * Simplifications for the Dawg (the Review Dog) * Use a copy for the EvalParams.TxnGroup * Set the logicsig on txns in the GroupContext, so check() can see it * Update test for explicit empty check * Three new globals for to help contract-to-contract usability (#3237) * Three new globals for to help contract-to-contract usability * detritis * Check error * doc comments * Gloadss (#3248) * Three new globals for to help contract-to-contract usability * detritis * Check error * doc comments * opcode, docs, and tests * specs update * Feature/contract to contract (#3285) * Update the Version, BuildNumber, genesistimestamp.data * Three new globals for to help contract-to-contract usability * detritis * Check error * doc comments * Support transaction arguments for `goal app method` (#3233) * Implement transactions as arguments * Fix indexing and dryrun issue * Add docstring * Satisfy review dog * Fix pointer issue * Fix group command * Rename e2e test * Fix filename variable * Add e2e test * Use tab * CI: use libboost-math-dev instead of libboost-all-dev (#3223) ## Summary Small change: libboost-math-dev requires just 4 packages to install, while libboost-all-dev requires > 100. Only Debian/Ubuntu distributions provide fine-grained boost packages like this, but should shave a little time off the CI builds. (Our only boost include is boost/math/distributions/binomial.hpp.) ## Test Plan Builds should pass as before. Now that we are no longer using Travis for Linux builds, the side effect of libboost-all-dev installing make and other missing build tools on Travis encountered in #2717 is no longer a concern. * testing: fixes to rest-participation-key e2e test (#3238) ## Summary - Test to make sure RES has the right input before counting line numbers for result size. - Rest RES to empty so that the same output is not recycled in case of an error. - exit 1 in case of an error - Reduce LAST_ROUND from 1200000 to 120 - "Get List of Keys" before getting NUM_IDS_3 otherwise it will recycle old RES value. * testing: interactive mode for e2e testing (#3227) ## Summary Some e2e tests require a python environment for testing. Unfortunately, setting up that environment adequately similar to the testing environment may not be trivial. This change introduces an interactive mode to the e2e.sh script which stops at the point of running the tests, and allows the user to run the tests from the same testing environment. ## Test Plan No tests needed. Tested the script locally. * Make dev-mode tests less flaky. (#3252) ## Summary Fix a couple flaws in the new go-e2e tests built ontop of DevMode: * Shutdown the fixture when finished. * Don't run in parallel. * Longer delays / better algorithms to wait for data flushing to complete. * Check for "out of order" keys. ## Test Plan N/A, this is a test. * adding libtool to ubuntu deps (#3251) ## Summary The sandbox is not building with dev config using master branch https://github.com/algorand/sandbox/issues/85, complains about libtool not being installed Guessing from this change https://github.com/algorand/go-algorand/pull/3223 Adding libtool to UBUNTU_DEPS in install scripts ## Test Plan Set config in sandbox to my branch `sandbox up dev` It built * Fix error shadowing in Eval (#3258) ## Summary Error from account preloading was shadowed by returning a wrong err variable. This caused subsequent problems in account updates and masked the original failure. ## Test Plan Use existing tests * Disable flaky test. (#3256) ## Summary This test doesn't work properly, disable it until #3255 addresses any underlying problems. * Update the Version, BuildNumber, genesistimestamp.data * Fix a data race in app tests (#3269) ## Summary A test helper function `commitRound` accessed `l.trackers.lastFlushTime` without taking a lock. Fixed. ## Test Plan ``` go test ./ledger -run TestAppEmpty -race -count=50 ok github.com/algorand/go-algorand/ledger 4.078s ``` * Fix e2e.sh mixed indent characters. (#3266) ## Summary Fix e2e.sh mixed indent characters. * Fix ParticipationKeyResponse type. (#3264) ## Summary Fix a small type discrepancy in the OpenAPI spec ahead of some other work that's about to happen. * disable parallelism for e2e-go tests (#3242) ## Summary This sets `-p 1` for the e2e-go tests, intended to make them more deterministic when running on a VM with relatively constrained resources. Since each e2e-go test might spin up a few nodes, it seems like it would help to avoid resource contention. ## Test Plan Tests should run as before. Desired effect can be verified by looking at the test output where the value of PARALLEL_FLAG is printed out before tests are run. * Updating Readme.md with circleci status badges (#3245) * Fix formatting for CircleCI badges (#3272) * Add Custom Scenario for Performance Testing (#3278) Add Custom Scenario for Performance Testing. Add README on how to run custom scenario and modify create_and_deploy_recipe.sh to accept a network template that will generate a new recipe. * Impose limits on the entire "tree" of inner calls. This also increases the realism of testing of multiple app calls in a group by creating the EvalParams with the real constructor, thus getting the pooling stuff tested here without playing games manipulating the ep after construction. * ParticipationRegistry - StateProof loading methods (#3261) ## Summary Add ParticipationRegistry methods for setting and retrieving state proof keys. Since they aren't in master yet there is a `type StateProofKey []byte` stub which will need to be updated later. ## Test Plan New unit tests. * Op base64 decode (#3220) b64 opcode, tests, and specs * Bump Version, Remove buildnumber.dat and genesistimestamp.dat files. * Change golang version to 1.16.11 in go-algorand (#2825) Upgrading to 1.16 to help alleviate issues with working on different go versions, and update to a supported, more secure version. Release notes for Go 1.15 and 1.16: https://tip.golang.org/doc/go1.16 https://tip.golang.org/doc/go1.15 * Compatibility mode for partkeyinfo. (#3291) ## Summary Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running Workarounds: 1) use an older `goal` binary. 2) use `algokey part info --keyfile <file>` ## Test Plan Tested manually: ``` ~$ goal account partkeyinfo -d /tmp/private_network/Node/ Dumping participation key info from /tmp/private_network/Node/... Participation ID: CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA Parent address: DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU Last vote round: 1 Last block proposal round: 2 Effective first round: 1 Effective last round: 3000000 First round: 0 Last round: 3000000 Key dilution: 10000 Selection key: 5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ= Voting key: PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs= ~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c Dumping participation key info from /tmp/private_network/Node/... ------------------------------------------------------------------ File: Wallet2.0.3000000.partkey { "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU", "last": 3000000, "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=", "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=", "voteKD": 10000 } ``` * TestEcdsa: fix flaky "tampering" of public key (#3282) ## Summary This test (TestEcdsa) tests the ecdsa_pk_decompress opcode and intentionally "tampers" with the public key by setting the first byte to zero. Occasionally this test is failing, likely because the first byte was already zero. (The test failures are for the cases where failure is expected, `pass=false`) ## Test Plan Existing test should pass, occasional flakiness should go away. * Move appID tracking into EvalContext, out of LedgerForLogic This change increases the seperation between AVM execution and the ledger being used to lookup resources. Previously, the ledger kept track of the appID being executed, to offer a narrower interface to those resources. But now, with app-to-app calls, the appID being executed must change, and the AVM needs to maintain the current appID. * Stupid linter * Support reference types in `goal app method` (#3275) * Fix method signature parse bug * Support reference types * Review dog fixes * Fix comments * Add a hash prefix for ARCs-related hashes (#3298) ## Summary This is to allow ARCs (github.com/algorandfoundation/ARCs) to have their own hash prefix without risk of collision. ## Test Plan It is purely informational. There is no real code change. * catchup: suspend the catchup session once the agreement service kicks in (#3299) The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on. The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it. In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete. When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup. A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch. However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete. This change fixes the return value from true to false. * Compatibility mode for partkeyinfo. (#3291) ## Summary Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running Workarounds: 1) use an older `goal` binary. 2) use `algokey part info --keyfile <file>` ## Test Plan Tested manually: ``` ~$ goal account partkeyinfo -d /tmp/private_network/Node/ Dumping participation key info from /tmp/private_network/Node/... Participation ID: CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA Parent address: DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU Last vote round: 1 Last block proposal round: 2 Effective first round: 1 Effective last round: 3000000 First round: 0 Last round: 3000000 Key dilution: 10000 Selection key: 5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ= Voting key: PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs= ~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c Dumping participation key info from /tmp/private_network/Node/... ------------------------------------------------------------------ File: Wallet2.0.3000000.partkey { "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU", "last": 3000000, "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=", "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=", "voteKD": 10000 } ``` * catchup: suspend the catchup session once the agreement service kicks in (#3299) The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on. The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it. In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete. When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup. A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch. However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete. This change fixes the return value from true to false. * Bump buildnumber.dat * testing: disable flaky test (#3268) Disable a flaky test, to be re-enabled later with #3267. * enumerate conditions that might cause this fetchAndWrite to return false (#3301) ## Summary The fetchAndWrite function contains some complex logic to ultimately determine if we should continue trying to catch up. The conditions that might cause it to return false should be more explicitly enumerated. ## Test Plan Just comments * Fix unit tests error messages * make sure the block service is not attempting to access the ledger after being stopped. (#3303) ## Summary The block service was attempting to serve block via the http handler even after it has been stopped. This lead to undesired downstream failures in the ledger, which was shutdown as well. ## Test Plan unit test added. * Avoid creating algod process for the sole purpose of retrieving the genesis-id. (#3308) ## Summary Avoid creating algod process for the sole purpose of retrieving the genesis-id. Existing code was calling `algod -G -d <data dir>` in order to obtain the genesis version string. The genesis version string can be easily retrieved by loading the genesis file. ## Test Plan Use existing e2e tests. * documentation: fix algorand specs link (#3309) ## Summary This PR fixes a link in a README. ## Testing I clicked on the new link. * testing: reword partitiontest lint message. (#3297) ## Summary The wording on this was tripping me, maybe I was having an off day. I think it would be slightly easier if the message were to tell exactly what you need to do (and not use the angle brackets). * testing: fix random data race in TestAppAccountDataStorage (#3315) fix random data race in unit test * Allow access to resources created in the same transaction group The method will be reworked, but the tests are correct and want to get them visible to team. Co-authored-by: DevOps Service <devops-service@algorand.com> Co-authored-by: Jason Paulos <jasonpaulos@users.noreply.github.com> Co-authored-by: chris erway <51567+cce@users.noreply.github.com> Co-authored-by: Shant Karakashian <55754073+algonautshant@users.noreply.github.com> Co-authored-by: John Lee <64482439+algojohnlee@users.noreply.github.com> Co-authored-by: Will Winder <wwinder.unh@gmail.com> Co-authored-by: Ben Guidarelli <ben.guidarelli@gmail.com> Co-authored-by: Pavel Zbitskiy <65323360+algorandskiy@users.noreply.github.com> Co-authored-by: Jack <87339414+algojack@users.noreply.github.com> Co-authored-by: John Lee <john.lee@algorand.com> Co-authored-by: algobarb <78746954+algobarb@users.noreply.github.com> Co-authored-by: Zeph Grunschlag <tzaffi@users.noreply.github.com> Co-authored-by: Fabrice Benhamouda <fabrice.benhamouda@normalesup.org> Co-authored-by: Tsachi Herman <tsachi.herman@algorand.com> Co-authored-by: Tolik Zinovyev <tolik@algorand.com> Co-authored-by: egieseke <eric_gieseke@yahoo.com> * add access to resources created in the current group (#3340) * Feature/contract to contract (#3357) * Update the Version, BuildNumber, genesistimestamp.data * Three new globals for to help contract-to-contract usability * detritis * Check error * doc comments * Support transaction arguments for `goal app method` (#3233) * Implement transactions as arguments * Fix indexing and dryrun issue * Add docstring * Satisfy review dog * Fix pointer issue * Fix group command * Rename e2e test * Fix filename variable * Add e2e test * Use tab * CI: use libboost-math-dev instead of libboost-all-dev (#3223) ## Summary Small change: libboost-math-dev requires just 4 packages to install, while libboost-all-dev requires > 100. Only Debian/Ubuntu distributions provide fine-grained boost packages like this, but should shave a little time off the CI builds. (Our only boost include is boost/math/distributions/binomial.hpp.) ## Test Plan Builds should pass as before. Now that we are no longer using Travis for Linux builds, the side effect of libboost-all-dev installing make and other missing build tools on Travis encountered in #2717 is no longer a concern. * testing: fixes to rest-participation-key e2e test (#3238) ## Summary - Test to make sure RES has the right input before counting line numbers for result size. - Rest RES to empty so that the same output is not recycled in case of an error. - exit 1 in case of an error - Reduce LAST_ROUND from 1200000 to 120 - "Get List of Keys" before getting NUM_IDS_3 otherwise it will recycle old RES value. * testing: interactive mode for e2e testing (#3227) ## Summary Some e2e tests require a python environment for testing. Unfortunately, setting up that environment adequately similar to the testing environment may not be trivial. This change introduces an interactive mode to the e2e.sh script which stops at the point of running the tests, and allows the user to run the tests from the same testing environment. ## Test Plan No tests needed. Tested the script locally. * Make dev-mode tests less flaky. (#3252) ## Summary Fix a couple flaws in the new go-e2e tests built ontop of DevMode: * Shutdown the fixture when finished. * Don't run in parallel. * Longer delays / better algorithms to wait for data flushing to complete. * Check for "out of order" keys. ## Test Plan N/A, this is a test. * adding libtool to ubuntu deps (#3251) ## Summary The sandbox is not building with dev config using master branch https://github.com/algorand/sandbox/issues/85, complains about libtool not being installed Guessing from this change https://github.com/algorand/go-algorand/pull/3223 Adding libtool to UBUNTU_DEPS in install scripts ## Test Plan Set config in sandbox to my branch `sandbox up dev` It built * Fix error shadowing in Eval (#3258) ## Summary Error from account preloading was shadowed by returning a wrong err variable. This caused subsequent problems in account updates and masked the original failure. ## Test Plan Use existing tests * Disable flaky test. (#3256) ## Summary This test doesn't work properly, disable it until #3255 addresses any underlying problems. * Update the Version, BuildNumber, genesistimestamp.data * Fix a data race in app tests (#3269) ## Summary A test helper function `commitRound` accessed `l.trackers.lastFlushTime` without taking a lock. Fixed. ## Test Plan ``` go test ./ledger -run TestAppEmpty -race -count=50 ok github.com/algorand/go-algorand/ledger 4.078s ``` * Fix e2e.sh mixed indent characters. (#3266) ## Summary Fix e2e.sh mixed indent characters. * Fix ParticipationKeyResponse type. (#3264) ## Summary Fix a small type discrepancy in the OpenAPI spec ahead of some other work that's about to happen. * disable parallelism for e2e-go tests (#3242) ## Summary This sets `-p 1` for the e2e-go tests, intended to make them more deterministic when running on a VM with relatively constrained resources. Since each e2e-go test might spin up a few nodes, it seems like it would help to avoid resource contention. ## Test Plan Tests should run as before. Desired effect can be verified by looking at the test output where the value of PARALLEL_FLAG is printed out before tests are run. * Updating Readme.md with circleci status badges (#3245) * Fix formatting for CircleCI badges (#3272) * Add Custom Scenario for Performance Testing (#3278) Add Custom Scenario for Performance Testing. Add README on how to run custom scenario and modify create_and_deploy_recipe.sh to accept a network template that will generate a new recipe. * Impose limits on the entire "tree" of inner calls. This also increases the realism of testing of multiple app calls in a group by creating the EvalParams with the real constructor, thus getting the pooling stuff tested here without playing games manipulating the ep after construction. * ParticipationRegistry - StateProof loading methods (#3261) ## Summary Add ParticipationRegistry methods for setting and retrieving state proof keys. Since they aren't in master yet there is a `type StateProofKey []byte` stub which will need to be updated later. ## Test Plan New unit tests. * Op base64 decode (#3220) b64 opcode, tests, and specs * Bump Version, Remove buildnumber.dat and genesistimestamp.dat files. * Change golang version to 1.16.11 in go-algorand (#2825) Upgrading to 1.16 to help alleviate issues with working on different go versions, and update to a supported, more secure version. Release notes for Go 1.15 and 1.16: https://tip.golang.org/doc/go1.16 https://tip.golang.org/doc/go1.15 * Compatibility mode for partkeyinfo. (#3291) ## Summary Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running Workarounds: 1) use an older `goal` binary. 2) use `algokey part info --keyfile <file>` ## Test Plan Tested manually: ``` ~$ goal account partkeyinfo -d /tmp/private_network/Node/ Dumping participation key info from /tmp/private_network/Node/... Participation ID: CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA Parent address: DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU Last vote round: 1 Last block proposal round: 2 Effective first round: 1 Effective last round: 3000000 First round: 0 Last round: 3000000 Key dilution: 10000 Selection key: 5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ= Voting key: PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs= ~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c Dumping participation key info from /tmp/private_network/Node/... ------------------------------------------------------------------ File: Wallet2.0.3000000.partkey { "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU", "last": 3000000, "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=", "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=", "voteKD": 10000 } ``` * TestEcdsa: fix flaky "tampering" of public key (#3282) ## Summary This test (TestEcdsa) tests the ecdsa_pk_decompress opcode and intentionally "tampers" with the public key by setting the first byte to zero. Occasionally this test is failing, likely because the first byte was already zero. (The test failures are for the cases where failure is expected, `pass=false`) ## Test Plan Existing test should pass, occasional flakiness should go away. * Move appID tracking into EvalContext, out of LedgerForLogic This change increases the seperation between AVM execution and the ledger being used to lookup resources. Previously, the ledger kept track of the appID being executed, to offer a narrower interface to those resources. But now, with app-to-app calls, the appID being executed must change, and the AVM needs to maintain the current appID. * Stupid linter * Support reference types in `goal app method` (#3275) * Fix method signature parse bug * Support reference types * Review dog fixes * Fix comments * Add a hash prefix for ARCs-related hashes (#3298) ## Summary This is to allow ARCs (github.com/algorandfoundation/ARCs) to have their own hash prefix without risk of collision. ## Test Plan It is purely informational. There is no real code change. * catchup: suspend the catchup session once the agreement service kicks in (#3299) The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on. The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it. In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete. When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup. A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch. However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete. This change fixes the return value from true to false. * Compatibility mode for partkeyinfo. (#3291) ## Summary Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running Workarounds: 1) use an older `goal` binary. 2) use `algokey part info --keyfile <file>` ## Test Plan Tested manually: ``` ~$ goal account partkeyinfo -d /tmp/private_network/Node/ Dumping participation key info from /tmp/private_network/Node/... Participation ID: CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA Parent address: DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU Last vote round: 1 Last block proposal round: 2 Effective first round: 1 Effective last round: 3000000 First round: 0 Last round: 3000000 Key dilution: 10000 Selection key: 5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ= Voting key: PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs= ~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c Dumping participation key info from /tmp/private_network/Node/... ------------------------------------------------------------------ File: Wallet2.0.3000000.partkey { "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU", "last": 3000000, "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=", "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=", "voteKD": 10000 } ``` * catchup: suspend the catchup session once the agreement service kicks in (#3299) The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on. The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it. In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete. When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup. A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch. However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete. This change fixes the return value from true to false. * Bump buildnumber.dat * testing: disable flaky test (#3268) Disable a flaky test, to be re-enabled later with #3267. * enumerate conditions that might cause this fetchAndWrite to return false (#3301) ## Summary The fetchAndWrite function contains some complex logic to ultimately determine if we should continue trying to catch up. The conditions that might cause it to return false should be more explicitly enumerated. ## Test Plan Just comments * Fix unit tests error messages * make sure the block service is not attempting to access the ledger after being stopped. (#3303) ## Summary The block service was attempting to serve block via the http handler even after it has been stopped. This lead to undesired downstream failures in the ledger, which was shutdown as well. ## Test Plan unit test added. * Avoid creating algod process for the sole purpose of retrieving the genesis-id. (#3308) ## Summary Avoid creating algod process for the sole purpose of retrieving the genesis-id. Existing code was calling `algod -G -d <data dir>` in order to obtain the genesis version string. The genesis version string can be easily retrieved by loading the genesis file. ## Test Plan Use existing e2e tests. * documentation: fix algorand specs link (#3309) ## Summary This PR fixes a link in a README. ## Testing I clicked on the new link. * testing: reword partitiontest lint message. (#3297) ## Summary The wording on this was tripping me, maybe I was having an off day. I think it would be slightly easier if the message were to tell exactly what you need to do (and not use the angle brackets). * testing: fix random data race in TestAppAccountDataStorage (#3315) fix random data race in unit test * Allow access to resources created in the same transaction group The method will be reworked, but the tests are correct and want to get them visible to team. * ledger: perform the catchpoint writing outside the trackers lock. (#3311) ## Summary This PR moves the catchpoint file writing to be performed outside of the trackers lock. This resolves the issue where a long catchpoint file writing blocks the agreement from validating and propagating votes. ## Test Plan * [x] Test manually & use existing tests. * [x] Implement a unit test * [x] Deploy a local network where the catchpoint writing takes a long time and verify it doesn't get blocked during catchpoint writing. * Separate tx and key validity for `goal account renewpartkey` (#3286) Always use currentRound+proto.MaxTxnLife as last valid round for the transaction when renewing instead of using the partkey validity period. This fixes #3283 * Add qkniep to THANKS.md (#3320) ## Summary Add qkniep to THANKS.md * Followup to opcode base64_decode (#3288) * alphabet begone in favor of encoding * unit test various padding and whitespace scenarios * padding permutations also fail * "Slicing" --> "Manipulation" * fix the codegen fail? * Documenting padding, whitespace, other character behavior * Add help and fish mode to e2e interactive mode. (#3313) ## Summary Minor improvements to e2e.sh interactive mode: * add to -h output * do not run start stop test in interactive mode * support fish shell ## Test Plan Manual testing: ``` ~$ ./e2e.sh -i ... lots of output removed ... ********** READY ********** The test environment is now set. You can now run tests in another terminal. Configure the environment: set -g VIRTUAL_ENV "/home/will/go/src/github.com/algorand/go-algorand/tmp/out/e2e/130013-1639576513257/ve" set -g PATH "$VIRTUAL_ENV/bin:$PATH" python3 "/home/will/go/src/github.com/algorand/go-algorand/test/scripts"/e2e_client_runner.py "/home/will/go/src/github.com/algorand/go-algorand/test/scripts"/e2e_subs/SCRIPT_FILE_NAME Press enter to shut down the test environment... ``` * Minimum Account Balance in Algod (#3287) * Access to apps created in group Also adds some tests that are currently skipped for testing - access to addresses of newly created apps - use of gaid in inner transactions Both require some work to implement the thing being tested. * Remove tracked created mechanism in favor of examining applydata. * Add convertAddress tool. (#3304) ## Summary New tool: convertAddress I share this tool with someone every few months, putting it in the repo along with some documentation should make it easier to share and encourage people to share it amongst themselves if it's useful. Merge `debug` into `tools` to make it easier to organize these miscellaneous tools. * tealdbg: increase intermediate reading/writing buffers (#3335) ## Summary Some large teal source files cause the tealdbg/cdt session to choke. Upping the buffer size to allow for larger source files. closes #3100 ## Test Plan Run tealdbg with a large source teal file, ensure the source file makes it to cdt without choking. * Adding method pseudo op to readme (#3338) * Allow v6 AVM code to use in-group created asas, apps (& their accts) One exception - apps can not mutate (put or del) keys from the app accounts, because EvalDelta cannot encode such changes. * lint docs * typo * The review dog needs obedience training. * add config.DeadlockDetectionThreshold (#3339) Summary This allows for the deadlock detection threshold to be set by configuration. Test Plan Existing tests should pass. * Use one EvalParams for logic evals, another for apps in dry run We used to use one ep per transaction, shared between sig and and app. But the new model of ep usage is to keep using one while evaluating an entire group. The app ep is now built logic.NewAppEvalParams which, hopefully, will prevent some bugs when we change something in the EvalParams and don't reflect it in what was a "raw" EvalParams construction in debugger and dry run. * Use logic.NewAppEvalParams to decrease copying and bugs in debugger * Simplify use of NewEvalParams. No more nil return when no apps. This way, NewEvalParams can be used for all creations of EvalParams, whether they are intended for logicsig or app use, greatly simplifying the way we make them for use by dry run or debugger (where they serve double duty). * Remove explicit PastSideEffects handling in tealdbg * Fix flaky test in randomized ABI encoding test (#3346) * update abi encoding test random testcase generator, scale down parameters to avoid flaky test * parameterized test script * add notes to explain why flaky test is eliminated * show more information from self-roundtrip testing * fully utilize require, remove fmt * Always create EvalParams to evaluate a transaction group. We used to have an optimization to avoid creating EvalParams unless there was an app call in the transaction group. But the interface to allow transaction processing to communicate changes into the EvalParams is complicated by that (we must only do it if there is one!) This also allows us to use the same construction function for eps created for app and logic evaluation, simplifying dry-run and debugger. The optimization is less needed now anyway: 1) The ep is now shared for the whole group, so it's only one. 2) The ep is smaller now, as we only store nil pointers instead of larger scratch space objects for non-app calls. * Correct mistaken commit * Update `goal app method` handling of args and return values (#3352) * Fix method call arg overflow handling * Only check last log for return value * Address feedback * Add comment explaining ABI return prefix * Support app creation in `goal app method` (#3353) * Support app creation in `goal app method` * Don't use nonprintable tab character * Link to specific gist version * Fix error messages * Rename `methodCreation` to `methodCreatesApp` * Spec improvments * Update license to 2022 (#3360) Update license on all source files to 2022. * Add totals checks into acct updates tests (#3367) ## Summary After #2922 there is some leftover unused code for totals calculations. Turned this code into actual asserts. ## Test Plan This is tests update * More spec improvments, including resource "availability" * Recursively return inner transaction tree * Lint * No need for ConfirmedRound, so don't deref a nil pointer! * license check * Shut up, dawg. Co-authored-by: DevOps Service <devops-service@algorand.com> Co-authored-by: Jason Paulos <jasonpaulos@users.noreply.github.com> Co-authored-by: chris erway <51567+cce@users.noreply.github.com> Co-authored-by: Shant Karakashian <55754073+algonautshant@users.noreply.github.com> Co-authored-by: John Lee <64482439+algojohnlee@users.noreply.github.com> Co-authored-by: Will Winder <wwinder.unh@gmail.com> Co-authored-by: Ben Guidarelli <ben.guidarelli@gmail.com> Co-authored-by: Pavel Zbitskiy <65323360+algorandskiy@users.noreply.github.com> Co-authored-by: Jack <87339414+algojack@users.noreply.github.com> Co-authored-by: John Lee <john.lee@algorand.com> Co-authored-by: algobarb <78746954+algobarb@users.noreply.github.com> Co-authored-by: Zeph Grunschlag <tzaffi@users.noreply.github.com> Co-authored-by: Fabrice Benhamouda <fabrice.benhamouda@normalesup.org> Co-authored-by: Tsachi Herman <tsachi.herman@algorand.com> Co-authored-by: Tolik Zinovyev <tolik@algorand.com> Co-authored-by: egieseke <eric_gieseke@yahoo.com> Co-authored-by: Quentin Kniep <kniepque@hu-berlin.de> Co-authored-by: Hang Su <87964331+ahangsu@users.noreply.github.com> Co-authored-by: Or Aharonee <or.aharonee@algorand.com> * Feature/contract to contract (#3389) * Update the Version, BuildNumber, genesistimestamp.data * Three new globals for to help contract-to-contract usability * detritis * Check error * doc comments * Support transaction arguments for `goal app method` (#3233) * Implement transactions as arguments * Fix indexing and dryrun issue * Add docstring * Satisfy review dog * Fix pointer issue * Fix group command * Rename e2e test * Fix filename variable * Add e2e test * Use tab * CI: use libboost-math-dev instead of libboost-all-dev (#3223) ## Summary Small change: libboost-math-dev requires just 4 packages to install, while libboost-all-dev requires > 100. Only Debian/Ubuntu distributions provide fine-grained boost packages like this, but should shave a little time off the CI builds. (Our only boost include is boost/math/distributions/binomial.hpp.) ## Test Plan Builds should pass as before. Now that we are no longer using Travis for Linux builds, the side effect of libboost-all-dev installing make and other missing build tools on Travis encountered in #2717 is no longer a concern. * testing: fixes to rest-participation-key e2e test (#3238) ## Summary - Test to make sure RES has the right input before counting line numbers for result size. - Rest RES to empty so that the same output is not recycled in case of an error. - exit 1 in case of an error - Reduce LAST_ROUND from 1200000 to 120 - "Get List of Keys" before getting NUM_IDS_3 otherwise it will recycle old RES value. * testing: interactive mode for e2e testing (#3227) ## Summary Some e2e tests require a python environment for testing. Unfortunately, setting up that environment adequately similar to the testing environment may not be trivial. This change introduces an interactive mode to the e2e.sh script which stops at the point of running the tests, and allows the user to run the tests from the same testing environment. ## Test Plan No tests needed. Tested the script locally. * Make dev-mode tests less flaky. (#3252) ## Summary Fix a couple flaws in the new go-e2e tests built ontop of DevMode: * Shutdown the fixture when finished. * Don't run in parallel. * Longer delays / better algorithms to wait for data flushing to complete. * Check for "out of order" keys. ## Test Plan N/A, this is a test. * adding libtool to ubuntu deps (#3251) ## Summary The sandbox is not building with dev config using master branch https://github.com/algorand/sandbox/issues/85, complains about libtool not being installed Guessing from this change https://github.com/algorand/go-algorand/pull/3223 Adding libtool to UBUNTU_DEPS in install scripts ## Test Plan Set config in sandbox to my branch `sandbox up dev` It built * Fix error shadowing in Eval (#3258) ## Summary Error from account preloading was shadowed by returning a wrong err variable. This caused subsequent problems in account updates and masked the original failure. ## Test Plan Use existing tests * Disable flaky test. (#3256) ## Summary This test doesn't work properly, disable it until #3255 addresses any underlying problems. * Update the Version, BuildNumber, genesistimestamp.data * Fix a data race in app tests (#3269) ## Summary A test helper function `commitRound` accessed `l.trackers.lastFlushTime` without taking a lock. Fixed. ## Test Plan ``` go test ./ledger -run TestAppEmpty -race -count=50 ok github.com/algorand/go-algorand/ledger 4.078s ``` * Fix e2e.sh mixed indent characters. (#3266) ## Summary Fix e2e.sh mixed indent characters. * Fix ParticipationKeyResponse type. (#3264) ## Summary Fix a small type discrepancy in the OpenAPI spec ahead of some other work that's about to happen. * disable parallelism for e2e-go tests (#3242) ## Summary This sets `-p 1` for the e2e-go tests, intended to make them more deterministic when running on a VM with relatively constrained resources. Since each e2e-go test might spin up a few nodes, it seems like it would help to avoid resource contention. ## Test Plan Tests should run as before. Desired effect can be verified by looking at the test output where the value of PARALLEL_FLAG is printed out before tests are run. * Updating Readme.md with circleci status badges (#3245) * Fix formatting for CircleCI badges (#3272) * Add Custom Scenario for Performance Testing (#3278) Add Custom Scenario for Performance Testing. Add README on how to run custom scenario and modify create_and_deploy_recipe.sh to accept a network template that will generate a new recipe. * Impose limits on the entire "tree" of inner calls. This also increases the realism of testing of multiple app calls in a group by creating the EvalParams with the real constructor, thus getting the pooling stuff tested here without playing games manipulating the ep after construction. * ParticipationRegistry - StateProof loading methods (#3261) ## Summary Add ParticipationRegistry methods for setting and retrieving state proof keys. Since they aren't in master yet there is a `type StateProofKey []byte` stub which will need to be updated later. ## Test Plan New unit tests. * Op base64 decode (#3220) b64 opcode, tests, and specs * Bump Version, Remove buildnumber.dat and genesistimestamp.dat files. * Change golang version to 1.16.11 in go-algorand (#2825) Upgrading to 1.16 to help alleviate issues with working on different go versions, and update to a supported, more secure version. Release notes for Go 1.15 and 1.16: https://tip.golang.org/doc/go1.16 https://tip.golang.org/doc/go1.15 * Compatibility mode for partkeyinfo. (#3291) ## Summary Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running Workarounds: 1) use an older `goal` binary. 2) use `algokey part info --keyfile <file>` ## Test Plan Tested manually: ``` ~$ goal account partkeyinfo -d /tmp/private_network/Node/ Dumping participation key info from /tmp/private_network/Node/... Participation ID: CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA Parent address: DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU Last vote round: 1 Last block proposal round: 2 Effective first round: 1 Effective last round: 3000000 First round: 0 Last round: 3000000 Key dilution: 10000 Selection key: 5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ= Voting key: PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs= ~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c Dumping participation key info from /tmp/private_network/Node/... ------------------------------------------------------------------ File: Wallet2.0.3000000.partkey { "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU", "last": 3000000, "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=", "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=", "voteKD": 10000 } ``` * TestEcdsa: fix flaky "tampering" of public key (#3282) ## Summary This test (TestEcdsa) tests the ecdsa_pk_decompress opcode and intentionally "tampers" with the public key by setting the first byte to zero. Occasionally this test is failing, likely because the first byte was already zero. (The test failures are for the cases where failure is expected, `pass=false`) ## Test Plan Existing test should pass, occasional flakiness should go away. * Move appID tracking into EvalContext, out of LedgerForLogic This change increases the seperation between AVM execution and the ledger being used to lookup resources. Previously, the ledger kept track of the appID being executed, to offer a narrower interface to those resources. But now, with app-to-app calls, the appID being executed must change, and the AVM needs to maintain the current appID. * Stupid linter * Support reference types in `goal app method` (#3275) * Fix method signature parse bug * Support reference types * Review dog fixes * Fix comments * Add a hash prefix for ARCs-related hashes (#3298) ## Summary This is to allow ARCs (github.com/algorandfoundation/ARCs) to have their own hash prefix without risk of collision. ## Test Plan It is purely informational. There is no real code change. * catchup: suspend the catchup session once the agreement service kicks in (#3299) The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on. The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it. In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete. When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup. A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch. However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete. This change fixes the return value from true to false. * Compatibility mode for partkeyinfo. (#3291) ## Summary Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running Workarounds: 1) use an older `goal` binary. 2) use `algokey part info --keyfile <file>` ## Test Plan Tested manually: ``` ~$ goal account partkeyinfo -d /tmp/private_network/Node/ Dumping participation key info from /tmp/private_network/Node/... Participation ID: CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA Parent address: DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU Last vote round: 1 Last block proposal round: 2 Effective first round: 1 Effective last round: 3000000 First round: 0 Last round: 3000000 Key dilution: 10000 Selection key: 5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ= Voting key: PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs= ~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c Dumping participation key info from /tmp/private_network/Node/... ------------------------------------------------------------------ File: Wallet2.0.3000000.partkey { "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU", "last": 3000000, "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=", "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=", "voteKD": 10000 } ``` * catchup: suspend the catchup session once the agreement service kicks in (#3299) The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on. The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it. In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete. When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup. A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch. However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete. This change fixes the return value from true to false. * Bump buildnumber.dat * testing: disable flaky test (#3268) Disable a flaky test, to be re-enabled later with #3267. * enumerate conditions that might cause this fetchAndWrite to return false (#3301) ## Summary The fetchAndWrite function contains some complex logic to ultimately determine if we should continue trying to catch up. The conditions that might cause it to return false should be more explicitly enumerated. ## Test Plan Just comments * Fix unit tests error messages * make sure the block service is not attempting to access the ledger after being stopped. (#3303) ## Summary The block service was attempting to serve block via the http handler even after it has been stopped. This lead to undesired downstream failures in the ledger, which was shutdown as well. ## Test Plan unit test added. * Avoid creating algod process for the sole purpose of retrieving the genesis-id. (#3308) ## Summary Avoid creating algod process for the sole purpose of retrieving the genesis-id. Existing code was calling `algod -G -d <data dir>` in order to obtain the genesis version string. The genesis version string can be easily retrieved by loading the genesis file. ## Test Plan Use existing e2e tests. * documentation: fix algorand specs link (#3309) ## Summary This PR fixes a link in a README. ## Testing I clicked on the new link. * testing: reword partitiontest lint message. (#3297) ## Summary The wording on this was tripping me, maybe I was having an off day. I think it would be slightly easier if the message were to tell exactly what you need to do (and not use the angle brackets). * testing: fix random data race in TestAppAccountDataStorage (#3315) fix random data race in unit test * Allow access to resources created in the same transaction group The method will be reworked, but the tests are correct and want to get them visible to team. * ledger: perform the catchpoint writing outside the trackers lock. (#3311) ## Summary This PR moves the catchpoint file writing to be performed outside of the trackers lock. This resolves the issue where a long catchpoint file writing blocks the agreement from validating and propagating votes. ## Test Plan * [x] Test manually & use existing tests. * [x] Implement a unit test * [x] Deploy a local network where the catchpoint writing takes a long time and verify it doesn't get blocked during catchpoint writing. * Separate tx and key validity for `goal account renewpartkey` (#3286) Always use currentRound+proto.MaxTxnLife as last valid round for the transaction when renewing instead of using the partkey validity period. This fixes #3283 * Add qkniep to THANKS.md (#3320) ## Summary Add qkniep to THANKS.md * Followup to opcode base64_decode (#3288) * alphabet begone in favor of encoding * unit test various padding and whitespace scenarios * padding permutations also fail * "Slicing" --> "Manipulation" * fix the codegen fail? * Documenting padding, whitespace, other character behavior * Add help and fish mode to e2e interactive mode. (#3313) ## Summary Minor improvements to e2e.sh interactive mode: * add to -h output * do not run start stop test in interactive mode * support fish shell ## Test Plan Manual testing: ``` ~$ ./e2e.sh -i ... lots of output removed ... ********** READY ********** The test environment is now set. You can now run tests in another terminal. Configure the environment: set -g VIRTUAL_ENV "/home/will/go/src/github.com/algorand/go-algorand/tmp/out/e2e/130013-1639576513257/ve" set -g PATH "$VIRTUAL_ENV/bin:$PATH" python3 "/home/will/go/src/github.com/algorand/go-algorand/test/scripts"/e2e_client_runner.py "/home/will/go/src/github.com/algorand/go-algorand/test/scripts"/e2e_subs/SCRIPT_FILE_NAME Press enter to shut down the test environment... ``` * Minimum Account Balance in Algod (#3287) * Access to apps created in group Also adds some tests that are currently skipped for testing - access to addresses of newly created apps - use of gaid in inner transactions Both require some work to implement the thing being tested. * Remove tracked created mechanism in favor of examining applydata. * Add convertAddress tool. (#3304) ## Summary New tool: convertAddress I share this tool with someone every few months, putting it in the repo along with some documentation should make it easier to share and encourage people to share it amongst themselves if it's useful. Merge `debug` into `tools` to make it easier to organize these miscellaneous tools. * tealdbg: increase intermediate reading/writing buffers (#3335) ## Summary Some large teal source files cause the tealdbg/cdt session to choke. Upping the buffer size to allow for larger source files. closes #3100 ## Test Plan Run tealdbg with a large source teal file, ensure the source file makes it to cdt without choking. * Adding method pseudo op to readme (#3338) * Allow v6 AVM code to use in-group created asas, apps (& their accts) One exception - apps can not mutate (put or del) keys from the app accounts, because EvalDelta cannot encode such changes. * lint docs * typo * The review dog needs obedience training. * add config.DeadlockDetectionThreshold (#3339) Summary This allows for the deadlock detection threshold to be set by configuration. Test Plan Existing tests should pass. * Use one EvalParams for logic evals, another for apps in dry run We used to use one ep per transaction, shared between sig and and app. But the new model of ep usage is to keep using one while evaluating an entire group. The app ep is now built logic.NewAppEvalParams which, hopefully, will prevent some bugs when we change something in the EvalParams and don't reflect it in what was a "raw" EvalParams construction in debugger and dry run. * Use logic.NewAppEvalParams to decrease copying and bugs in debugger * Simplify use of NewEvalParams. No more nil return when no apps. This way, NewEvalParams can be used for all creations of EvalParams, whether they are intended for logicsig or app use, greatly simplifying the way we make them for use by dry run or debugger (where they serve double duty). * Remove explicit PastSideEffects handling in tealdbg * Fix flaky test in randomized ABI encoding test (#3346) * update abi encoding test random testcase generator, scale down parameters to avoid flaky test * parameterized test script * add notes to explain why flaky test is eliminated * show more information from self-roundtrip testing * fully utilize require, remove fmt * Always create EvalParams to evaluate a transaction group. We used to have an optimization to avoid creating EvalParams unless there was an app call in the transaction group. But the interface to allow transaction processing to communicate changes into the EvalParams is complicated by that (we must only do it if there is one!) This also allows us to use the same construction function for eps created for app and logic evaluation, simplifying dry-run and debugger. The optimization is less needed now anyway: 1) The ep is now shared for the whole group, so it's only one. 2) The ep is smaller now, as we only store nil pointers instead of larger scratch space objects for non-app calls. * Correct mistaken commit * ledger: perform the catchpoint writing outside the trackers lock. (#3311) ## Summary This PR moves the catchpoint file writing to be performed outside of the trackers lock. This resolves the issue where a long catchpoint file writing blocks the agreement from validating and propagating votes. ## Test Plan * [x] Test manually & use existing tests. * [x] Implement a unit test * [x] Deploy a local network where the catchpoint writing takes a long time and verify it doesn't get blocked during catchpoint writing. * Bump version number * Update `goal app method` handling of args and return values (#3352) * Fix method call arg overflow handling * Only check last log for return value * Address feedback * Add comment explaining ABI return prefix * Support app creation in `goal app method` (#3353) * Support app creation in `goal app method` * Don't use nonprintable tab character * Link to specific gist version * Fix error messages * Rename `methodCreation` to `methodCreatesApp` * Spec improvments * Update license to 2022 (#3360) Update license on all source files to 2022. * Add totals checks into acct updates tests (#3367) ## Summary After #2922 there is some leftover unused code for totals calculations. Turned this code into actual asserts. ## Test Plan This is tests update * More spec improvments, including resource "availability" * Recursively return inner transaction tree * Lint * No need for ConfirmedRound, so don't deref a nil pointer! * remove buildnumber.dat * license check * Shut up, dawg. * PKI State Proof Incremental Key Loading (#3281) ## Summary Followup to #3261 (contained in diff). Use the new key loading routine from the REST API. ## Test Plan New unit tests. * Limit number of simultaneous REST connections (#3326) ## Summary This PR limits the number of simultaneous REST connections we process to prevent the exhaustion of resources and ultimately a crash. Two limits are introduced: soft and hard. When the soft limit is exceeded, new connections are returned the 429 Too Many Requests http code. When the hard limit is exceeded, new connections are accepted and immediately closed. Partially resolves https://github.com/algorand/go-algorand-internal/issues/1814. ## Test Plan Added unit tests. * Use rejecting limit listener in WebsocketNetwork. (#3380) ## Summary Replace the standard limit listener with the new rejecting limit listener in `WebsocketNetwork`. This will let the dialing node know that connection is impossible faster. ## Test Plan Probably not necessary. * Delete unused constant. (#3379) ## Summary This PR deletes an unused constant. ## Test Plan None. * Add test to exercise lookup corner cases (#3376) ## Summary This test attempts to cover the case when an accountUpdates.lookupX method can't find the requested address, falls through looking at deltas and the LRU accounts cache, then hits the database — only to discover that the round stored in the database (committed in `accountUpdates.commitRound`) is out of sync with `accountUpdates.cachedDBRound` (updated a little bit later in `accountUpdates.postCommit`). In this case, the lookup method waits and tries again, iterating the `for { }` it is in. We did not have coverage for this code path before. ## Test Plan Adds new test. * Test for catchup stop on completion (#3306) Adding a test for the fix in #3299 ## Test Plan This is a test * Delete unused AtomicCommitWriteLock(). (#3383) ## Summary This PR deletes unused `AtomicCommitWriteLock()` and simplifies code. ## Test Plan None. Co-authored-by: DevOps Service <devops-service@algorand.com> Co-authored-by: Jason Paulos <jasonpaulos@users.noreply.github.com> Co-authored-by: chris erway <51567+cce@users.noreply.github.com> Co-authored-by: Shant Karakashian <55754073+algonautshant@users.noreply.github.com> Co-authored-by: John Lee <64482439+algojohnlee@users.noreply.github.com> Co-authored-by: Will Winder <wwinder.unh@gmail.com> Co-authored-by: Ben Guidarelli <ben.guidarelli@gmail.com> Co-authored-by: Pavel Zbitskiy <65323360+algorandskiy@users.noreply.github.com> Co-authored-by: Jack <87339414+algojack@users.noreply.github.com> Co-authored-by: John Lee <john.lee@algorand.com> Co-authored-by: algobarb <78746954+algobarb@users.noreply.github.com> Co-authored-by: Zeph Grunschlag <tzaffi@users.noreply.github.com> Co-authored-by: Fabrice Benhamouda <fabrice.benhamouda@normalesup.org> Co-authored-by: Tsachi Herman <tsachi.herman@algorand.com> Co-authored-by: Tolik Zinovyev <tolik@algorand.com> Co-authored-by: egieseke <eric_gieseke@yahoo.com> Co-authored-by: Quentin Kniep <kniepque@hu-berlin.de> Co-authored-by: Hang Su <87964331+ahangsu@users.noreply.github.com> Co-authored-by: Or Aharonee <or.aharonee@algorand.com> Co-authored-by: Barbara Poon <barbara.poon@algorand.com> * Feature/contract to contract (#3390) * Update the Version, BuildNumber, genesistimestamp.dat…

CI: use libboost-math-dev instead of libboost-all-dev

b7fc69d

add apt make dependency to .travis.yml

9fc1ab7

tsachiherman mentioned this pull request Aug 11, 2021

Use go-algorand types algorand/indexer#611

Merged

fix apt make dependency for .travis.yml external arm64

b60a68c

Add make to install_linux_deps

31b4dc6

cce closed this Aug 13, 2021

cce deleted the ci-less-boost branch August 13, 2021 14:21

This was referenced Nov 17, 2021

CI: use libboost-math-dev instead of libboost-all-dev #3222

Closed

CI: use libboost-math-dev instead of libboost-all-dev #3223

Merged

jannotti mentioned this pull request Jan 18, 2022

Creating clean branch / PR for audit #3433

Closed

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: use libboost-math-dev instead of libboost-all-dev #2717

CI: use libboost-math-dev instead of libboost-all-dev #2717

cce commented Aug 9, 2021 •

edited

tsachiherman commented Aug 9, 2021

codecov-commenter commented Aug 11, 2021 •

edited

tsachiherman commented Aug 11, 2021 •

edited

CI: use libboost-math-dev instead of libboost-all-dev #2717

CI: use libboost-math-dev instead of libboost-all-dev #2717

Conversation

cce commented Aug 9, 2021 • edited

Summary

Test Plan

tsachiherman commented Aug 9, 2021

codecov-commenter commented Aug 11, 2021 • edited

Codecov Report

tsachiherman commented Aug 11, 2021 • edited

cce commented Aug 9, 2021 •

edited

codecov-commenter commented Aug 11, 2021 •

edited

tsachiherman commented Aug 11, 2021 •

edited