Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

go-algorand 3.9.2-stable Release PR #4506

Merged
merged 97 commits into from
Sep 2, 2022

Conversation

Algo-devops-service
Copy link
Contributor

@Algo-devops-service Algo-devops-service commented Sep 1, 2022

GitHub Logo

IMPORTANT
This release requires a protocol upgrade.
This release contains a consensus protocol upgrade, which implements the following spec: https://github.com/algorandfoundation/specs/tree/433d8e9a7274b6fca703d91213e05c7e6a589e69

What's New

  • State Proofs: lightweight, cryptographically provable, portable proofs of Algorand state that let dapps on other blockchains trustlessly verify Algorand transactions, making it safer and easier to develop and use cross-chain products and services.
  • Max TPS of 6k
    • Larger blocks: ~5x increase in max-block size (from 1MB to 5MiB), allowing up to ~25k transactions per block.
    • Shorter round times: ~0.5s reduction in average round times (approaching ~3.9s).
  • AVM v7
    • On-chain randomness beacons and oracles can now be built by taking advantage the new block and vrf_verify opcodes.
    • Unfunded accounts (with 0 microalgo) can now issue transactions as long as their fees are covered by fee pooling in the transaction group.
    • App calls are now implicitly given access to the app accounts for the apps specified in the foreign apps array.

Changelog

New Features

  • Algod: State Proofs (#4226)
  • AVM: randomness support (#3900)
  • AVM: Refactor Pseudo-ops (#4170)
  • AVM: Provide access to large programs in a txn, for read and inner submit (#4148)
  • Goal Network Catchup: automatically fetch catchpoint if not provided (#4012)

Enhancements

  • 320 Rounds: Add logging for new logic (#4332)
  • AVM: Spec updates for v7 (#4347)
  • AVM: Fix crypto benchmarks from bitrot (#4340)
  • AVM: Move elliptic curve math up to v8 so it doesn't get released (#4282)
  • AVM: Consolidate TEAL and AVM versioning (#4269)
  • AVM: Don't parallelize tests that use shared for loop var (#4253)
  • Algod: rename some API operations for clarity (#4376)
  • Ci: Run go vet on the entire repo (#4312)
  • Ci: Add parameter to run Nightly Test (#4299)
  • Cicd: add mac arm64 to integration tests (#4167)
  • Cicd: upgrade xcode on circleci macos executors (#4188)
  • Cicd: remove unused genesis in stash (#4155)
  • Config: set MaxAcctLookback to 4 (#4296)
  • Config: update vFuture consensus and configuration for 5MB blocks (#4273)
  • Consensus: introduce v33, v34 (#4334)
  • Crypto: remove batch verification flag (#3781)
  • Docs: Control the specs generated seperately from logic.LogicVersion (#4361)
  • Enhancement: Detect when valid partkeys represent an offline or closed account (#4030)
  • Github-actions: add Not-Yet-Enabled and Skip-Release-Notes labels (#4196)
  • Goal: Allow --signer to send txns from rekeyed accounts (#4175)
  • Goal: Make the -n flag (network name) optional. Get it from template file (#4159)
  • Ledger: restore pointer receivers and args from 4003 (#4239)
  • Ledger: shorter deltas lookback implementation (320 rounds project) (#4003)
  • Loadgenerator: add multiple accounts config option (#4321)
  • Logging: Add a way to inject an external logger. (#4158)
  • Metrics: Don't allocate when invoking the Inc() fastpath (#4193)
  • Network: make txBacklogSize responsive to block size (#4377)
  • Optimization: Use bits package for Muldiv. 266x speedup (#4210)
  • Tests: extended logging on expect test abort (#4343)
  • Tests: cluster scenarios for 2022-07 (#4331)
  • Tests: touch fields_string.go so everything is rebuilt in codegen verification (#4330)
  • Tests: Make e2e subs report individual results better (#4286)
  • Tests: Fix typo in ledger_test.go (#4297)
  • Tests: Support multiple e2e test archives. (#4284)
  • Tests: Fail codegen_verification if dump_genesis.sh is bad (#4277)
  • Tests: use T.TempDir to create temporary test directory (#4256)
  • Tests: fixes to catchpoint e2e tests (#4236)
  • Tests: pingpong fixes and heap watch reporting (#4069)
  • Tools: adapt loadgenerator to cluster testing (#4324)
  • Tools: Fixing updater to accept m1 (Darwin arm64) (#4099)

Bugfixes

  • AVM: Make txn FirstValidTime and block opcode available in logicsigs (#4371)
  • Algod: fix issues causing database is locked sqlite error (#4110)
  • Bug Fix: Adds ephemeral Flag and Better Error messages (#4245)
  • Catchup: Download StateProofVotersLookback more amount of blocks if needed (#4392)
  • Catchpointdump: fix downloading logic (#4171)
  • Goal: Fix source map mappings key and use relative line values (#4198)
  • Ledger: fix accountsMu usage (#4280)
  • Ledger: remove txtail from data race test (#4315)
  • Ledger: Initialize the accountUpdatesLedgerEvaluator properly (#4456)
  • Ledger: fix txtail restoring from catchpoints (#4460)
  • Ledger: extend catchpoint blocks lookback (#4463)
  • REST API: Change State Proof message to no longer be embedded (#4395)
  • State Proofs: update state proof totals calculation (#4445)
  • Tests: un-shadow loadgen privateKeys var (#4380)
  • Tests: fix TestAssetValidRounds (#4351)
  • Tests: fix TestAccountsCanSendMoney (#4354 #4374 #4379)
  • Tests: close ledger after TestCatchupUnmatchedCertificate (#4348)
  • Tests: simplify TestWebsocketNetworkMessageOfInterest (#4341)
  • Tests: pass fixture by pointer instead of by value, since it contains a mutex (#4310)
  • Tests: fix TestParticipationDB_Locking (#4265)
  • Tests: make SP e2e tests easier for arm (#4402)
  • Tests: Remove using unreleased semicolon support in AVM test
  • Tests: ledger reload accessing txtail history (#4473)
  • Tools: goal logging send fixes (#4281)
  • Txns: Allow "empty" accounts to be Senders if someone else pays (#4151)

Activations (previously in vFuture)

  • AVM: Add bn256 pairing opcodes experimentally (#4013)
  • AVM: base64_decode can decode padded or unpadded encodings (#4015)
  • AVM: Allow immutable access to foreign app accounts (#3994)
  • AVM: Unify inner transaction ID calculation (#3927)
  • AVM: replace opcodes (#4129)
  • AVM: Set proper costs for json_ref (#4096)
  • StateProofs: New block header field - SHA256 merkle root of the transactions (#3829)
  • StateProofs: fix GetProof endpoint response (#3985)

Additional Resources

id-ms and others added 30 commits June 21, 2022 14:29
Always use the new (batch) verification system, having confirmed no incompatibilities with existing blockchain.
…lgorand#4159)

`goal network create` required a `-n` flag to specify the network
name.  But the name can be set in the template file.  Now we error
only if neither place specifies a network name.
This fixes an issue introduced in algorand#4130.
The new code did not handle end of stream correctly.
Simplify a lot of work with rekeyed accounts.
go test -run ^NOTHING -bench 'BenchmarkOldMulDiv|BenchmarkNewMulDiv'
goos: darwin
goarch: amd64
pkg: github.com/algorand/go-algorand/data/basics
cpu: Intel(R) Core(TM) i7-1068NG7 CPU @ 2.30GHz
BenchmarkOldMulDiv-8   	 2501655	       479.3 ns/op
BenchmarkNewMulDiv-8   	653492884	         1.801 ns/op
* Fix race condition in pingpong by adding mutex and  some internal
rearranging to facilitate efficient locking.
* Fix infinite loop in sendFromTo()
* pingpong run adds --pidfile path and --cpuprofile path
* Add csv report to test/heapwatch/client_ram_report.py
* Add script test/scripts/tps.py to get recent TPS from algod
…lgorand#4003)

Reduce deltas size from 320 to 8 by introducing a new online accounts tracker
that preserves history of state (online/offline, stake, protos) for at least
MaxBalLookback = 2 x SeedRefreshInterval x SeedLookback rounds back from Latest.
New data are stores in new tables and they are excluded from catchpoints.

TxTail stores its data into a table as well in order to prevent full blocks loading on startup
and preserves up to MaxTxnLife + DeeperBlockHistory blocks and caches their headers.

Catchpoint generation is made in two stages: data file at X-320 and catchpoint itself at round X.

Regular nodes see ~3x memory consumption decrease on high 6,000 TPS load.

Co-authored-by: Tsachi Herman <tsachi.herman@algorand.com>
Co-authored-by: nicholasguoalgorand <67928479+nicholasguoalgorand@users.noreply.github.com>
Co-authored-by: chris erway <chris.erway@algorand.com>
…nd#4239)

In algorand#4003 some pointer receivers and pointer arguments were changed
to pass by value and value receivers, which could lead to a performance regression.
This changes them back.
This commit replaces `ioutil.TempDir` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.

Prior to this commit, temporary directory created using `ioutil.TempDir`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
	defer func() {
		if err := os.RemoveAll(dir); err != nil {
			t.Fatal(err)
		}
	}
is also tedious, but `t.TempDir` handles this for us nicely.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
)

* use WAL on ErasableAccessor to allow concurrent reads from table A while writing to table B
* Unwrap errors in dbretry (in case someone return a wrapped from the Atomic fn)
  and test locking within the Atomic scope instead of outside it.
* fix a bug where DB handle was not closed by algokey
* Prevent the main thread quitting before the flush routine
* Make the test more robust by counting warning messages
* Cut the execution time few times by reducing number of iterations
Aharonee and others added 20 commits August 9, 2022 16:35
* Rename GetProof to GetTransactionProof to avoid ambiguity
* Rename GetProofForLightBlockHeader to GetLightBlockHeaderProof
…gorand#4392)

A fast-caught up node should be able to be part of state proofs creation and must download all necessary blocks.
Currently, if the state proof chain is lagging the node download lowestStateProofRound - stateproofInterval
rounds back on fast catchup. This is not accurate since the balances comes from
lowestStateProofRound - stateproofInterval - StateProofVotersLookback round.
* After the protocol upgrade nodes required to have MaxTxnLife + DeeperBlockHeaderHistory
  blocks that violated the test's expectations.
* Fixed a rare case when a block exists in a local ledger but peer's stat is still updated
  even if no communications were made.
* Catchpoints only have minimal data, and online accounts/txtail derive it
  by applying catchpoint
* Deriving code is reused from DB schema migration but the migration only assumed
  MaxTxnLife worth of block history
* In fact, it needs to restore MaxTxnLife + DeeperBlockHeaderHistory
  when used as part of catchpoint apply code path
Tests: Remove using unreleased semicolon support in AVM test
* replaying transactions after catchpoint requires 1001 blocks
  from the very first replayed block
* Ensure blk opcode can be replayed after restart
* Fix e2e fast catchup tests
Copy link
Contributor

@algoanne algoanne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good w.r.t. AVM changes.

@codecov
Copy link

codecov bot commented Sep 1, 2022

Codecov Report

Merging #4506 (55f0082) into rel/stable (641a1f1) will increase coverage by 1.11%.
The diff coverage is 67.92%.

@@              Coverage Diff               @@
##           rel/stable    #4506      +/-   ##
==============================================
+ Coverage       54.59%   55.70%   +1.11%     
==============================================
  Files             391      403      +12     
  Lines           48732    50820    +2088     
==============================================
+ Hits            26605    28311    +1706     
- Misses          19908    20108     +200     
- Partials         2219     2401     +182     
Impacted Files Coverage Δ
catchup/ledgerFetcher.go 40.22% <0.00%> (+2.59%) ⬆️
catchup/service.go 72.34% <0.00%> (+2.22%) ⬆️
cmd/goal/application.go 11.16% <0.00%> (ø)
cmd/goal/asset.go 16.25% <0.00%> (ø)
cmd/goal/clerk.go 8.75% <0.00%> (-0.18%) ⬇️
cmd/goal/logging.go 7.93% <0.00%> (-1.00%) ⬇️
cmd/goal/network.go 16.66% <ø> (-0.92%) ⬇️
cmd/tealdbg/localLedger.go 65.21% <0.00%> (-0.41%) ⬇️
config/config.go 47.05% <ø> (ø)
config/localTemplate.go 42.85% <ø> (ø)
... and 160 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@algojohnlee algojohnlee merged commit a32df45 into algorand:rel/stable Sep 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.