Create light block response as default for AN block lookup API #2148
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
FVM Benchstat comparisonThis branch with compared with the base branch onflow:master commit 6e43f80 The command Bench tests were run a total of 7 times on each branch. Results
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
synzhu
approved these changes
Mar 15, 2022
peterargue
approved these changes
Mar 16, 2022
Codecov Report
@@ Coverage Diff @@
## master #2148 +/- ##
==========================================
- Coverage 57.50% 57.46% -0.04%
==========================================
Files 637 637
Lines 37261 37272 +11
==========================================
- Hits 21426 21420 -6
- Misses 13151 13170 +19
+ Partials 2684 2682 -2
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
zhangchiqing
added a commit
that referenced
this pull request
Mar 23, 2022
* Computation Measuring Prototype * Add adidtional metrics and tests to fvm bench * Colaps the bench results output * create executable for gathering samples * minor improvements to data collection * more transactions and weights * add storage migration for ordered map * tests execute to completion * commit changes for draft PR * sanity checks now pass * works with arrays * add more tests * move migrationRuntimeInterface out to its own file * only fetch ordered map once for each domain/owner pair * add test for complex values * rename ambiguous local variable * respond to review comments * fix build error * rebased cadence version to an older build * fix reporters * increase account reporter memory limit * fix dropping of slabPayloads * respond to review * remove migration runtime interface * flaky test monitor - added commit SHA, commit date, job run date to each test result * move splitPayloads to utils * flaky test monitor - level 1 - updated output data to match BigQuery nested rows * flaky test monitor - level 2 - updated level 1 data input - output field * flaky test monitor - level 1 - removed skipped tests, updated no result tests to return -100 result * flaky test monitor - level 1 - changed tests to use 1 or 0 for their result instead of pass or fail * flaky test monitor - level 2 - changed tests to use 1 or 0 for their result instead of pass or fail * [utils/reporter] refactor balance reporter into fungible token tracker (#2012) * flaky test monitor - level 1 - updated parser and all JSON test files to be BigQuery compliant; updated tests to use require.contains() without requiring results in specific order anymore * flaky test monitor - level 1 - removed PackageResult since not storing package level data anymore * undefined features have 0 weight * flaky test monitor - level 1 - test data - added 1st struct driven test (instead of JSON file driven) * flaky test monitor - level 1 - test data - added 2nd struct test, added printTestResult() for better failure messaging * flaky test monitor - level 1 - test data - added 3rd struct test * flaky test monitor - level 1 - test data - added 4th struct test * flaky test monitor - level 1 - test data - added 5th struct test * flaky test monitor - level 1 - test data - refactored to create convenience functions: getPassedTest(), getPassedTestElapsed() * flaky test monitor - level 1 - test data - added 6th struct test * cleanup * cleanup * lint fix * flaky test monitor - level 1 - test data - added 7th struct test * flaky test monitor - level 1 - test data - added 8th struct test * flaky test monitor - level 1 - test data - added 9th struct test * flaky test monitor - level 1 - test data - added 10th struct test * flaky test monitor - test data - moved test data to common package so can be used by multiple levels * flaky test monitor - level 2 - added 1st struct driven test * cleanup * flaky test monitor - level 2 - added 2nd struct driven test (nil test) * increase gas limit in the parameter calibration * data for static loop lenghts * shorter loops * temporarily remove mixed transactions * dont adjust parameter range * flaky test monitor - level 2 - added 3rd struct driven test - many level 1 summaries, many no-result tests * flaky test monitor - renaming level 1, level 2 types to be more clear * flaky test monitor - renaming level 1, 2, 3 types to be more clear * flaky test monitor - added 4th struct driven test - many level 1 summaries, many failures, many passes * Remove some features from collection * Fix potential overflow in ledger metrics delta * Update generated mocks for LedgerMetrics * flaky test monitor - added 5th struct driven test - many level 1 summaries, many failures, many passes, many no-result tests * flaky test monitor - test data refactoring * flaky test monitor - level 2 tests updated to test with JSON file * Change fee deduction logic * flaky test monitor - level 2 - refactored to use same code for processing level 1 structs or JSON files * flaky test monitor - level 2 test data - deleted all JSON files that have been replaced by Level 1 Summary structs * flaky test monitor - level 1 test data - deleted all JSON files that have been replaced by Level 1 Summary structs (left 1 JSON file test) * core contracts update * update core contracts * upgrade core contracts * upgrade core-contracts * cleanup * flaky test monitor - level 2 - added explanation for -100 test result * flaky test monitor - level 1 - extracted hard coded env vars (commit sha, commit data, job date) into testdata package * flaky test monitor - test data - added more comments to explain contruction of level 1, 2 test data * core-contracts upgrade * remove unnecesary changes * remove the parameter calibration to new PR * fix lint * Update relic_build.sh * Target westmere * Target westmere with x86_64 only * address review comments * Refactor computation metering * flaky test monitor - level 1 tests - updated to use require.ElementsMatch() for comparing expected, actual results * flaky test monitor - test data - added more comments to explain different loop counters * fix tests * Add MTrie.regCount and MTrie.regSize to reduce mem Reduce RAM use, file size, and improve speed. E.g. reduce checkpoint file size by over 4GB and RAM usage by a similar amount. Speed will also benefit from these changes (benchmarks coming.) lowestHeightTouched returned by update() is the lowest height reached during recursive update. Unlike maxDepth, lowestHeightTouched isn't affected by prune flag. It's mostly new/updated node height. It can also be height of new node created from compact leaf at a higher height. Changes include: - remove Node.regCount (8 bytes) and Node.maxDepth (2 bytes) - remove regCount (8 bytes) and maxDepth (2 bytes) from checkpoint node serialization - add MTrie.regCount (8 bytes) and MTrie.regSize (8 bytes) - add regCount (8 bytes) and regSize (b bytes) to checkpoint trie serialization - modify trie update() to return regCountDelta, regSizeDelta, and lowestHeightTouched so that NewTrieWithUpdatedRegisters() can compute regCount, regSize, and maxDepthTouched for the updated trie - fix Payload.Equals() crash bug when p is nil - fix Payload.Equals() bug when comparing nil payload with empty payload * computation metereing changes * refactor fvm metring * improve docs * clean up * flaky test monitor - no-result tests modified to count as failure, added new field * read weights from state * remove unnecesary change * Avoid heap allocs for goroutine in update() Output variables shared outside goroutine are allocated on the heap. Use channel to transfer output variables to avoid heap allocation. * Check for nil payload in Payload funcs * fix linter * fix linter * don't ignore errors * fix linter * fix linter * update to latest cadence version * add more tests * handling service txs limits on a higher level * bump cadence version * revert an unwanted change * respect computation limit set by service tx * increase limits for service transactions * test fix * clean up * add WithGasLimit as depricated method for now * [BFT Testing] Adds attacker implementation (#2121) * adds attacker and event * refactors attacker * updates mock of attacker * adds attacker test * fixes lint * fixes lint * fixes context cancel error on corrupted node side * improves the decomposition of testing * isolates test package for adversary * refactors attacker test with zero port * refactors event content * fix linting * Update crypto/relic_build.sh Co-authored-by: Tarak Ben Youssef <50252200+tarakby@users.noreply.github.com> * Add test and fix build * reference the 4th address for flow fees * Update crypto/relic_build.sh Co-authored-by: Tarak Ben Youssef <50252200+tarakby@users.noreply.github.com> * cleanup * apply pr's comments * Avoid deep copy of compactLeaf payload in update When compactLeaf from a higher height needs to be created at a lower height during update, its payload doesn't need to deep copied because we are reusing the trie internal reference and payloads of leafs are immutable. * move service tx limits to another pr * set mem limit to 2G * [Execution] enable cadence runtime tracing given node level flag (#2108) * Use empty payload value to test unallocated reg A register is allocated if and only if it has a non-empty value. A register is unallocated if and only if its value is empty. In update(), use empty payload value to distinguish allocated vs unallocated registers to adjust allocatedRegCountDelta and allocatedRegSizeDelta. * Move updateResult type outside of update() * read the weights from the state * Create light block response as default for AN block lookup API (#2148) * upgrade to cadence v0.11.0 * state comitment fix due to CC change * Encode length of encoded payload value in 4 bytes In v0, length of encoded payload value is encoded in 8 bytes. In v1, length of encoded payload value is encoded in 4 bytes. Payload decoding handles both v0 and v1. Payload is encoded according to version received as parameter. This allows gradual migration of other encoding formats using payload, such as TrieUpdate and TrieProof. * Store and retrieve transaction results by EN RPC * Bump flow dependency to v0.2.4, remove unnecessary conversion * go mod tidy * go mod tidy * [Observer] Add bootstrap command to generate observer networking key * don't leak implementation details of key generation * Apply suggestions from code review Co-authored-by: Simon Zhu <simon.zsiyan@gmail.com> * pass direct path instead of dir * mark output-file required * fix test and linting * update comments for latest changes * use slice length directly * Move set execution weights to separate processor * fix bench test * remove some memory metering * add more meter tests * meter with higer internal precision * Move basic meter specific things into basic meter * [BFT Testing] Implements attack network (#2144) * adds attack vector interface * adds attack network interface * wip implementing attacker * wip * wip * wip * implements start method for attack network * implements attack network * adds skeleton for wintermute orchestrator * adds start for orchestrator * adds component startup to attack network * adds a comment * refactors orchestrator interface * implements processing observe message on attakcer * wires logger and codec for the attacker constructor * adds godoc for wintermute orchestrator * encapsulates the logic of listen and serve * adds documentation for the attacker * adds attacker registerer interface * wip * integrates attacker registration with attack network * refactors orchestrator * adds mocks * refactors attacker startup * adds startup test * refactors attacker startup * adds with scenario helper * adds tests for single and multiple concurrent messages * removes redundant files * implements mock corruptible conduit factory * adds attacker and event * refactors attacker * updates mock of attacker * adds attacker test * fixes lint * fixes lint * implements corrupted clients * refactors attack network with corrupted connection and connector * moves attack network to attacker logic * adds registration functionality of attacker for iorchestrator * removes redundant attacker stop from test * generates mocks * generates mocks * fixes context cancel error on corrupted node side * integrates attack network with attacker * encapsulates mock corrupted connector connect * refactors with attacker helper * improves the decomposition of testing * isolates test package for adversary * wip * implements test attack network * generalizes testing attack network * fixes compile issues * renames attacker to attack network * adds comments * adds comments * deletes mock corruptible conduit factory * adds comment * refactors attack network code * refactors interface of attack network * generates mocks for attack network interface * refactors implementation of attack network to reflect to interface * fixes broken tests * renames conduit master to conduit controller * refactors targets to target num on insecure message * fixes lint * removes unnecessary return * adds todos * adds godocs * weighted meter as a separate meter * remove StateHolder from Environment * change location of reading weights from state * move meter setup to new env call * fail transaction on Fatal error on create meter * cleanup * renaming * lint fix * change back to SetWeights instead of set meter * minor cleanups * cleanup and test fix * Bump checkpoint file format to v5 Bump version of checkpoint file to v5. Support reading v4 for compatibility. Add tests and test data to read v4. * Support V4 and V5 in checkpoint extraction cmd * add more weighted meter tests * apply review comments * Create transaction result by index api (#2159) * Create Handler for GetTransactionResultByIndex in AN * Add comments about WaitGroup vs channel in update * Fix typo in a comment * Add test for trie update with mixed prune flag TestTrieAllocatedRegCountRegSizeWithMixedPruneFlag tests allocated register count and size for updated trie with mixed pruning flag. It tests the following updates with prune flag set to true: * step 1 : update empty trie with new paths and payloads (255 allocated registers) * step 2 : remove a payload without pruning (254 allocated registers) * step 3a: remove previously removed payload with pruning (254 allocated registers) * step 3b: update trie from step 2 with a new payload (sibling of removed payload) with pruning (255 allocated registers) * Remove some blank lines after comments * Extract some logic from trie update to new funcs * Fix comment for TestTrieAllocatedRegCountRegSizeWithMixedPruneFlag * [Observability] Adds metrics for HeroCache (#2093) * [WIP] adds metrics interface * fixes available slots bug * adds herocache interface * implements herocache collector * implements noop collector for herocache * renames collector file * wip refactoring metrics factory * refactors transactions mempool * fixes compile errors * wires metrics and removes telemetry logs * wires in size metrics for networking receive cache * fixes lints * refactors receive cache size name * fixes metrics duplication issue * reverts back logging telemetry * fixes a log * renames a metric * renames a metric * refactors metrics name * removes unused subsystems * adds available slot histogram * fixes a comment * renames a metric * removes size method * refactors factory of hero cache collector * fixes lints * fixes lints * fixes lint * fixes circular dependency * adds with back data option * refactors to new hero receive cache * decouples dns cache from dns resolver * removes factory with backdata for backend * renames metrics * extends interface of metrics * wires additional metrics * adds epoch number to mempool label * refactors parameter organization * revises comments for herocache metrics collector * renames metrics * renames metric variables in herocache collector * adds comment for available slots histogram * refactors a godoc * fixes a comment * fixes a godoc * adds logger key for dns cache * removes a commented code * renames a variable * enables herocache metrics collection switchable through a flag * Add "compute" prefix to two function names - computeAllocatedRegDeltasFromHigherHeight - computeAllocatedRegDeltas Co-authored-by: Janez Podhostnik <janez.podhostnik@gmail.com> Co-authored-by: Daniel Sainati <sainatidaniel@gmail.com> Co-authored-by: gomisha <misha@gomisha.com> Co-authored-by: Ramtin M. Seraj <ramtinms@users.noreply.github.com> Co-authored-by: Faye Amacker <33205765+fxamacker@users.noreply.github.com> Co-authored-by: Yago <yagop@users.noreply.github.com> Co-authored-by: Yago Perez <yagoperezs@gmail.com> Co-authored-by: ramtinms <ramtin@axiomzen.co> Co-authored-by: Yahya Hassanzadeh <yahya@dapperlabs.com> Co-authored-by: Tarak Ben Youssef <50252200+tarakby@users.noreply.github.com> Co-authored-by: Misha <misha.rybalov@dapperlabs.com> Co-authored-by: Simon Zhu <simon.zsiyan@gmail.com> Co-authored-by: bors[bot] <26634292+bors[bot]@users.noreply.github.com> Co-authored-by: Amlandeep Bhadra <koko1123@users.noreply.github.com> Co-authored-by: Maks Pawlak <120831+m4ksio@users.noreply.github.com> Co-authored-by: Peter Argue <89119817+peterargue@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
closes #2051
block.go, as hash validation check is only possible with the full block response