General updates#1
Merged
Merged
Conversation
added 7 commits
July 4, 2023 17:12
Kay-Zee
approved these changes
Jul 11, 2023
peterargue
approved these changes
Jul 20, 2023
Co-authored-by: Peter Argue <89119817+peterargue@users.noreply.github.com>
tarakby
added a commit
that referenced
this pull request
Dec 8, 2023
* update to Cadence v0.24.6 * Remove header caching This 1 item caching introduces global lock and source of a 3+% slowdown. ``` $ go test -tags relic -bench='BenchmarkComputeBlock' -count 10 -run='^$' ./engine/execution/computation/. ... $ benchstat old new name old us/tx new us/tx delta ComputeBlock/1/cols/16/txes-8 590 ± 3% 803 ±62% ~ (p=0.404 n=10+10) ComputeBlock/1/cols/32/txes-8 554 ± 2% 559 ± 2% +0.93% (p=0.043 n=9+8) ComputeBlock/1/cols/64/txes-8 557 ± 4% 538 ± 2% -3.46% (p=0.000 n=10+10) ComputeBlock/1/cols/128/txes-8 560 ± 1% 538 ± 2% -3.92% (p=0.000 n=7+9) ComputeBlock/4/cols/16/txes-8 558 ± 3% 539 ± 1% -3.51% (p=0.000 n=10+10) ComputeBlock/4/cols/32/txes-8 555 ± 1% 540 ± 1% -2.81% (p=0.000 n=10+10) ComputeBlock/4/cols/64/txes-8 556 ± 2% 538 ± 2% -3.13% (p=0.000 n=10+10) ComputeBlock/4/cols/128/txes-8 546 ± 1% 527 ± 2% -3.50% (p=0.000 n=10+10) ComputeBlock/16/cols/16/txes-8 552 ± 3% 533 ± 2% -3.44% (p=0.000 n=10+10) ComputeBlock/16/cols/32/txes-8 548 ± 1% 527 ± 2% -3.68% (p=0.000 n=9+10) ComputeBlock/16/cols/64/txes-8 548 ± 1% 530 ± 1% -3.34% (p=0.000 n=10+10) ComputeBlock/16/cols/128/txes-8 522 ± 1% 519 ± 2% ~ (p=0.327 n=9+9) ``` * update crypto module version * fix DST dashes * update crypto version to v0.24.4 * fix import errors * added prometheus gauges * go fmt * check if block is nil * fix error check * update help message * update test to support Epoch Tx * fixed mock * comments * update Cadence * update Cadence * update location creation * add logs * update Go SDK * [BFT Testing] refactors gRPC interface (#2599) * empty wintermute test * fix for brining up corrupted node * added testnet.AsCorrupted() to specify corrupted node config * added happy path from verification test; dummy orchestrator WIP implementation while BootstrapNetwork() is refactored * dummy orchestrator WIP - added attack network * adds corrupted identities * implements dummy orchestrator * wires in dummy orchestrator * refactors wintermute * adds log for attack network * refactors lifecycle of attacker * adds port exposing * enables port exposing for corrupted nodes * adds logging for corrupted nodes * adds logging for default conduit factory * fixes bug with building corrupted images * refactors chain id as a parameter * implements corrupted port mapping * temp solution adds attacker docker address * adds port mapping to test suite * uncomments attacker registration * fixes bug: empty publish list * adds docker and local mode to attacker * new Github workflow for BFT integration tests (#2418) * added another bft branch filter * implements dummy orchestrator tracker * adds error for attacker address * adds comment to corrupted connector * removes redundant epose method * adds a godoc * adds comment * removed unit-integration test category * adds safety check for orrupted and ghost nodes * adds godoc for dummy orchestrator * adds more encapsulation * adds comments * makes generating execution receipt exportable * (no branch): Auto stash before checking out "HEAD" * removes corrupted execution result * makes generating result approval exportable * adds result approval generation to ccf * fixes the tests * adds attestation fixture * adds message content as a parameter to message fixture * fixes bug with publish * wip * adds test for result approval * adds corrupted execution receipt test * adds todos to factory * refactors output types of wintermute orchestrator * fixes lint issue * removes unused variable * Misha/6221 BFT Dummy Orchestrator - CI fix (#2445) * CI fix - put back unit-integration test category, removed error test category check in test-setup.sh * CI debug - test-setup.sh modified for debugging * CI debug - run-tests.sh modified for debugging * CI debug - run-tests.sh modified to send echo statements to standard error * CI fix - test-setup.sh modified to send echo statements to standard error * CI fix - added logging to test-setup.sh for unmatched test category * CI testing - re-enabled unit test running * CI - skipped-test-tracker.yml ready for CI test - added BFT integration tests, CI logging to test running * CI - skipped-test-tracker.yml ready for CI test - run-tests.sh, test-setup.sh reverted to original actions to test in CI * CI - skipped-test-tracker-integration.yml ready for CI test - added BFT integration tests, CI logging to test running * CI - skipped-test-tracker-integration.yml, skipped-test-tracker.yml reverted back to not using tee for running tests - tests were taking too long to complete run * CI - skipped-test-tracker-integration.yml, skipped-test-tracker.yml reverted to not using integration-bft test because it's consistently failing * CI - skipped-test-tracker-integration.yml, skipped-test-tracker.yml reverted to not using BFT branches * adds attack state getter * adds wait for block by id * adds wintermute test suite * refactors wintermute attack orchestrator * adds wintermute test * Update insecure/wintermute/helpers.go Co-authored-by: Misha <misha.rybalov@dapperlabs.com> * Update insecure/wintermute/helpers.go Co-authored-by: Misha <misha.rybalov@dapperlabs.com> * fixes lint * fixes receipt and approval bug with factory * adds pass through test for receipt * adds approval test * adds chunk data pack with start state * fixes bugs with wintermute orchestrator * adds comments to attack orchestrator * adds comments to wintermute orchestrator * revises a comment * adds comments * implements test for passing through miscellaneous events * fixes a bug * adds testing for result approval pass through * adds test wintermute result approval * adds wait for approvals from any * refactors result approval from * makes the code more DRY * refactors wintermute attack test * adds comment to wintermute test * refactors wintermute test * adds is flag set * lint fix * lint fix * nit * refactors grpc interface * updates grpc interface * refactors attacknetwork interface * refactors corrupted client * updates mocks * refactors corrupted connection * refactors lifecycle management of corrupted connector * refactors corrupted connector interface * refactors factory * removes grpc from attack network * makes corrupted connector ready done aware * fixes corrupted connector * fixes corrupted connector tests * re-generates mocks * implements with mock ccfs * wip * removes mocks that we no longer need * refactors corrupted connector interface * regenerates mocks * fixes corrupted connector tests * wip: fixes attack network tests * fixes all tests in attack network * refactors mock attacker observer client * fixes channel bug with factory * fixes factory tests * refactors integration tests * refactors channel-based delivery * fixes a bug with mock ccf * BFT CI - fixed building all docker images for BFT tests, not just corrupted images * increases timeout * bumps up the timeout * switches log level * adds logging to factory * switches back log levels * fixes race condition * skipping the test * skipping test * skips other tests that need a ccf grpc server * fixes race condition in ccf * adds more logging info * makes attacker registration exported * adds checking for attacker registration prior to sending messages * updated to go 1.18 * switches all logs to debug * unskips pass through * adds block rate to pass through * adds hotstuff timeout * replaces testnet trakcer * refactors log level * excludes attacker * switches all nodes to honest * removes timeout * skips wintermute * comments out push approval channel * comments in push approvals * lint fix * fixes lint and unskips passthrough * skips passthrough and unskips wintermute * skips wintermute, unskips passthrough * bft integration tests running sequentially * Update insecure/wintermute/attackOrchestrator.go Co-authored-by: Jordan Schalm <jordan@dapperlabs.com> * Update integration/tests/bft/wintermute/wintermute_test.go Co-authored-by: Jordan Schalm <jordan@dapperlabs.com> * Update integration/tests/lib/result_approval_state.go Co-authored-by: Jordan Schalm <jordan@dapperlabs.com> * Update integration/tests/bft/wintermute/suite.go Co-authored-by: Jordan Schalm <jordan@dapperlabs.com> * lint fix * refactors sealing condition * lint fix * Update integration/tests/lib/result_approval_state.go Co-authored-by: Jordan Schalm <jordan@dapperlabs.com> * adds more logging to block state * adds more logging to state tracker * bumps up wintermute timeout * adds more logging to factory * fixes a bug with factory * fixes wintermute sealing condition * Update integration/tests/bft/wintermute/wintermute_test.go Co-authored-by: Jordan Schalm <jordan@dapperlabs.com> * Update integration/tests/common/sealing_and_verification.go Co-authored-by: Jordan Schalm <jordan@dapperlabs.com> * Update integration/tests/bft/wintermute/wintermute_test.go Co-authored-by: Peter Argue <89119817+peterargue@users.noreply.github.com> * Update insecure/wintermute/attackOrchestrator_test.go Co-authored-by: Peter Argue <89119817+peterargue@users.noreply.github.com> * refactors wait group * lint fix * escalates log levels * revises attack network tests * revises a comment * hammerings on corrupted node connection * unskips factory tests * fixes a log message * revises corrupted connectors test * adds a comment * adds a comment * adds a comment * refactors factory * Makefile - added clarification why need to run bft tests sequentially * adds comments to mockAttackerObserveClient * nit * Update insecure/attacknetwork/corruptedConnector.go Co-authored-by: Peter Argue <89119817+peterargue@users.noreply.github.com> * renames grpc method * makes receive loop blocking * go fmt * go fmt * moving lock from orchestrator to attacker * adds a comment Co-authored-by: gomisha <misha@gomisha.com> Co-authored-by: Misha <misha.rybalov@dapperlabs.com> Co-authored-by: Jordan Schalm <jordan@dapperlabs.com> Co-authored-by: Peter Argue <89119817+peterargue@users.noreply.github.com> * fix migration test * update test * add legacy controller migration to the migrations * remove time.Sleep * update core contracts * update outdated mocks * add done channel to HotstuffFollower * update HotstuffFollower mock * fix unit tests * update HotstuffFollower mock * Revert "update HotstuffFollower mock" This reverts commit fa42d2546661fc78edf2e3e1132132f2e0f06349. * close the done channel without defer * removed tempfix comment * fix lint * update flow-core-contract module version * fix linter issue * update state commitments in tests after service account contract updates * update to read only channel * update state commitments * improve error documentation for middleware - add BenignNetworkingError sentinel that covers the entire class of benign networking errors - add MiddlewareStartError sentinel that covers the entire class of errors returned from Middleware.start - return appropriate sentinel errors and update godocs * Update access_api_proxy_test.go * comments and warning log * fixed interface * fix bls tests * update to read only channel * update to master of flow-go-sdk * update to read only channel * added a check for 0 execution nodes being passed * Unskipping passing flaky test * Moving emulator-based tests to the integration package (#2738) * [Exec] Remove global lock from script execution * process block responses * remove unused import * Update consensus/hotstuff/follower_loop.go Co-authored-by: Leo Zhang <zhangchiqing@gmail.com> * send proposalTask by pointer * lint fix * handleExecutionReceipt check for missing blocks * call metric within the callback for handleCollection * benchstat improvement * go fmt * [BFT Testing] Add BFT tests to regular CI workflows (#2725) * added integration-bft test-category to flaky-test-monitor-integration.yml * added integration-bft test-category to workflows - 1) CI 2) flaky-test-monitor 3) skipped-test-tracker 4) skipped-test-tracker-integration * added corrupted Docker images to docker-build-flow target * Makefile minor clean up Co-authored-by: Yahya Hassanzadeh <yahya@dapperlabs.com> * reduce benchmark_repetitions a little * removed uneccesary mock * wrapping not found error * record metrics inside callback * fix return empty list without error * [Exec] Remove Stop The World pauses from trx execution * remove STW from script execution * simplify * fixed getTransactionResultsByBlockIDFromAnyExeNode * factor runtime metrics to a separate file * rename the file * fixed lint * add a comment to the GetHeapAllocsBytes() * [Networking] Fixes flakey DNS cache expiry test (#2670) * Debug logging messages for expire cache test * Added comments to the code * Swapped to sequential queries for testing purposes * adding log messages * Logging and locking cache * switches back to sequential * wip * adds logger to cache and makes update atomic * cleans up resolver test * cleans up resolver test * cleans up resolver * wip * fixes race condition * refactors dns cache interface * implements locking ip and txt domains * fixes broken tests * refactors dns locking signature * adds test for dns cache lock * adds comments for tests * adds go doc * moves locking from resolver to cache * cleans up logs * refactors dns cache tests * adds update ip domain to dns cache * adds update txt domain to dns cache * revises a comment * adds dns cache update test * refactors dns cache interface * updates dns cache docs * refactors records * updates dns cache docs * fixes a bug with dns cache * revises dns cache * refactors with resolution result * reverts log level for unitttests * adds return for error path of locking a domain * fixes a comment * reduces test timeouts Co-authored-by: Leon Yu <leon.yu.87@gmail.com> * Update command used * remove sentinels, instead document benign errors * return errors directly * Update engine/access/ingestion/engine.go Co-authored-by: Peter Argue <89119817+peterargue@users.noreply.github.com> * update maxReceiptHeight * update maxCollectionHeight * add comment to the metric name string Co-authored-by: Leo Zhang <zhangchiqing@gmail.com> * add consistency to error docs * optimize emergency-sealing check limit emergency sealing count add comments update comments rename variables and add comments suggestions for PR #2673 (#2703) * expanded documentation around emergency sealing and renamed a few variables to improve clarity * typo * Update engine/consensus/approvals/Readme.md Co-authored-by: Leo Zhang <zhangchiqing@gmail.com> rename config fix tests fix tests Apply suggestions from code review Co-authored-by: Jordan Schalm <jordan@dapperlabs.com> update comments * comments and switch followerBlockProposal struct for inRangeBlockResponse argument * chore(localnet): faster account creation * wrap error message Co-authored-by: Leo Zhang <zhangchiqing@gmail.com> * comments Co-authored-by: Leo Zhang <zhangchiqing@gmail.com> * Update engine/common/follower/engine.go Co-authored-by: Leo Zhang <zhangchiqing@gmail.com> * comments and go fmt * Fixed docker-build-loader docker image build failure * tweak Programs locking + minor refactoring - reduce Get's critical section (don't hold lock when looking up parent's entry) - guard HasChanges * remove disk size metrics * Don't leak signerLock outside of Account * chore(loader): faster worker Stop() * add signer ids to access API fix access/handler fix tests fix tests * Add Andrew Meyer to performance stream Adding myself to the performance stream as suggested by @SaveTheRbtz * chore(loader): switch from Client to Access API * [Execution] check if state commitment exists before executing scripts (#2763) * optimize inner loop * add tests and test fix * lint fix * remove wal_metrics * [AN] - Trigger Cache Eviction on Overwrite (#2760) * trigger eviction logic for overwriting an entry * comment with reasoning Co-authored-by: Vishal <1117327+vishalchangrani@users.noreply.github.com> * chore(loader): cleanup follower * address PR comments * cleanup comments * consistent naming * remove some details from comment * chore(loader): wait for account creation results * go mod tidy * Update state/protocol/util.go Co-authored-by: Alexander Hentschel <alex.hentschel@axiomzen.co> * Update access/handler.go Co-authored-by: Alexander Hentschel <alex.hentschel@axiomzen.co> * Update state/protocol/util.go Co-authored-by: Alexander Hentschel <alex.hentschel@axiomzen.co> * Update state/protocol/util.go Co-authored-by: Alexander Hentschel <alex.hentschel@axiomzen.co> * Update README.md Update note about setting up the crypto module locally for every new version * add unit test for unhandled sealing segment case * handle multi-block root sealing segment * s/Rem/Remove/g to improve code readability * duplicate error in the return statement * improve naming * [BFT Testing] Fixes wintermute orchestrator race condition (#2780) * adds some comments * adds mutual exclusion to attack orchestrator * removes debug lines * Unskipping join and leave ln test * Apply suggestions from code review Co-authored-by: Khalil Claybon <khalil.claybon@dapperlabs.com> * Apply suggestions from code review Co-authored-by: Khalil Claybon <khalil.claybon@dapperlabs.com> * remove timeout case * unused import * work in progress * resolved circular import * added mock, linted code * more cleanup * organized imports * organized imports * organized imports * :-( * still trying to convince linter :-( :-( * next try to convince linter * wip * attempt failed * [localnet] Putting limit to EXECUTION config to avoid potential confusion * [localnet] also apply the same limit check for Consensus node * wip * linted code * reducing unnecessary changes * complete and mature draft * Apply suggestions from code review Co-authored-by: Jordan Schalm <jordan@dapperlabs.com> * addressing reviewer comments part 2 * extended `UnknownBlockError` * linted code * Add comments for returned errors * change unittest fixture to use *flow.Header instead of flow.Header Make test code more consistent with regular code * add explicit note that root block is self-sealing * Invoke contracts directly instead of via currying * chore(localnet): fix observer's trace endpoint * fix flaky test: TestStaticEpochTransition (#2698) * added timestamp to logs for block_state, testnet_state_tracker * added CI flaky test debug workflow for TestEpochStaticTransition/TestStaticEpochTransition * fixed typo in test-category in CI file * fixed docker build image target * updated workflow to run cron scheduler on non-master branch * fixed bug in CI file - flaky-test-debug.yml * Update static epoch test to use same testing flow (#2761) * update static epoch test to use same testing flow * updated bft-tests.yml for debugging TestEpochStaticTransition/TestStaticEpochTransition * added scheduled run every hour * kick off another CI test run for TestEpochStaticTransition/TestStaticEpochTransition #1 * kick off another CI test run for TestEpochStaticTransition/TestStaticEpochTransition #2 * #3 kick off another CI test run for TestEpochStaticTransition/TestStaticEpochTransition * #4 kick off another CI test run for TestEpochStaticTransition/TestStaticEpochTransition * #5 kick off another CI test run for TestEpochStaticTransition/TestStaticEpochTransition * #6 kick off another CI test run for TestEpochStaticTransition/TestStaticEpochTransition * #7 kick off another CI test run for TestEpochStaticTransition/TestStaticEpochTransition * #8 kick off another CI test run for TestEpochStaticTransition/TestStaticEpochTransition * #9 kick off another CI test run for TestEpochStaticTransition/TestStaticEpochTransition * #10 kick off another CI test run for TestEpochStaticTransition/TestStaticEpochTransition * remove sleep * Dummy commit * dummy commit * dummy commit * dummy commit * dummy commit * dummy commit * dummy commit * dummy commit * dummy commit * dummy commit * 7699 dummy commit * 3651 dummy commit * 21538 dummy commit * 7108 dummy commit * 7186 dummy commit * 1772 dummy commit * 30464 dummy commit * 15633 dummy commit * 11129 dummy commit * 15839 dummy commit * remove dummy file * remove dummy file Co-authored-by: gomisha <misha@gomisha.com> * minor cleanup * more minor cleanup Co-authored-by: Jordan Schalm <jordan@dapperlabs.com> * chore(trace): rename file * [FVM] removing legacy controller from storage (#2713) * remove legacy controller * fix tests * update chunk verifier * update test * update emulator to support controller removal * update flow protobuf version * go mod tidy * apply PR's comments * update flow proto files * update flow emulator version * go mod tidy * update state commitment * no in-place payload update * expected update to state commitments * update recent changes * apply PR's comment * Skipped broken tests related to active pacemaker * Skipped more tests * Skipped more tests Co-authored-by: Bastian Müller <bastian@axiomzen.co> Co-authored-by: bors[bot] <26634292+bors[bot]@users.noreply.github.com> Co-authored-by: GetElastech <codesubmit@getelastech.com> Co-authored-by: Alexey Ivanov <rbtz@dapperlabs.com> Co-authored-by: Tarak Ben Youssef <tarak.benyoussef@dapperlabs.com> Co-authored-by: Daniel Holmes <43529937+danielholmes839@users.noreply.github.com> Co-authored-by: danielholmes839 <danielh839@gmail.com> Co-authored-by: Khalil Claybon <khalil.claybon@dapperlabs.com> Co-authored-by: Janez Podhostnik <janez.podhostnik@gmail.com> Co-authored-by: Janez Podhostnik <67895329+janezpodhostnik@users.noreply.github.com> Co-authored-by: ramtinms <ramtin@axiomzen.co> Co-authored-by: Ramtin M. Seraj <ramtinms@users.noreply.github.com> Co-authored-by: Simon Zhu <simon.zsiyan@gmail.com> Co-authored-by: Yahya Hassanzadeh <yahya@dapperlabs.com> Co-authored-by: gomisha <misha@gomisha.com> Co-authored-by: Misha <misha.rybalov@dapperlabs.com> Co-authored-by: Jordan Schalm <jordan@dapperlabs.com> Co-authored-by: Peter Argue <89119817+peterargue@users.noreply.github.com> Co-authored-by: Faye Amacker <33205765+fxamacker@users.noreply.github.com> Co-authored-by: Daniel Sainati <sainatidaniel@gmail.com> Co-authored-by: lolpuddle <oocean.cheung@gmail.com> Co-authored-by: Tarak Ben Youssef <50252200+tarakby@users.noreply.github.com> Co-authored-by: Leon Yu <leon.yu.87@gmail.com> Co-authored-by: Leo Zhang <zhangchiqing@gmail.com> Co-authored-by: Tony Z <tony.zhang@dapperlabs.com> Co-authored-by: Patrick Lee <patrick.lee@dapperlabs.com> Co-authored-by: Alexey Ivanov <SaveTheRbtz@GMail.com> Co-authored-by: Andrew Meyer <ajm9088@gmail.com> Co-authored-by: Vishal <1117327+vishalchangrani@users.noreply.github.com> Co-authored-by: Alexander Hentschel <alex.hentschel@axiomzen.co> Co-authored-by: danielholmes839 <daniel.holmes@dapperlabs.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
flow-gorelated context.cand.h) and removeflow gomention from existing headersonflow/cadence)