Jenkins reborn #285
Merged
Jenkins reborn #285
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ABastionOfSanity
added a commit
that referenced
this pull request
Oct 31, 2022
* initial build and output testing... lots of trial and error * clean up for working (but failing) unit test geth with ubuntu foundation image
ABastionOfSanity
added a commit
that referenced
this pull request
Nov 3, 2022
* initial build and output testing... lots of trial and error * clean up for working (but failing) unit test geth with ubuntu foundation image
ABastionOfSanity
added a commit
that referenced
this pull request
Nov 4, 2022
* Statediff Geth Handle conflicts (#244) * Handle conflicts * Update go mod file versions * Make lint changes Disassociate block number from the indexer object Update ipld-eth-db ref Refactor builder code to make it reusable Use prefix comparison for account selective statediffing Update builder unit tests Add mode to write to CSV files in statediff file writer (#249) * Change file writing mode to csv files * Implement writer interface for file indexer * Implement option for csv or sql in file mode * Close files in CSV writer * Add tests for CSV file mode * Implement CSV file for watched addresses * Separate test configs for CSV and SQL * Refactor common code for file indexer tests Update indexer to include block hash in receipts and logs (#256) * Update indexer to include block hash in receipts and logs * Upgrade ipld-eth-db image in docker-compose to run tests Use watched addresses from direct indexing params by default while serving statediff APIs (#262) * Use watched addresses from direct indexing params in statediff APIs by default * Avoid using indexer object when direct indexing is off * Add nil check before accessing watched addresses from direct indexing params Rebase missed these changes needed at 1.10.20 Flags cleanup for CLI changes and linter complaints Linter appeasements to achieve perfection enforce go 1.18 for check (#267) * enforce go 1.18 for check * tests on 1.18 as well * adding db yml for possible change in docker-compose behavior in yml parsing Add indexer tests for handling non canonical blocks (#254) * Add indexer tests for header and transactions in a non canonical block * Add indexer tests for receipts in a non-canonical block and refactor * Add indexer tests for logs in a non-canonical block * Add indexer tests for state and storage nodes in a non-canonical block * Add indexer tests for non-canonical block at another height * Avoid passing address of a pointer * Update refs in GitHub workflow * Add genesis file path to stack-orchestrator config in GitHub workflow * Add descriptive comments fix non-deterministic ordering in unit tests Refactor indexer tests to avoid duplicate code (#270) * Refactor indexer tests to avoid duplicate code * Refactor file mode indexer tests * Fix expected db stats for sqlx after tx closure * Refactor indexer tests for legacy block * Refactor mainnet indexer tests * Refactor tests for watched addressess methods * Fix query in legacy indexer test rebase and resolve onto 1.10.23... still error out of index related to GetLeafKeys changed trie.Commit behavior was subtle about not not flushing to disk without an Update * no merge nodeset throws nil * linter appeasement Cerc refactor (#281) * first pass cerc refactor in cicd * 1st attempt to publish binary to git.vdb.to from github release * docker build step mangled * docker build step mangled * wrong username for docker login... which still succeeded * circcicd is not cerccicd * bad hostname adding manual override of binary publish to git.vdb.to for development/emergency (#282) Cerc io publish fix (#284) * adding manual override of binary publish to git.vdb.to for development/emergency * Create manual_binary_publish.yaml (#283) * github did not pick up workflow added outside of its UI and I still cannot spell cerc right rawdb helper functions for cold levelDB sync export Jenkins reborn (#285) * initial build and output testing... lots of trial and error * clean up for working (but failing) unit test geth with ubuntu foundation image
ABastionOfSanity
added a commit
that referenced
this pull request
Nov 4, 2022
* eth/protocols/snap: fix problems due to idle-but-busy peers (ethereum#25651) * eth/protocols/snap: throttle trie heal requests when peers DoS us (ethereum#25666) * eth/protocols/snap: throttle trie heal requests when peers DoS us * eth/protocols/snap: lower heal throttle log to debug Co-authored-by: Martin Holst Swende <martin@swende.se> * eth/protocols/snap: fix comment Co-authored-by: Martin Holst Swende <martin@swende.se> * trie: check childrens' existence concurrently for snap heal (ethereum#25694) * eth: fix a rare datarace on CHT challenge reply / shutdown (ethereum#25831) * eth/filters: change filter block to be by-ref (ethereum#26054) This PR changes the block field in the filter to be a pointer, to disambiguate between empty hash and no hash * rpc: handle wrong HTTP batch response length (ethereum#26064) * params: release geth v1.10.26 stable * V1.10.25 statediff v4 wip (#275) * Statediff Geth Handle conflicts (#244) * Handle conflicts * Update go mod file versions * Make lint changes Disassociate block number from the indexer object Update ipld-eth-db ref Refactor builder code to make it reusable Use prefix comparison for account selective statediffing Update builder unit tests Add mode to write to CSV files in statediff file writer (#249) * Change file writing mode to csv files * Implement writer interface for file indexer * Implement option for csv or sql in file mode * Close files in CSV writer * Add tests for CSV file mode * Implement CSV file for watched addresses * Separate test configs for CSV and SQL * Refactor common code for file indexer tests Update indexer to include block hash in receipts and logs (#256) * Update indexer to include block hash in receipts and logs * Upgrade ipld-eth-db image in docker-compose to run tests Use watched addresses from direct indexing params by default while serving statediff APIs (#262) * Use watched addresses from direct indexing params in statediff APIs by default * Avoid using indexer object when direct indexing is off * Add nil check before accessing watched addresses from direct indexing params Rebase missed these changes needed at 1.10.20 Flags cleanup for CLI changes and linter complaints Linter appeasements to achieve perfection enforce go 1.18 for check (#267) * enforce go 1.18 for check * tests on 1.18 as well * adding db yml for possible change in docker-compose behavior in yml parsing Add indexer tests for handling non canonical blocks (#254) * Add indexer tests for header and transactions in a non canonical block * Add indexer tests for receipts in a non-canonical block and refactor * Add indexer tests for logs in a non-canonical block * Add indexer tests for state and storage nodes in a non-canonical block * Add indexer tests for non-canonical block at another height * Avoid passing address of a pointer * Update refs in GitHub workflow * Add genesis file path to stack-orchestrator config in GitHub workflow * Add descriptive comments fix non-deterministic ordering in unit tests Refactor indexer tests to avoid duplicate code (#270) * Refactor indexer tests to avoid duplicate code * Refactor file mode indexer tests * Fix expected db stats for sqlx after tx closure * Refactor indexer tests for legacy block * Refactor mainnet indexer tests * Refactor tests for watched addressess methods * Fix query in legacy indexer test rebase and resolve onto 1.10.23... still error out of index related to GetLeafKeys changed trie.Commit behavior was subtle about not not flushing to disk without an Update * no merge nodeset throws nil * linter appeasement Cerc refactor (#281) * first pass cerc refactor in cicd * 1st attempt to publish binary to git.vdb.to from github release * docker build step mangled * docker build step mangled * wrong username for docker login... which still succeeded * circcicd is not cerccicd * bad hostname adding manual override of binary publish to git.vdb.to for development/emergency (#282) Cerc io publish fix (#284) * adding manual override of binary publish to git.vdb.to for development/emergency * Create manual_binary_publish.yaml (#283) * github did not pick up workflow added outside of its UI and I still cannot spell cerc right rawdb helper functions for cold levelDB sync export Jenkins reborn (#285) * initial build and output testing... lots of trial and error * clean up for working (but failing) unit test geth with ubuntu foundation image * linter problem on comments in version * trying linter appeasement with gofmt output on versions.go Co-authored-by: Martin Holst Swende <martin@swende.se> Co-authored-by: Péter Szilágyi <peterke@gmail.com> Co-authored-by: Jordan Krage <jmank88@gmail.com> Co-authored-by: Felix Lange <fjl@twurst.com>
ABastionOfSanity
added a commit
that referenced
this pull request
Feb 20, 2023
* Statediff Geth Handle conflicts (#244) * Handle conflicts * Update go mod file versions * Make lint changes Disassociate block number from the indexer object Update ipld-eth-db ref Refactor builder code to make it reusable Use prefix comparison for account selective statediffing Update builder unit tests Add mode to write to CSV files in statediff file writer (#249) * Change file writing mode to csv files * Implement writer interface for file indexer * Implement option for csv or sql in file mode * Close files in CSV writer * Add tests for CSV file mode * Implement CSV file for watched addresses * Separate test configs for CSV and SQL * Refactor common code for file indexer tests Update indexer to include block hash in receipts and logs (#256) * Update indexer to include block hash in receipts and logs * Upgrade ipld-eth-db image in docker-compose to run tests Use watched addresses from direct indexing params by default while serving statediff APIs (#262) * Use watched addresses from direct indexing params in statediff APIs by default * Avoid using indexer object when direct indexing is off * Add nil check before accessing watched addresses from direct indexing params Rebase missed these changes needed at 1.10.20 Flags cleanup for CLI changes and linter complaints Linter appeasements to achieve perfection enforce go 1.18 for check (#267) * enforce go 1.18 for check * tests on 1.18 as well * adding db yml for possible change in docker-compose behavior in yml parsing Add indexer tests for handling non canonical blocks (#254) * Add indexer tests for header and transactions in a non canonical block * Add indexer tests for receipts in a non-canonical block and refactor * Add indexer tests for logs in a non-canonical block * Add indexer tests for state and storage nodes in a non-canonical block * Add indexer tests for non-canonical block at another height * Avoid passing address of a pointer * Update refs in GitHub workflow * Add genesis file path to stack-orchestrator config in GitHub workflow * Add descriptive comments fix non-deterministic ordering in unit tests Refactor indexer tests to avoid duplicate code (#270) * Refactor indexer tests to avoid duplicate code * Refactor file mode indexer tests * Fix expected db stats for sqlx after tx closure * Refactor indexer tests for legacy block * Refactor mainnet indexer tests * Refactor tests for watched addressess methods * Fix query in legacy indexer test rebase and resolve onto 1.10.23... still error out of index related to GetLeafKeys changed trie.Commit behavior was subtle about not not flushing to disk without an Update * no merge nodeset throws nil * linter appeasement Co-authored-by: Abdul Rabbani <abdulrabbani00@gmail.com> init iterator package * stripped down Nodeiterator interface * prefix-bound iterator for traversing a specific range of keys * visitor over subtries divided at a specific depth cleanup fmt improve prefix gen use stateful iterator, and allow splitting into powers of 2 rather than 16 doc fill nodeiterator interface iterator slice instead of callback refactor into slice generator, and factory don't prepend (nil, 0], make first bin start at nil Fix termination behavior Boundary must be (a, b] to conform to NodeIterator behavior (stop _at_ next valid key rather than before) go fmt, cleanup move to root path Upgrade geth to v1.10.9. Patches, test, comments bump to v1.10.15-statediff-2.0.0 simplify and clean up export end path field fix edge behavior and constructor add fixture/chaindata add test cases comments cleanup Upgrade geth version and remove vulcanize geth dependency Update go and geth Revert "fix edge behavior and constructor" This reverts commit c134d25. Fix tests for revert Update geth version ``` 08:19:37:~/GitHub/cerc/go-eth-state-node-iterator go test ./... -v === RUN TestMakePaths --- PASS: TestMakePaths (0.00s) === RUN TestIterator === RUN TestIterator/in_bounds === RUN TestIterator/in_bounds/case === RUN TestIterator/in_bounds/case#01 === RUN TestIterator/in_bounds/case#02 === RUN TestIterator/in_bounds/case#03 === RUN TestIterator/in_bounds/case#04 === RUN TestIterator/in_bounds/case#05 === RUN TestIterator/trie_is_covered === RUN TestIterator/trie_is_covered/1_bins === RUN TestIterator/trie_is_covered/2_bins === RUN TestIterator/trie_is_covered/4_bins === RUN TestIterator/trie_is_covered/8_bins === RUN TestIterator/trie_is_covered/16_bins === RUN TestIterator/trie_is_covered/32_bins --- PASS: TestIterator (0.15s) --- PASS: TestIterator/in_bounds (0.00s) --- PASS: TestIterator/in_bounds/case (0.00s) --- PASS: TestIterator/in_bounds/case#01 (0.00s) --- PASS: TestIterator/in_bounds/case#02 (0.00s) --- PASS: TestIterator/in_bounds/case#03 (0.00s) --- PASS: TestIterator/in_bounds/case#04 (0.00s) --- PASS: TestIterator/in_bounds/case#05 (0.00s) --- PASS: TestIterator/trie_is_covered (0.02s) --- PASS: TestIterator/trie_is_covered/1_bins (0.00s) --- PASS: TestIterator/trie_is_covered/2_bins (0.00s) --- PASS: TestIterator/trie_is_covered/4_bins (0.00s) --- PASS: TestIterator/trie_is_covered/8_bins (0.00s) --- PASS: TestIterator/trie_is_covered/16_bins (0.01s) --- PASS: TestIterator/trie_is_covered/32_bins (0.00s) PASS ok github.com/vulcanize/go-eth-state-node-iterator 0.325s ? github.com/vulcanize/go-eth-state-node-iterator/fixture [no test files] ``` Create issues-notion-sync.yml go-ethereum 1.10.20 update with go mod tidy go mody tidy for geth 1.10.21 Store start path in prefix bound iterator object (#17) dependency update to 1.10.23 of vulcanize statediffing geth Add tracker package Moved from snapshot repo repkg move pkg update go mod graphql: return correct logs for tx (ethereum#25612) * graphql: fix tx logs * minor * Use optimized search for selecting tx logs graphql: fixes missing tx logs (ethereum#25745) * graphql: fix tx logs * graphql: refactor test service setup * graphql: add test for tx logs Release Geth v1.10.24 params: set TerminalTotalDifficultyPassed to true (ethereum#25769) * params: set TerminalTotalDifficultyPassed to true * Update params/config.go Co-authored-by: Martin Holst Swende <martin@swende.se> params: release Geth v1.10.25 V1.10.25 statediff v4 wip (#275) * Statediff Geth Handle conflicts (#244) * Handle conflicts * Update go mod file versions * Make lint changes Disassociate block number from the indexer object Update ipld-eth-db ref Refactor builder code to make it reusable Use prefix comparison for account selective statediffing Update builder unit tests Add mode to write to CSV files in statediff file writer (#249) * Change file writing mode to csv files * Implement writer interface for file indexer * Implement option for csv or sql in file mode * Close files in CSV writer * Add tests for CSV file mode * Implement CSV file for watched addresses * Separate test configs for CSV and SQL * Refactor common code for file indexer tests Update indexer to include block hash in receipts and logs (#256) * Update indexer to include block hash in receipts and logs * Upgrade ipld-eth-db image in docker-compose to run tests Use watched addresses from direct indexing params by default while serving statediff APIs (#262) * Use watched addresses from direct indexing params in statediff APIs by default * Avoid using indexer object when direct indexing is off * Add nil check before accessing watched addresses from direct indexing params Rebase missed these changes needed at 1.10.20 Flags cleanup for CLI changes and linter complaints Linter appeasements to achieve perfection enforce go 1.18 for check (#267) * enforce go 1.18 for check * tests on 1.18 as well * adding db yml for possible change in docker-compose behavior in yml parsing Add indexer tests for handling non canonical blocks (#254) * Add indexer tests for header and transactions in a non canonical block * Add indexer tests for receipts in a non-canonical block and refactor * Add indexer tests for logs in a non-canonical block * Add indexer tests for state and storage nodes in a non-canonical block * Add indexer tests for non-canonical block at another height * Avoid passing address of a pointer * Update refs in GitHub workflow * Add genesis file path to stack-orchestrator config in GitHub workflow * Add descriptive comments fix non-deterministic ordering in unit tests Refactor indexer tests to avoid duplicate code (#270) * Refactor indexer tests to avoid duplicate code * Refactor file mode indexer tests * Fix expected db stats for sqlx after tx closure * Refactor indexer tests for legacy block * Refactor mainnet indexer tests * Refactor tests for watched addressess methods * Fix query in legacy indexer test rebase and resolve onto 1.10.23... still error out of index related to GetLeafKeys changed trie.Commit behavior was subtle about not not flushing to disk without an Update * no merge nodeset throws nil * linter appeasement Co-authored-by: Abdul Rabbani <abdulrabbani00@gmail.com> Cerc refactor (#281) * first pass cerc refactor in cicd * 1st attempt to publish binary to git.vdb.to from github release * docker build step mangled * docker build step mangled * wrong username for docker login... which still succeeded * circcicd is not cerccicd * bad hostname adding manual override of binary publish to git.vdb.to for development/emergency (#282) Cerc io publish fix (#284) * adding manual override of binary publish to git.vdb.to for development/emergency * Create manual_binary_publish.yaml (#283) * github did not pick up workflow added outside of its UI and I still cannot spell cerc right rawdb helper functions for cold levelDB sync export Jenkins reborn (#285) * initial build and output testing... lots of trial and error * clean up for working (but failing) unit test geth with ubuntu foundation image V1.10.26 rebase wip (#289) * eth/protocols/snap: fix problems due to idle-but-busy peers (ethereum#25651) * eth/protocols/snap: throttle trie heal requests when peers DoS us (ethereum#25666) * eth/protocols/snap: throttle trie heal requests when peers DoS us * eth/protocols/snap: lower heal throttle log to debug Co-authored-by: Martin Holst Swende <martin@swende.se> * eth/protocols/snap: fix comment Co-authored-by: Martin Holst Swende <martin@swende.se> * trie: check childrens' existence concurrently for snap heal (ethereum#25694) * eth: fix a rare datarace on CHT challenge reply / shutdown (ethereum#25831) * eth/filters: change filter block to be by-ref (ethereum#26054) This PR changes the block field in the filter to be a pointer, to disambiguate between empty hash and no hash * rpc: handle wrong HTTP batch response length (ethereum#26064) * params: release geth v1.10.26 stable * V1.10.25 statediff v4 wip (#275) * Statediff Geth Handle conflicts (#244) * Handle conflicts * Update go mod file versions * Make lint changes Disassociate block number from the indexer object Update ipld-eth-db ref Refactor builder code to make it reusable Use prefix comparison for account selective statediffing Update builder unit tests Add mode to write to CSV files in statediff file writer (#249) * Change file writing mode to csv files * Implement writer interface for file indexer * Implement option for csv or sql in file mode * Close files in CSV writer * Add tests for CSV file mode * Implement CSV file for watched addresses * Separate test configs for CSV and SQL * Refactor common code for file indexer tests Update indexer to include block hash in receipts and logs (#256) * Update indexer to include block hash in receipts and logs * Upgrade ipld-eth-db image in docker-compose to run tests Use watched addresses from direct indexing params by default while serving statediff APIs (#262) * Use watched addresses from direct indexing params in statediff APIs by default * Avoid using indexer object when direct indexing is off * Add nil check before accessing watched addresses from direct indexing params Rebase missed these changes needed at 1.10.20 Flags cleanup for CLI changes and linter complaints Linter appeasements to achieve perfection enforce go 1.18 for check (#267) * enforce go 1.18 for check * tests on 1.18 as well * adding db yml for possible change in docker-compose behavior in yml parsing Add indexer tests for handling non canonical blocks (#254) * Add indexer tests for header and transactions in a non canonical block * Add indexer tests for receipts in a non-canonical block and refactor * Add indexer tests for logs in a non-canonical block * Add indexer tests for state and storage nodes in a non-canonical block * Add indexer tests for non-canonical block at another height * Avoid passing address of a pointer * Update refs in GitHub workflow * Add genesis file path to stack-orchestrator config in GitHub workflow * Add descriptive comments fix non-deterministic ordering in unit tests Refactor indexer tests to avoid duplicate code (#270) * Refactor indexer tests to avoid duplicate code * Refactor file mode indexer tests * Fix expected db stats for sqlx after tx closure * Refactor indexer tests for legacy block * Refactor mainnet indexer tests * Refactor tests for watched addressess methods * Fix query in legacy indexer test rebase and resolve onto 1.10.23... still error out of index related to GetLeafKeys changed trie.Commit behavior was subtle about not not flushing to disk without an Update * no merge nodeset throws nil * linter appeasement Cerc refactor (#281) * first pass cerc refactor in cicd * 1st attempt to publish binary to git.vdb.to from github release * docker build step mangled * docker build step mangled * wrong username for docker login... which still succeeded * circcicd is not cerccicd * bad hostname adding manual override of binary publish to git.vdb.to for development/emergency (#282) Cerc io publish fix (#284) * adding manual override of binary publish to git.vdb.to for development/emergency * Create manual_binary_publish.yaml (#283) * github did not pick up workflow added outside of its UI and I still cannot spell cerc right rawdb helper functions for cold levelDB sync export Jenkins reborn (#285) * initial build and output testing... lots of trial and error * clean up for working (but failing) unit test geth with ubuntu foundation image * linter problem on comments in version * trying linter appeasement with gofmt output on versions.go Co-authored-by: Martin Holst Swende <martin@swende.se> Co-authored-by: Péter Szilágyi <peterke@gmail.com> Co-authored-by: Jordan Krage <jmank88@gmail.com> Co-authored-by: Felix Lange <fjl@twurst.com> bad tag variable in publish workflow statediff: Use delayed Tx A lazy Tx object that caches statements, then builds/commits the whole underlying Tx at Commit() bump minor statediff version indexer/database/postgres.DB updates to toggle no-UPSERT mode integrate upsert mode toggle into CLI bump statediff patch version remove known_gaps integration logging tweaks Additional statediff metrics and logging. (#304) Add --statediff.db.logstatements option. (#307) * Add --statediff.db.logstatements option. Move message from Debug to Trace. (#308) Add driver detail to flag. (#310) Change behavior of WriteStateDiffAt to async (#306) - returns job ID and provides status update stream - won't start redundant jobs remove logrus, use geth logging pkg Add a new option to the config for capping the maximum difficulty. (#312) * consensus/ethash/consensus.go * Add a new param to the config for capping the maximum difficulty. This is useful for speeding up block creation on a test network. Add finer-grained logging and timers for statediffing. (#313) * More logging for statediffs. * Add count * Tweak logging * lint fix: Unify and fix indexer metrics. (#314) Fix duration logging and add new metrics for fine-grained processing stats. (#315) Add timers and update StandardTimer to include total time. (#316) * Add timers and update StandardTimer to include total time. * Even more timers. Move noisiest timer message to trace. (#317)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Mergeable with default.