IAVL Iterator #440

mconcat · 2021-10-13T14:37:53Z

adapt cosmos-sdk style iterator to iavl tree

ValarDragon · 2021-10-14T23:16:04Z

immutable_tree.go

+  next traversal
+}
+
+func (t *ImmutableTree) Iterator(start, end []byte, ascending bool) *Iterator {


So the goal is the SDK should switch to using this, callback-free, iterator right

Can you add a comment to the Iterate function that its iteration with a callback, which would hopefully make this division of roles simpler

ValarDragon · 2021-10-18T05:24:05Z

immutable_tree.go

+
+	valid bool
+
+	next traversal


Can you add a comment explaining the function-based scheme for traversing the tree that you explained to me?

ValarDragon · 2021-10-18T05:25:23Z

node.go

+	return stop
+}
+
+func (node *Node) traversal(t *ImmutableTree, start, end []byte, ascending bool, inclusive bool, depth uint8, post bool, condition bool, continuation traversal) traversal {


Can this function get a comment explaining it? (e.g. what is post, condition, continuation)

Also can the arguments be put into multiple lines?

ValarDragon · 2021-10-19T06:58:49Z

Is the idea with testing for this PR that the old callback functionality had tests, and that being correct implies correctness of the traversal function and Iterator.Next()?

tac0turtle · 2021-10-19T14:24:28Z

is this meant to go to master or 0.6.x?

mconcat · 2021-10-19T17:30:34Z

Is the idea with testing for this PR that the old callback functionality had tests, and that being correct implies correctness of the traversal function and Iterator.Next()?

traversal is checked by the existing tests, Iterator is not. However it is being tested indirectly by the SDK side PR(as the sdk iavlstore is using Iterator). If you think it is better to test them explicitly in this PR, I can add that.

mconcat · 2021-10-19T17:31:21Z

is this meant to go to master or 0.6.x?

I am not sure, do you have any idea? @ValarDragon

ValarDragon · 2021-10-19T17:52:38Z

Should go into a patch release ASAP, so probably 6.x & master?

immutable_tree.go

node.go

ValarDragon · 2021-10-24T05:30:25Z

I think just getting a 2x with this is huge, since after this I think it will also be easier to debug further improvements. I suspect its in large part just hard to detect. Even if not, and its only 2x, this will make it much easier to debug.

Heres some benchmarks from Osmosis epoch after switching to leveldb (not using this branch):

Here we see the logic for distribution using 30seconds, iavl.Commit taking 100 seconds, but IAVL iterator in a second thread is taking 250 seconds! Furthermore, the pthread here is spending 110 seconds waiting in the main thread (pthread_cond_wait), and 160seconds in waking up the thread. (runtime.semawakeup) (Not shown, Level DB compaction taking 240s in a second blocking thread)

So shaving the pthread & wakeup is a total of 270seconds of processing work vs 250 seconds of base function work, so 270/520 ~= 50%, so it would be an ~2x speedup. At least as far as this large benchmark on leveldb is concerned, it does look the best is 2x here. I was wrong previously when I thought it was a much larger factor (perhaps this is only true on memdb backends & uncommitted leveldb? We were previously benchmarking on an uncommitted leveldb)

So actually in this more compreshensive benchmark using leveldb, its only 2x.

leveldb / remaining iavliterator flamegraph (likely untouched by this PR)

Looking at that flamegraph, I don't think we need to be too caught up on optimizing this PR actually -- the important thing is to mitigate the need for file opening in lockstep

robert-zaremba · 2021-10-24T20:24:01Z

I wouldn't spend too much time on IAVL. We are moving towards ADR-40 based store, where iterations will be way faster. I'm sure we can optimize SMT implementation.

ValarDragon · 2021-10-24T21:02:53Z

Its going to be January-March before SMT lands on most chains, no? (Needs SMT feature completion, 2 IBC upgrades on all chains to handle SMT & IAVL proofs, and state sync updates) We need to fix Osmosis epoch problems, and querying problems for all the chains much sooner, in order to not risk node failures. The 2x here is crucial.

Fully agreed that the problems are due to leveldb keys being node hashes, instead of the actual key address. These should be stored sequentially, which is happening as part of SMT!

mconcat · 2021-10-28T15:17:17Z

Documentation added and ready for review

robert-zaremba · 2021-10-28T16:55:02Z

@mconcat I've already did a first review few weeks ago. Could you go through comments, respond & update and then we will do a second round. OK?

mconcat · 2021-11-02T14:18:05Z

All resolved!

tac0turtle · 2021-11-08T16:15:53Z

@robert-zaremba can we get your eyes on this?

tac0turtle · 2021-11-09T10:15:31Z

merging this, I will try to get this released shortly

## Description Closes: #10335 This PR adds a custom version of cosmos/iavl(cosmos/iavl#440) that removes the usage of channel-based approach on iterating IAVL tree. replaces #10404 --- ### Author Checklist *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... - [ ] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] added `!` to the type prefix if API or client breaking change - [ ] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting)) - [ ] provided a link to the relevant issue or specification - [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules) - [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing) - [ ] added a changelog entry to `CHANGELOG.md` - [ ] included comments for [documenting Go code](https://blog.golang.org/godoc) - [ ] updated the relevant documentation or specification - [ ] reviewed "Files changed" and left comments if necessary - [ ] confirmed all CI checks have passed ### Reviewers Checklist *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... - [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] confirmed `!` in the type prefix if API or client breaking change - [ ] confirmed all author checklist items have been addressed - [ ] reviewed state machine logic - [ ] reviewed API design and naming - [ ] reviewed documentation is accurate - [ ] reviewed tests and test coverage - [ ] manually tested (if applicable)

## Description Closes: #10335 This PR adds a custom version of cosmos/iavl(cosmos/iavl#440) that removes the usage of channel-based approach on iterating IAVL tree. replaces #10404 --- ### Author Checklist *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... - [ ] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] added `!` to the type prefix if API or client breaking change - [ ] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting)) - [ ] provided a link to the relevant issue or specification - [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules) - [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing) - [ ] added a changelog entry to `CHANGELOG.md` - [ ] included comments for [documenting Go code](https://blog.golang.org/godoc) - [ ] updated the relevant documentation or specification - [ ] reviewed "Files changed" and left comments if necessary - [ ] confirmed all CI checks have passed ### Reviewers Checklist *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... - [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] confirmed `!` in the type prefix if API or client breaking change - [ ] confirmed all author checklist items have been addressed - [ ] reviewed state machine logic - [ ] reviewed API design and naming - [ ] reviewed documentation is accurate - [ ] reviewed tests and test coverage - [ ] manually tested (if applicable) (cherry picked from commit b5dd352) # Conflicts: # go.mod # go.sum

* chore: Iavl iterator (#10544)  ## Description Closes: #10335 This PR adds a custom version of cosmos/iavl(cosmos/iavl#440) that removes the usage of channel-based approach on iterating IAVL tree. replaces #10404 --- ### Author Checklist *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... - [ ] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] added `!` to the type prefix if API or client breaking change - [ ] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting)) - [ ] provided a link to the relevant issue or specification - [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules) - [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing) - [ ] added a changelog entry to `CHANGELOG.md` - [ ] included comments for [documenting Go code](https://blog.golang.org/godoc) - [ ] updated the relevant documentation or specification - [ ] reviewed "Files changed" and left comments if necessary - [ ] confirmed all CI checks have passed ### Reviewers Checklist *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... - [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] confirmed `!` in the type prefix if API or client breaking change - [ ] confirmed all author checklist items have been addressed - [ ] reviewed state machine logic - [ ] reviewed API design and naming - [ ] reviewed documentation is accurate - [ ] reviewed tests and test coverage - [ ] manually tested (if applicable) (cherry picked from commit b5dd352) # Conflicts: # go.mod # go.sum * solve conflicts Co-authored-by: Marko <marbar3778@yahoo.com>

## Description Closes: cosmos#10335 This PR adds a custom version of cosmos/iavl(cosmos/iavl#440) that removes the usage of channel-based approach on iterating IAVL tree. replaces cosmos#10404 --- ### Author Checklist *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... - [ ] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] added `!` to the type prefix if API or client breaking change - [ ] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting)) - [ ] provided a link to the relevant issue or specification - [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules) - [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing) - [ ] added a changelog entry to `CHANGELOG.md` - [ ] included comments for [documenting Go code](https://blog.golang.org/godoc) - [ ] updated the relevant documentation or specification - [ ] reviewed "Files changed" and left comments if necessary - [ ] confirmed all CI checks have passed ### Reviewers Checklist *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... - [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] confirmed `!` in the type prefix if API or client breaking change - [ ] confirmed all author checklist items have been addressed - [ ] reviewed state machine logic - [ ] reviewed API design and naming - [ ] reviewed documentation is accurate - [ ] reviewed tests and test coverage - [ ] manually tested (if applicable)

* chore: Iavl iterator (#10544)  ## Description Closes: #10335 This PR adds a custom version of cosmos/iavl(cosmos/iavl#440) that removes the usage of channel-based approach on iterating IAVL tree. replaces cosmos/cosmos-sdk#10404

* chores: revert bls integration * fix: --home flag parsing (backport #10226) (#10272) * fix: --home flag parsing (#10226)  ## Description Closes: #XXXX  * fix: null guard for tx fee amounts (backport #10327) (#10343) * fix: null guard for tx fee amounts (#10327) ## Description It is possible to submit a TX with a fees object containing a Coin with a nil amount. This results in a rather cryptic redacted panic response when the basic validation checks fee Coins for negative amounts. This PR adds an additional check for nil to provide a friendlier error message. * perf: Only do memory allocation when zero coin is found (backport #10339) (#10362) * perf: Only do memory allocation when zero coin is found (#10339) ## Description Closes: #10333 Added a loop that checks for zero coins before making any memory allocations. If no zero coins are found, just return the `coins` slice as is. If a zero coin is found, then allocate the new array, stop looking for the first zero, and restart the loop to "remove" the 0's. * build(deps): bump github.com/tendermint/tendermint from 0.34.13 to 0.34.14 (backport #10357) (#10376) * build(deps): bump github.com/tendermint/tendermint from 0.34.13 to 0.34.14 (#10357) Bumps [github.com/tendermint/tendermint](https://github.com/tendermint/tendermint) from 0.34.13 to 0.34.14. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/tendermint/tendermint/releases">github.com/tendermint/tendermint's releases</a>.</em></p> <blockquote> <h2>0.34.14 (WARNING: BETA SOFTWARE)</h2> <p><a href="https://github.com/tendermint/tendermint/blob/v0.34.14/CHANGELOG.md#v0.34.14">https://github.com/tendermint/tendermint/blob/v0.34.14/CHANGELOG.md#v0.34.14</a></p> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/tendermint/tendermint/blob/master/CHANGELOG.md">github.com/tendermint/tendermint's changelog</a>.</em></p> <blockquote> <h2>v0.34.14</h2> <p>This release backports the <code>rollback</code> feature to allow recovery in the event of an incorrect app hash.</p> <h3>FEATURES</h3> <ul> <li><a href="https://github-redirect.dependabot.com/tendermint/tendermint/pull/6982">#6982</a> The tendermint binary now has built-in suppport for running the end-to-end test application (with state sync support) (<a href="https://github.com/cmwaters"><code>@cmwaters</code></a>).</li> <li>[cli] <a href="https://github-redirect.dependabot.com/tendermint/tendermint/pull/7033">#7033</a> Add a <code>rollback</code> command to rollback to the previous tendermint state. This may be useful in the event of non-determinstic app hash or when reverting an upgrade. <a href="https://github.com/cmwaters"><code>@cmwaters</code></a></li> </ul> <h3>IMPROVEMENTS</h3> <ul> <li><a href="https://github-redirect.dependabot.com/tendermint/tendermint/pull/7104">#7103</a> Remove IAVL dependency (backport of <a href="https://github-redirect.dependabot.com/tendermint/tendermint/issues/6550">#6550</a>) (<a href="https://github.com/cmwaters"><code>@cmwaters</code></a>)</li> </ul> <h3>BUG FIXES</h3> <ul> <li><a href="https://github-redirect.dependabot.com/tendermint/tendermint/pull/7057">#7057</a> Import Postgres driver support for the psql indexer (<a href="https://github.com/creachadair"><code>@creachadair</code></a>).</li> <li>[ABCI] <a href="https://github-redirect.dependabot.com/tendermint/tendermint/issues/7110">#7110</a> Revert "change client to use multi-reader mutexes (<a href="https://github-redirect.dependabot.com/tendermint/tendermint/issues/6873">#6873</a>)" (<a href="https://github.com/tychoish"><code>@tychoish</code></a>).</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/tendermint/tendermint/commit/85870def7b628effad73af942e638bbddf2ba8fd"><code>85870de</code></a> release: prepare changelog for 0.34.14 (<a href="https://github-redirect.dependabot.com/tendermint/tendermint/issues/7105">#7105</a>)</li> <li><a href="https://github.com/tendermint/tendermint/commit/ff2758b32e637b50734ad5ff1d0b24403af0deb2"><code>ff2758b</code></a> dep: remove IAVL dependency (backport <a href="https://github-redirect.dependabot.com/tendermint/tendermint/issues/6550">#6550</a>) (<a href="https://github-redirect.dependabot.com/tendermint/tendermint/issues/7104">#7104</a>)</li> <li><a href="https://github.com/tendermint/tendermint/commit/a82cb7dcda832b8c6475481bf404257a56aa2d18"><code>a82cb7d</code></a> Revert "abci: change client to use multi-reader mutexes (<a href="https://github-redirect.dependabot.com/tendermint/tendermint/issues/6306">#6306</a>)" (backport <a href="https://github-redirect.dependabot.com/tendermint/tendermint/issues/7">#7</a>...</li> <li><a href="https://github.com/tendermint/tendermint/commit/1dfb3451eaaa438067dadb6835e9aa948689f39e"><code>1dfb345</code></a> e2e: light nodes should use builtin abci app (<a href="https://github-redirect.dependabot.com/tendermint/tendermint/issues/7095">#7095</a>) (<a href="https://github-redirect.dependabot.com/tendermint/tendermint/issues/7096">#7096</a>)</li> <li><a href="https://github.com/tendermint/tendermint/commit/9f13b9b083da0ec45a48e57d5a283afdf698ba5b"><code>9f13b9b</code></a> e2e: abci protocol should be consistent across networks (backport <a href="https://github-redirect.dependabot.com/tendermint/tendermint/issues/7078">#7078</a>) (<a href="https://github-redirect.dependabot.com/tendermint/tendermint/issues/7085">#7085</a>)</li> <li><a href="https://github.com/tendermint/tendermint/commit/16ba782fa6549ea1e50a7fe7addb538ff15bf619"><code>16ba782</code></a> cli: allow node operator to rollback last state (backport <a href="https://github-redirect.dependabot.com/tendermint/tendermint/issues/7033">#7033</a>) (<a href="https://github-redirect.dependabot.com/tendermint/tendermint/issues/7080">#7080</a>)</li> <li><a href="https://github.com/tendermint/tendermint/commit/474ed04273bade74df583c7bfdc63e6e1a70a919"><code>474ed04</code></a> Import Postgres driver support for the psql indexer (backport). (<a href="https://github-redirect.dependabot.com/tendermint/tendermint/issues/7057">#7057</a>)</li> <li><a href="https://github.com/tendermint/tendermint/commit/2d8287d0f70962cc1d7bda0250def18962abf282"><code>2d8287d</code></a> e2e: allow running of single node using the e2e app (backport) (<a href="https://github-redirect.dependabot.com/tendermint/tendermint/issues/7024">#7024</a>)</li> <li><a href="https://github.com/tendermint/tendermint/commit/294a9695b4a768b4c8a9cb71c3562cdbff58e64c"><code>294a969</code></a> e2e: backport minor reliability improvements (<a href="https://github-redirect.dependabot.com/tendermint/tendermint/issues/6967">#6967</a>)</li> <li>See full diff in <a href="https://github.com/tendermint/tendermint/compare/v0.34.13...v0.34.14">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=github.com/tendermint/tendermint&package-manager=go_modules&previous-version=0.34.13&new-version=0.34.14)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) * fix: unmarshalling issue with multisig keys in master (backport #10061) (#10113) * fix: unmarshalling issue with multisig keys in master (#10061) (cherry picked from commit 3d3bc7c) # Conflicts: # CHANGELOG.md # crypto/keys/multisig/multisig_test.go * fix conflicts * fix conflicts * Update crypto/keys/multisig/multisig_test.go * Removed unused imports * fix changelog * Update CHANGELOG.md * Fix TestLegacyMultisig test (#10275) Co-authored-by: Henrik Aasted Sørensen <has@bitcraft.dk> Co-authored-by: Aleksandr Bezobchuk <aleks.bezobchuk@gmail.com> Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> Co-authored-by: Robert Zaremba <robert@zaremba.ch> Co-authored-by: Mario Karagiorgas <salem8@gmail.com> * fix: query account balance by ibc denom (backport #10394) (#10524) * fix: query account balance by ibc denom (#10394)  ## Description Closes: #10381  * chore: Iavl iterator (backport #10544) (#10546) * chore: Iavl iterator (#10544)  ## Description Closes: #10335 This PR adds a custom version of cosmos/iavl(cosmos/iavl#440) that removes the usage of channel-based approach on iterating IAVL tree. replaces cosmos/cosmos-sdk#10404 * build(deps): Bump github.com/cosmos/iavl from 0.17.2 to 0.17.3 (backport #10654) (#10664) * build(deps): Bump github.com/cosmos/iavl from 0.17.2 to 0.17.3 (#10654) Bumps [github.com/cosmos/iavl](https://github.com/cosmos/iavl) from 0.17.2 to 0.17.3. - [Release notes](https://github.com/cosmos/iavl/releases) - [Changelog](https://github.com/cosmos/iavl/blob/v0.17.3/CHANGELOG.md) - [Commits](cosmos/iavl@v0.17.2...v0.17.3) * fix: Add Events to TxResponse (backport #10630) (#10644) * fix: Add Events to TxResponse (#10630) (cherry picked from commit c4bedf8) # Conflicts: # CHANGELOG.md # go.sum # types/abci.pb.go # types/result.go * Michael sucks * Who Let the Bugs Out?? * Reinventing the wheel. Again. * No changes made Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com> Co-authored-by: Aleksandr Bezobchuk <aleks.bezobchuk@gmail.com> * fix: bech32 address test to satisfy the specification (#10163) (#10164) (#10665) ## Description Closes: #10163  * build(deps): Use self-maintained btcutil (#10082) (backport #10201) (#10622) * build(deps): Use self-maintained btcutil (#10082) (#10201) ## Description Closes: #10082 * chore: v0.42.11 changelog and release notes (#10695) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Henrik Aasted Sørensen <has@bitcraft.dk> Co-authored-by: Aleksandr Bezobchuk <aleks.bezobchuk@gmail.com> Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> Co-authored-by: Robert Zaremba <robert@zaremba.ch> Co-authored-by: Mario Karagiorgas <salem8@gmail.com> Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com>

* [capricorn] cosmos-sdk v0.42.9 to v0.42.10 (#117) * fix: file keyring fails to add/import/export keys when input is not stdin (fix #9566) (backport #9821) (#9880) * fix: file keyring fails to add/import/export keys when input is not stdin (fix #9566) (#9821) ## Description Add a test case to reproduce the issue described in #9566. The test currently fails, and I've pointed some possible solutions over cosmos/cosmos-sdk#9566 (comment). But I feel this needs more works in order to provide a more robust solution. I'll keep poking at better options, but taking any pointers if anyone has ideas. (cherry picked from commit f479b51) # Conflicts: # client/keys/add.go # client/keys/add_test.go # client/keys/export_test.go * fix: merge conflict backporting pr-9821 (#9888) Co-authored-by: daeMOn <flavien.binet@gmail.com> * fix: Fix CLI query tx docs for acc/seq (#9942) (#9951)  ## Description ref: user from #validators-verfied channel on Cosmos Discord  * build(deps): bump TM to v0.34.12 (backport #9956) (#9961) * perf: Remove telemetry from wrappings of store (backport #10077) (#10084) * docs: update to v0.44 version tag (#10069) (#10097) (cherry picked from commit d6c3017) # Conflicts: # docs/.vuepress/enhanceApp.js Co-authored-by: Ryan Christoffersen <12519942+ryanchristo@users.noreply.github.com> * fix!: update ABCI query to use request height (backport #9879) (#10144) * fix: use keyring in config for add-genesis-account cmd (backport #9969) (#10065) * fix: use keyring in config for add-genesis-account cmd (#9969)  ## Description ref: #9644 +use the keyring backend (if specified) in the binary config for the add-genesis-account command +update existing test to check for the case of keyring in config * build(deps): bump github.com/tendermint/tendermint from 0.34.12 to 0.34.13 (backport #10106) (#10107) * build(deps): bump github.com/tendermint/tendermint (#10106) (cherry picked from commit 8ee5e50) * fix go.sum Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Robert Zaremba <robert@zaremba.ch> * perf: Make CacheKV store interleaved iterator and insertion not O(n^2) (backport #10026) (#10114) * perf: Make CacheKV store interleaved iterator and insertion not O(n^2) (#10026) (cherry picked from commit 28bf2c1) # Conflicts: # CHANGELOG.md # store/cachekv/store.go * Fix merge conflict * fix changelog Co-authored-by: Dev Ojha <ValarDragon@users.noreply.github.com> Co-authored-by: ValarDragon <dojha12@gmail.com> Co-authored-by: Robert Zaremba <robert@zaremba.ch> * chore: bump IAVL version (backport #10040) (#10186) * chore: bump IAVL version (#10040) * chore: bump IAVL version * test master * update IAVL version * adding missing go.sum entry * adding missing entries * fix go.sum * clear go.sum * fixing go.sum Co-authored-by: marbar3778 <marbar3778@yahoo.com> (cherry picked from commit 693ffb1) * add changelog update * tidy go.mod Co-authored-by: Robert Zaremba <robert@zaremba.ch> * feat: backport reject redundant transactions from ibc-go (#10211) * backport refund of redundant packets from ibc-go * update changelog * address review comments and lint errors * added nolint Co-authored-by: Carlos Rodriguez <crodveg@gmail.com> * chore: Changelog and Release Notes for 0.42.10 (#10242) * chores: bump tendermint version * fix: add required helper for group module Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Ryan Christoffersen <12519942+ryanchristo@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Robert Zaremba <robert@zaremba.ch> Co-authored-by: Dev Ojha <ValarDragon@users.noreply.github.com> Co-authored-by: ValarDragon <dojha12@gmail.com> Co-authored-by: Carlos Rodriguez <crodveg@yahoo.es> Co-authored-by: Carlos Rodriguez <crodveg@gmail.com> Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> * fix: add hack to handle group v2 messages in baseapp (#118) * fix: add hack to handle group v2 messages in baseapp * chores: disable lint warning * feat: backport caelum bls on v0.42 sdk (#119) * Bls (#104) * bls signature for basic account * benchmark for bls and ed25519 * added bls sig verify cost to genesis * Revert "Merge branch 'fetchai:master' into bls" This reverts commit a5dd8ea, reversing changes made to 082e071. * format using go tools * nuisance golangci-lint errors * Bls (#105) * bls signature for basic account * benchmark for bls and ed25519 * added bls sig verify cost to genesis * Revert "Merge branch 'fetchai:master' into bls" This reverts commit a5dd8ea, reversing changes made to 082e071. * format using go tools * nuisance golangci-lint errors * POP interfaces in accounts and authentication * add bls multsig operations * fixed golangci-lint error * changes after comments * change codespace for invalid pop error * extend migrate with bls cost * set bls cost multiplier * fix ante test errors * feat: Caelum (#111) * bls signature for basic account * benchmark for bls and ed25519 * added bls sig verify cost to genesis * Revert "Merge branch 'fetchai:master' into bls" This reverts commit a5dd8ea, reversing changes made to 082e071. * format using go tools * nuisance golangci-lint errors * Bls (#104) * bls signature for basic account * benchmark for bls and ed25519 * added bls sig verify cost to genesis * Revert "Merge branch 'fetchai:master' into bls" This reverts commit a5dd8ea, reversing changes made to 082e071. * format using go tools * nuisance golangci-lint errors * POP interfaces in accounts and authentication * add bls multsig operations * fixed golangci-lint error * changes after comments * initial commit from regen-ledger/x/group v1.0.0 * minor changes to bls12381 key generation * initial commit from regen-ledger/proto/regen/group v1.0.0 * group module compatibility for fetchai cosomos-sdk * add bls account restriction to group members * fix bug in setting pop * make msg uniqueness checking optional * add bls basic/aggregate vote * add checking on empty messages/public keys * add gas caclulation/consumption for verifying aggregated votes * minor change to gas calculation for voteagg * initial commit for orm and types from regen-ledger v2.0.0-beta1 * upgrade testsuite to regen-ledger v2.0.0-beta1 * make bls requirement for group members optional * add tests for bls related group operations * client and server for poll and aggregated votes and integration tests * fix bls related test errors * fix proto-lint errors * goimport format * proto comments * update blst to v0.3.5 and more tests for bls * Update x/auth/ante/sigverify.go Co-authored-by: daeMOn <flavien.binet@gmail.com> * Update x/group/client/util.go Co-authored-by: daeMOn <flavien.binet@gmail.com> Co-authored-by: daeMOn <flavien.binet@gmail.com> * fix: bls12381 compat with sdk v0.42 * fix: buf lint / breaking command changes * fix: staking test want more gas * chores: drop arm from CI test matrix * chores: fix lint Co-authored-by: kitounliu <58184672+kitounliu@users.noreply.github.com> Co-authored-by: kitty <jia.liu@fetch.ai> * fix: more gas for simulations. Disable test caching Co-authored-by: kitounliu <58184672+kitounliu@users.noreply.github.com> Co-authored-by: kitty <jia.liu@fetch.ai> * chores: revert bls integration (#121) * chores: bump sdk v0.42.11 (#123) * chores: revert bls integration * fix: --home flag parsing (backport #10226) (#10272) * fix: --home flag parsing (#10226)  ## Description Closes: #XXXX  * fix: null guard for tx fee amounts (backport #10327) (#10343) * fix: null guard for tx fee amounts (#10327) ## Description It is possible to submit a TX with a fees object containing a Coin with a nil amount. This results in a rather cryptic redacted panic response when the basic validation checks fee Coins for negative amounts. This PR adds an additional check for nil to provide a friendlier error message. * perf: Only do memory allocation when zero coin is found (backport #10339) (#10362) * perf: Only do memory allocation when zero coin is found (#10339) ## Description Closes: #10333 Added a loop that checks for zero coins before making any memory allocations. If no zero coins are found, just return the `coins` slice as is. If a zero coin is found, then allocate the new array, stop looking for the first zero, and restart the loop to "remove" the 0's. * build(deps): bump github.com/tendermint/tendermint from 0.34.13 to 0.34.14 (backport #10357) (#10376) * build(deps): bump github.com/tendermint/tendermint from 0.34.13 to 0.34.14 (#10357) Bumps [github.com/tendermint/tendermint](https://github.com/tendermint/tendermint) from 0.34.13 to 0.34.14. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/tendermint/tendermint/releases">github.com/tendermint/tendermint's releases</a>.</em></p> <blockquote> <h2>0.34.14 (WARNING: BETA SOFTWARE)</h2> <p><a href="https://github.com/tendermint/tendermint/blob/v0.34.14/CHANGELOG.md#v0.34.14">https://github.com/tendermint/tendermint/blob/v0.34.14/CHANGELOG.md#v0.34.14</a></p> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/tendermint/tendermint/blob/master/CHANGELOG.md">github.com/tendermint/tendermint's changelog</a>.</em></p> <blockquote> <h2>v0.34.14</h2> <p>This release backports the <code>rollback</code> feature to allow recovery in the event of an incorrect app hash.</p> <h3>FEATURES</h3> <ul> <li><a href="https://github-redirect.dependabot.com/tendermint/tendermint/pull/6982">#6982</a> The tendermint binary now has built-in suppport for running the end-to-end test application (with state sync support) (<a href="https://github.com/cmwaters"><code>@cmwaters</code></a>).</li> <li>[cli] <a href="https://github-redirect.dependabot.com/tendermint/tendermint/pull/7033">#7033</a> Add a <code>rollback</code> command to rollback to the previous tendermint state. This may be useful in the event of non-determinstic app hash or when reverting an upgrade. <a href="https://github.com/cmwaters"><code>@cmwaters</code></a></li> </ul> <h3>IMPROVEMENTS</h3> <ul> <li><a href="https://github-redirect.dependabot.com/tendermint/tendermint/pull/7104">#7103</a> Remove IAVL dependency (backport of <a href="https://github-redirect.dependabot.com/tendermint/tendermint/issues/6550">#6550</a>) (<a href="https://github.com/cmwaters"><code>@cmwaters</code></a>)</li> </ul> <h3>BUG FIXES</h3> <ul> <li><a href="https://github-redirect.dependabot.com/tendermint/tendermint/pull/7057">#7057</a> Import Postgres driver support for the psql indexer (<a href="https://github.com/creachadair"><code>@creachadair</code></a>).</li> <li>[ABCI] <a href="https://github-redirect.dependabot.com/tendermint/tendermint/issues/7110">#7110</a> Revert "change client to use multi-reader mutexes (<a href="https://github-redirect.dependabot.com/tendermint/tendermint/issues/6873">#6873</a>)" (<a href="https://github.com/tychoish"><code>@tychoish</code></a>).</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/tendermint/tendermint/commit/85870def7b628effad73af942e638bbddf2ba8fd"><code>85870de</code></a> release: prepare changelog for 0.34.14 (<a href="https://github-redirect.dependabot.com/tendermint/tendermint/issues/7105">#7105</a>)</li> <li><a href="https://github.com/tendermint/tendermint/commit/ff2758b32e637b50734ad5ff1d0b24403af0deb2"><code>ff2758b</code></a> dep: remove IAVL dependency (backport <a href="https://github-redirect.dependabot.com/tendermint/tendermint/issues/6550">#6550</a>) (<a href="https://github-redirect.dependabot.com/tendermint/tendermint/issues/7104">#7104</a>)</li> <li><a href="https://github.com/tendermint/tendermint/commit/a82cb7dcda832b8c6475481bf404257a56aa2d18"><code>a82cb7d</code></a> Revert "abci: change client to use multi-reader mutexes (<a href="https://github-redirect.dependabot.com/tendermint/tendermint/issues/6306">#6306</a>)" (backport <a href="https://github-redirect.dependabot.com/tendermint/tendermint/issues/7">#7</a>...</li> <li><a href="https://github.com/tendermint/tendermint/commit/1dfb3451eaaa438067dadb6835e9aa948689f39e"><code>1dfb345</code></a> e2e: light nodes should use builtin abci app (<a href="https://github-redirect.dependabot.com/tendermint/tendermint/issues/7095">#7095</a>) (<a href="https://github-redirect.dependabot.com/tendermint/tendermint/issues/7096">#7096</a>)</li> <li><a href="https://github.com/tendermint/tendermint/commit/9f13b9b083da0ec45a48e57d5a283afdf698ba5b"><code>9f13b9b</code></a> e2e: abci protocol should be consistent across networks (backport <a href="https://github-redirect.dependabot.com/tendermint/tendermint/issues/7078">#7078</a>) (<a href="https://github-redirect.dependabot.com/tendermint/tendermint/issues/7085">#7085</a>)</li> <li><a href="https://github.com/tendermint/tendermint/commit/16ba782fa6549ea1e50a7fe7addb538ff15bf619"><code>16ba782</code></a> cli: allow node operator to rollback last state (backport <a href="https://github-redirect.dependabot.com/tendermint/tendermint/issues/7033">#7033</a>) (<a href="https://github-redirect.dependabot.com/tendermint/tendermint/issues/7080">#7080</a>)</li> <li><a href="https://github.com/tendermint/tendermint/commit/474ed04273bade74df583c7bfdc63e6e1a70a919"><code>474ed04</code></a> Import Postgres driver support for the psql indexer (backport). (<a href="https://github-redirect.dependabot.com/tendermint/tendermint/issues/7057">#7057</a>)</li> <li><a href="https://github.com/tendermint/tendermint/commit/2d8287d0f70962cc1d7bda0250def18962abf282"><code>2d8287d</code></a> e2e: allow running of single node using the e2e app (backport) (<a href="https://github-redirect.dependabot.com/tendermint/tendermint/issues/7024">#7024</a>)</li> <li><a href="https://github.com/tendermint/tendermint/commit/294a9695b4a768b4c8a9cb71c3562cdbff58e64c"><code>294a969</code></a> e2e: backport minor reliability improvements (<a href="https://github-redirect.dependabot.com/tendermint/tendermint/issues/6967">#6967</a>)</li> <li>See full diff in <a href="https://github.com/tendermint/tendermint/compare/v0.34.13...v0.34.14">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=github.com/tendermint/tendermint&package-manager=go_modules&previous-version=0.34.13&new-version=0.34.14)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) * fix: unmarshalling issue with multisig keys in master (backport #10061) (#10113) * fix: unmarshalling issue with multisig keys in master (#10061) (cherry picked from commit 3d3bc7c) # Conflicts: # CHANGELOG.md # crypto/keys/multisig/multisig_test.go * fix conflicts * fix conflicts * Update crypto/keys/multisig/multisig_test.go * Removed unused imports * fix changelog * Update CHANGELOG.md * Fix TestLegacyMultisig test (#10275) Co-authored-by: Henrik Aasted Sørensen <has@bitcraft.dk> Co-authored-by: Aleksandr Bezobchuk <aleks.bezobchuk@gmail.com> Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> Co-authored-by: Robert Zaremba <robert@zaremba.ch> Co-authored-by: Mario Karagiorgas <salem8@gmail.com> * fix: query account balance by ibc denom (backport #10394) (#10524) * fix: query account balance by ibc denom (#10394)  ## Description Closes: #10381  * chore: Iavl iterator (backport #10544) (#10546) * chore: Iavl iterator (#10544)  ## Description Closes: #10335 This PR adds a custom version of cosmos/iavl(cosmos/iavl#440) that removes the usage of channel-based approach on iterating IAVL tree. replaces cosmos/cosmos-sdk#10404 * build(deps): Bump github.com/cosmos/iavl from 0.17.2 to 0.17.3 (backport #10654) (#10664) * build(deps): Bump github.com/cosmos/iavl from 0.17.2 to 0.17.3 (#10654) Bumps [github.com/cosmos/iavl](https://github.com/cosmos/iavl) from 0.17.2 to 0.17.3. - [Release notes](https://github.com/cosmos/iavl/releases) - [Changelog](https://github.com/cosmos/iavl/blob/v0.17.3/CHANGELOG.md) - [Commits](cosmos/iavl@v0.17.2...v0.17.3) * fix: Add Events to TxResponse (backport #10630) (#10644) * fix: Add Events to TxResponse (#10630) (cherry picked from commit c4bedf8) # Conflicts: # CHANGELOG.md # go.sum # types/abci.pb.go # types/result.go * Michael sucks * Who Let the Bugs Out?? * Reinventing the wheel. Again. * No changes made Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com> Co-authored-by: Aleksandr Bezobchuk <aleks.bezobchuk@gmail.com> * fix: bech32 address test to satisfy the specification (#10163) (#10164) (#10665) ## Description Closes: #10163  * build(deps): Use self-maintained btcutil (#10082) (backport #10201) (#10622) * build(deps): Use self-maintained btcutil (#10082) (#10201) ## Description Closes: #10082 * chore: v0.42.11 changelog and release notes (#10695) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Henrik Aasted Sørensen <has@bitcraft.dk> Co-authored-by: Aleksandr Bezobchuk <aleks.bezobchuk@gmail.com> Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> Co-authored-by: Robert Zaremba <robert@zaremba.ch> Co-authored-by: Mario Karagiorgas <salem8@gmail.com> Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com> * chores: go mod tidy Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Ryan Christoffersen <12519942+ryanchristo@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Robert Zaremba <robert@zaremba.ch> Co-authored-by: Dev Ojha <ValarDragon@users.noreply.github.com> Co-authored-by: ValarDragon <dojha12@gmail.com> Co-authored-by: Carlos Rodriguez <crodveg@yahoo.es> Co-authored-by: Carlos Rodriguez <crodveg@gmail.com> Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> Co-authored-by: kitounliu <58184672+kitounliu@users.noreply.github.com> Co-authored-by: kitty <jia.liu@fetch.ai> Co-authored-by: Henrik Aasted Sørensen <has@bitcraft.dk> Co-authored-by: Aleksandr Bezobchuk <aleks.bezobchuk@gmail.com> Co-authored-by: Mario Karagiorgas <salem8@gmail.com> Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com>

* chore: Iavl iterator (#10544)  ## Description Closes: #10335 This PR adds a custom version of cosmos/iavl(cosmos/iavl#440) that removes the usage of channel-based approach on iterating IAVL tree. replaces cosmos/cosmos-sdk#10404

* chore: Iavl iterator (cosmos#10544)  Closes: cosmos#10335 This PR adds a custom version of cosmos/iavl(cosmos/iavl#440) that removes the usage of channel-based approach on iterating IAVL tree. replaces cosmos#10404 --- *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... - [ ] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] added `!` to the type prefix if API or client breaking change - [ ] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting)) - [ ] provided a link to the relevant issue or specification - [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules) - [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing) - [ ] added a changelog entry to `CHANGELOG.md` - [ ] included comments for [documenting Go code](https://blog.golang.org/godoc) - [ ] updated the relevant documentation or specification - [ ] reviewed "Files changed" and left comments if necessary - [ ] confirmed all CI checks have passed *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... - [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] confirmed `!` in the type prefix if API or client breaking change - [ ] confirmed all author checklist items have been addressed - [ ] reviewed state machine logic - [ ] reviewed API design and naming - [ ] reviewed documentation is accurate - [ ] reviewed tests and test coverage - [ ] manually tested (if applicable) (cherry picked from commit b5dd352) * solve conflicts Co-authored-by: Marko <marbar3778@yahoo.com>

mconcat added 5 commits October 13, 2021 17:26

move traverse into CPS

0fc7809

add iterator

4b30f08

add ascending

82d9a9d

fix iter.valid

bd8137a

fix test

a06214c

ValarDragon reviewed Oct 14, 2021

View reviewed changes

mconcat added 2 commits October 15, 2021 23:56

gofmt

688d385

simplify logicl, add comments

84c41c2

ValarDragon reviewed Oct 18, 2021

View reviewed changes

mconcat added 2 commits October 19, 2021 17:05

rm using iterator inside iavl

1261f14

add detailed comments

a33fa2e

mconcat mentioned this pull request Oct 19, 2021

Switch IAVL iterator logic cosmos/cosmos-sdk#10404

Closed

19 tasks

add comments per review

45a7cc2