Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix iterators with existant start, nil end #12661

Closed
4 tasks
ValarDragon opened this issue Jul 21, 2022 · 0 comments · Fixed by #12945
Closed
4 tasks

Fix iterators with existant start, nil end #12661

ValarDragon opened this issue Jul 21, 2022 · 0 comments · Fixed by #12945
Assignees

Comments

@ValarDragon
Copy link
Contributor

ValarDragon commented Jul 21, 2022

Summary

Both of the following iterators fail:

func (suite *TestSuite) TestIterator() {
	key := suite.App.GetKVStoreKey()[types.StoreKey]
	store := suite.Ctx.KVStore(key)

	store.Set([]byte("AB"), []byte{1})

	iter := store.Iterator([]byte("AA"), nil)
	suite.Require().True(iter.Valid())
	iter.Close()

	iter = store.Iterator([]byte("A"), nil)
	suite.Require().True(iter.Valid())
	iter.Close()
}

They should work. The root cause is that there is a check in the CacheKvStore, that unsafe converts []byte to a string, and then compares the string representations.

The typecast of a nil []byte, will always return as less than a correctly formed string. Thus you get start > end and the CacheKVStore iterator returns a no-op.

This is a bug in the SDK, that we have to fix at the caller level, by making nil 0xff. (This took forever in debugging to figure out)

(Just flagging here, the entire CacheKVStore should likely be rewritten due to the massive number of issues and complexity it has with iterators right now. Will try to make a larger issue about this later)

Proposal

Fix the implementation here https://github.com/cosmos/cosmos-sdk/blob/main/store/cachekv/store.go#L283-L287 by making more explicit handling of this nil end time case.


For Admin Use

  • Not duplicate issue
  • Appropriate labels applied
  • Appropriate contributors tagged
  • Contributor assigned/self-assigned
@kocubinski kocubinski self-assigned this Aug 15, 2022
kocubinski added a commit that referenced this issue Aug 16, 2022
#12661 is incomplete, this
is only an issue with dirty items in the cache iterator.
These (failing) tests cases demonstrate this.
@mergify mergify bot closed this as completed in #12945 Aug 17, 2022
mergify bot pushed a commit that referenced this issue Aug 17, 2022
## Description

Closes: #12661 

Adds support for nil end semantics to iterators in cachekv store, addressing [this workaround](https://github.com/osmosis-labs/osmosis/blob/4176b287d48338870bfda3029bfa20a6e45ac126/osmoutils/store_helper.go#L37-L41).  

Note that this has the effect of sorting the dirty cache into the BTree cache store in the bounds `[startIndex, end-of-cache-space]`



---

### Author Checklist

*All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.*

I have...

- [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [ ] added `!` to the type prefix if API or client breaking change
- [x] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#pr-targeting))
- [x] provided a link to the relevant issue or specification
- [x] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/main/docs/building-modules)
- [x] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#testing)
- [ ] added a changelog entry to `CHANGELOG.md`
- [x] included comments for [documenting Go code](https://blog.golang.org/godoc)
- [ ] updated the relevant documentation or specification
- [x] reviewed "Files changed" and left comments if necessary
- [ ] confirmed all CI checks have passed

### Reviewers Checklist

*All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.*

I have...

- [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [ ] confirmed `!` in the type prefix if API or client breaking change
- [ ] confirmed all author checklist items have been addressed 
- [ ] reviewed state machine logic
- [ ] reviewed API design and naming
- [ ] reviewed documentation is accurate
- [ ] reviewed tests and test coverage
- [ ] manually tested (if applicable)
yihuang pushed a commit to yihuang/cosmos-sdk that referenced this issue Jan 5, 2023
Closes: cosmos#12661

Adds support for nil end semantics to iterators in cachekv store, addressing [this workaround](https://github.com/osmosis-labs/osmosis/blob/4176b287d48338870bfda3029bfa20a6e45ac126/osmoutils/store_helper.go#L37-L41).

Note that this has the effect of sorting the dirty cache into the BTree cache store in the bounds `[startIndex, end-of-cache-space]`

---

*All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.*

I have...

- [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [ ] added `!` to the type prefix if API or client breaking change
- [x] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#pr-targeting))
- [x] provided a link to the relevant issue or specification
- [x] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/main/docs/building-modules)
- [x] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#testing)
- [ ] added a changelog entry to `CHANGELOG.md`
- [x] included comments for [documenting Go code](https://blog.golang.org/godoc)
- [ ] updated the relevant documentation or specification
- [x] reviewed "Files changed" and left comments if necessary
- [ ] confirmed all CI checks have passed

*All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.*

I have...

- [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [ ] confirmed `!` in the type prefix if API or client breaking change
- [ ] confirmed all author checklist items have been addressed
- [ ] reviewed state machine logic
- [ ] reviewed API design and naming
- [ ] reviewed documentation is accurate
- [ ] reviewed tests and test coverage
- [ ] manually tested (if applicable)
tac0turtle pushed a commit that referenced this issue Jan 5, 2023
Closes: #12661

Adds support for nil end semantics to iterators in cachekv store, addressing [this workaround](https://github.com/osmosis-labs/osmosis/blob/4176b287d48338870bfda3029bfa20a6e45ac126/osmoutils/store_helper.go#L37-L41).

Note that this has the effect of sorting the dirty cache into the BTree cache store in the bounds `[startIndex, end-of-cache-space]`

---

*All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.*

I have...

- [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [ ] added `!` to the type prefix if API or client breaking change
- [x] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#pr-targeting))
- [x] provided a link to the relevant issue or specification
- [x] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/main/docs/building-modules)
- [x] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#testing)
- [ ] added a changelog entry to `CHANGELOG.md`
- [x] included comments for [documenting Go code](https://blog.golang.org/godoc)
- [ ] updated the relevant documentation or specification
- [x] reviewed "Files changed" and left comments if necessary
- [ ] confirmed all CI checks have passed

*All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.*

I have...

- [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [ ] confirmed `!` in the type prefix if API or client breaking change
- [ ] confirmed all author checklist items have been addressed
- [ ] reviewed state machine logic
- [ ] reviewed API design and naming
- [ ] reviewed documentation is accurate
- [ ] reviewed tests and test coverage
- [ ] manually tested (if applicable)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

2 participants