Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to query archival data with recent upgrade 14470501 #2313

Closed
5 tasks
Tracked by #2617
aimxhaisse opened this issue Mar 17, 2023 · 32 comments · Fixed by cosmos/cosmos-sdk#15683
Closed
5 tasks
Tracked by #2617

Unable to query archival data with recent upgrade 14470501 #2313

aimxhaisse opened this issue Mar 17, 2023 · 32 comments · Fixed by cosmos/cosmos-sdk#15683
Assignees
Labels
type: bug Issues that need priority attention -- something isn't working

Comments

@aimxhaisse
Copy link

Summary of Bug

Since the Gaia migration at block 14470501, historical data is screwed up for blocks below.

Version

gaiad-v9.0.1-linux-amd64

Steps to Reproduce

Blocks >= 14470501: OK

$ grpcurl --max-msg-sz 400194304 -H "x-cosmos-block-height: 14470501" -plaintext NODE:PORT  cosmos.gov.v1beta1.Query/Proposals

[...]

ok

Blocks < 14470501: KO

$ grpcurl --max-msg-sz 400194304 -H "x-cosmos-block-height: 14470500" -plaintext NODE:PORT cosmos.gov.v1beta1.Query/Proposals

ERROR:
  Code: Unknown
  Message: failed to load state at height 14470500; version mismatch on immutable IAVL tree; version does not exist. Version has either been pruned, or is for a future block height (latest height: 14504397): invalid request

This happens for all gRPC calls, no matter which module is used, using a fresh Quicksync full archival dump using gaiad-v9.0.1-linux-amd64 version.

Is this a regression?


For Admin Use

  • Not duplicate issue
  • Appropriate labels applied
  • Appropriate contributors tagged
  • Contributor assigned/self-assigned
  • Is a spike necessary to map out how the issue should be approached?
@aimxhaisse aimxhaisse added type: bug Issues that need priority attention -- something isn't working status: waiting-triage This issue/PR has not yet been triaged by the team. labels Mar 17, 2023
@MSalopek
Copy link
Contributor

Thank you for opening the issue!

Would you mind sharing your node's configuration and please confirm that pruning is turned off in app.toml?

# default: the last 362880 states are kept, pruning at 10 block intervals
# nothing: all historic states will be saved, nothing will be deleted (i.e. archiving node)
# everything: 2 latest states will be kept; pruning at 10 block intervals.
# custom: allow pruning options to be manually specified through 'pruning-keep-recent', and 'pruning-interval'
pruning = "nothing"

This configuration should allow your node to keep all historic blocks.

@aimxhaisse
Copy link
Author

Yup, pruning is disabled:

# default: the last 100 states are kept in addition to every 500th state; pruning at 10 block intervals
# nothing: all historic states will be saved, nothing will be deleted (i.e. archiving node)
# everything: all saved states will be deleted, storing only the current and previous state; pruning at 10 block intervals
# custom: allow pruning options to be manually specified through 'pruning-keep-recent', 'pruning-keep-every', and 'pruning-interval'
pruning = "nothing"

@mmulji-ic mmulji-ic removed the status: waiting-triage This issue/PR has not yet been triaged by the team. label Mar 20, 2023
@yj0x0x
Copy link

yj0x0x commented Mar 22, 2023

Here is the same issue.

{"timestamp":"2023-03-22T11:25:14.915","level":"WARN","thread":"atom","logger":,"message":"Unable to get account 
balance at cosmos1j6a0cxzhk8jqcyr9m94hv5r8nqle5yrtfn7hrx at blockheight 14280567 using http://cosmos-fullnode:1317","exception":"java.lang.IllegalStateException: Invalid response code (400): {\n  \"code\": 3,\n  
\"message\": \"failed to load state at height 14280567; version mismatch on immutable IAVL tree; version does not exist. 
Version has either been pruned, or is for a future block height (latest height: 14476324)

Config set up is pruning nothing.

So it is archive node.

@aimxhaisse
Copy link
Author

I've found a hacky fix by patching the cosmos-sdk as follows:

diff --git a/store/iavl/store.go b/store/iavl/store.go
index 50324c89e..619b722b7 100644
--- a/store/iavl/store.go
+++ b/store/iavl/store.go
@@ -108,7 +108,7 @@ func UnsafeNewStore(tree *iavl.MutableTree) *Store {
 // Any mutable operations executed will result in a panic.
 func (st *Store) GetImmutable(version int64) (*Store, error) {
        if !st.VersionExists(version) {
-               return &Store{tree: &immutableTree{&iavl.ImmutableTree{}}}, fmt.Errorf("version mismatch on immutable IAVL tree; version does not exist. Version has either been pruned, or is for a future block height")
+               return &Store{tree: &immutableTree{&iavl.ImmutableTree{}}}, nil
        }
 
        iTree, err := st.tree.GetImmutable(version)

I'm not sure of all implications but I guess this is a regression that was introduced in the SDK bump during the Gaia v8 upgrade.

@aimxhaisse
Copy link
Author

This affected Juno as well, quoting Reece:

Reece | Juno-Network — Today at 16:09
I did this back for Juno’s v13 upgrade as well. Is because of a past issue with a chain upgrade which did not add a new modules storeKey

4-5 chains have had issues with this since the change (implemented in 45.12 I believe) 😔

So adding the new storeKey is the way forward, this is backwards compatible. I'll try to come up with a cleaner patch when I find some time if that sounds good (unless it's solved before :)).

@faddat
Copy link
Contributor

faddat commented Mar 28, 2023

I'd surely say it's a regression.

@mmulji-ic
Copy link
Contributor

Validator feedback:
We found that this 1st happened with the vega upgrade (i.e. from gravity-dex->v6 vega)
But then from v6 (vega) -> v7 (theta) or from v7 (theta) -> v8 (rho) we didn’t see any regressions.
and now from v8 (rho) -> v9 (lambda) we see a regression again.

If this is how the system works, then for historical reporting purposes looks like we are being asked to run a dedicated node with older binaries until upgrade heights for every single cosmos release ?

@danbryan
Copy link

danbryan commented Apr 7, 2023

I can confirm this resolved the issue temporarily for me.

Modify go.mod with

replace github.com/cosmos/cosmos-sdk => github.com/reecepbcups/cosmos-sdk v0.45.13-ics.archive.1

then

go mod tidy
make install

@yj0x0x
Copy link

yj0x0x commented Apr 10, 2023

@MSalopek Still this issue on going?
Nothing to patched in latest release?

@faddat
Copy link
Contributor

faddat commented Apr 10, 2023

Hey there, We have written a patch. You can find the patched version in the pull requests. We have submitted the patch to the cosmos SDK. Cosmo's SDK 45 is deprecated. The Cosmos SDK team simultaneously will not merge the patch to 45 and recommends against using a fork.

#2390

mergify bot pushed a commit to cosmos/cosmos-sdk that referenced this issue Apr 10, 2023
## Description
Closes: [cosmos/gaia#2313](cosmos/gaia#2313)

The gaia v9 upgrade causes node to errors out when querying historical states before the upgrade. This is because we added a new module store in the upgrade which is the `provider module store`. Thus, every times we load the multi store for querying at a height prior to the upgrade, we try to load the `provider module store` and fail since it has not existed before the upgrade.

I make changes to `CacheMultiStoreWithVersion` so that if we fail to load any module store at a given height, we check if that module store has existed at that height or not (using `getCommitInfo()`), we can ignore the error if the store hasn't existed at that given height.

---

### Author Checklist

*All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.*

I have...

* [ ] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
* [ ] added `!` to the type prefix if API or client breaking change
* [ ] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#pr-targeting))
* [ ] provided a link to the relevant issue or specification
* [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/main/docs/docs/building-modules)
* [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#testing)
* [ ] added a changelog entry to `CHANGELOG.md`
* [ ] included comments for [documenting Go code](https://blog.golang.org/godoc)
* [ ] updated the relevant documentation or specification
* [ ] reviewed "Files changed" and left comments if necessary
* [ ] confirmed all CI checks have passed

### Reviewers Checklist

*All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.*

I have...

* [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
* [ ] confirmed `!` in the type prefix if API or client breaking change
* [ ] confirmed all author checklist items have been addressed 
* [ ] reviewed state machine logic
* [ ] reviewed API design and naming
* [ ] reviewed documentation is accurate
* [ ] reviewed tests and test coverage
* [ ] manually tested (if applicable)
mergify bot pushed a commit to cosmos/cosmos-sdk that referenced this issue Apr 10, 2023
## Description
Closes: [cosmos/gaia#2313](cosmos/gaia#2313)

The gaia v9 upgrade causes node to errors out when querying historical states before the upgrade. This is because we added a new module store in the upgrade which is the `provider module store`. Thus, every times we load the multi store for querying at a height prior to the upgrade, we try to load the `provider module store` and fail since it has not existed before the upgrade.

I make changes to `CacheMultiStoreWithVersion` so that if we fail to load any module store at a given height, we check if that module store has existed at that height or not (using `getCommitInfo()`), we can ignore the error if the store hasn't existed at that given height.

---

### Author Checklist

*All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.*

I have...

* [ ] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
* [ ] added `!` to the type prefix if API or client breaking change
* [ ] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#pr-targeting))
* [ ] provided a link to the relevant issue or specification
* [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/main/docs/docs/building-modules)
* [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#testing)
* [ ] added a changelog entry to `CHANGELOG.md`
* [ ] included comments for [documenting Go code](https://blog.golang.org/godoc)
* [ ] updated the relevant documentation or specification
* [ ] reviewed "Files changed" and left comments if necessary
* [ ] confirmed all CI checks have passed

### Reviewers Checklist

*All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.*

I have...

* [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
* [ ] confirmed `!` in the type prefix if API or client breaking change
* [ ] confirmed all author checklist items have been addressed
* [ ] reviewed state machine logic
* [ ] reviewed API design and naming
* [ ] reviewed documentation is accurate
* [ ] reviewed tests and test coverage
* [ ] manually tested (if applicable)

(cherry picked from commit d5a618d)

# Conflicts:
#	store/CHANGELOG.md
mergify bot pushed a commit to cosmos/cosmos-sdk that referenced this issue Apr 10, 2023
## Description
Closes: [cosmos/gaia#2313](cosmos/gaia#2313)

The gaia v9 upgrade causes node to errors out when querying historical states before the upgrade. This is because we added a new module store in the upgrade which is the `provider module store`. Thus, every times we load the multi store for querying at a height prior to the upgrade, we try to load the `provider module store` and fail since it has not existed before the upgrade.

I make changes to `CacheMultiStoreWithVersion` so that if we fail to load any module store at a given height, we check if that module store has existed at that height or not (using `getCommitInfo()`), we can ignore the error if the store hasn't existed at that given height.

---

### Author Checklist

*All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.*

I have...

* [ ] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
* [ ] added `!` to the type prefix if API or client breaking change
* [ ] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#pr-targeting))
* [ ] provided a link to the relevant issue or specification
* [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/main/docs/docs/building-modules)
* [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#testing)
* [ ] added a changelog entry to `CHANGELOG.md`
* [ ] included comments for [documenting Go code](https://blog.golang.org/godoc)
* [ ] updated the relevant documentation or specification
* [ ] reviewed "Files changed" and left comments if necessary
* [ ] confirmed all CI checks have passed

### Reviewers Checklist

*All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.*

I have...

* [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
* [ ] confirmed `!` in the type prefix if API or client breaking change
* [ ] confirmed all author checklist items have been addressed
* [ ] reviewed state machine logic
* [ ] reviewed API design and naming
* [ ] reviewed documentation is accurate
* [ ] reviewed tests and test coverage
* [ ] manually tested (if applicable)

(cherry picked from commit d5a618d)

# Conflicts:
#	store/CHANGELOG.md
@mmulji-ic
Copy link
Contributor

@MSalopek Still this issue on going? Nothing to patched in latest release?

Hi @yj0x0x , we'll discuss with the SDK to see if they can update this issue into 0.45.x . Otherwise we'll discuss with the hub team on using a fork (which is definitely not preferred).

@mmulji-ic mmulji-ic reopened this Apr 11, 2023
@yj0x0x
Copy link

yj0x0x commented Apr 11, 2023

@mmulji-ic Thank you for reply.
As a user of cosmos node, I think this issue is critical.
Once HF done, And Can't get block data before HF. How could be verified data in the cosmos network?

Hope that this issue will raise in serious manner, fix in ASAP personally.

@glnro
Copy link
Contributor

glnro commented Apr 14, 2023

This issue has been addressed in the SDK and backported for v0.46 and v0.47. There are a few temporary workarounds until the Hub upgrades to SDK v0.47 in the meantime which we'll provide documentation for shortly.

@mmulji-ic mmulji-ic assigned mmulji-ic and unassigned glnro Apr 17, 2023
@yj0x0x
Copy link

yj0x0x commented Apr 19, 2023

@mmulji-ic Thank you letting me know.
Please let me know how to fix this temporary in ASAP.

Looking forward hearing from you.

@yj0x0x
Copy link

yj0x0x commented Apr 27, 2023

@mmulji-ic @faddat @glnro Ping.

@yj0x0x
Copy link

yj0x0x commented May 9, 2023

@mmulji-ic @MSalopek
Consensus update is coming, any updates about this issue?

@yj0x0x
Copy link

yj0x0x commented May 26, 2023

@mmulji-ic @MSalopek @jackzampolin Anybody knows how's going on?

@yj0x0x
Copy link

yj0x0x commented Jun 5, 2023

@mmulji-ic @MSalopek @faddat Any update please?

@mmulji-ic
Copy link
Contributor

mmulji-ic commented Jun 26, 2023

Hi @yj0x0x checking, unfortunately we don't have the infra setup, so it takes some time. Dan mentioned a patch above, for the SDK that you can also try. We've talked to the SDK folks and they won't backport that to v0.45 line unfortunately.

jhernandezb added a commit to public-awesome/cosmos-sdk that referenced this issue Jul 31, 2023
Closes: [cosmos/gaia#2313](cosmos/gaia#2313)

The gaia v9 upgrade causes node to errors out when querying historical states before the upgrade. This is because we added a new module store in the upgrade which is the `provider module store`. Thus, every times we load the multi store for querying at a height prior to the upgrade, we try to load the `provider module store` and fail since it has not existed before the upgrade.

I make changes to `CacheMultiStoreWithVersion` so that if we fail to load any module store at a given height, we check if that module store has existed at that height or not (using `getCommitInfo()`), we can ignore the error if the store hasn't existed at that given height.

---

*All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.*

I have...

* [ ] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
* [ ] added `!` to the type prefix if API or client breaking change
* [ ] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#pr-targeting))
* [ ] provided a link to the relevant issue or specification
* [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/main/docs/docs/building-modules)
* [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#testing)
* [ ] added a changelog entry to `CHANGELOG.md`
* [ ] included comments for [documenting Go code](https://blog.golang.org/godoc)
* [ ] updated the relevant documentation or specification
* [ ] reviewed "Files changed" and left comments if necessary
* [ ] confirmed all CI checks have passed

*All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.*

I have...

* [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
* [ ] confirmed `!` in the type prefix if API or client breaking change
* [ ] confirmed all author checklist items have been addressed
* [ ] reviewed state machine logic
* [ ] reviewed API design and naming
* [ ] reviewed documentation is accurate
* [ ] reviewed tests and test coverage
* [ ] manually tested (if applicable)
jhernandezb added a commit to public-awesome/cosmos-sdk that referenced this issue Aug 2, 2023
* feat: More flexibility for `CacheMultiStoreWithVersion` (cosmos#15683)

Closes: [cosmos/gaia#2313](cosmos/gaia#2313)

The gaia v9 upgrade causes node to errors out when querying historical states before the upgrade. This is because we added a new module store in the upgrade which is the `provider module store`. Thus, every times we load the multi store for querying at a height prior to the upgrade, we try to load the `provider module store` and fail since it has not existed before the upgrade.

I make changes to `CacheMultiStoreWithVersion` so that if we fail to load any module store at a given height, we check if that module store has existed at that height or not (using `getCommitInfo()`), we can ignore the error if the store hasn't existed at that given height.

---

*All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.*

I have...

* [ ] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
* [ ] added `!` to the type prefix if API or client breaking change
* [ ] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#pr-targeting))
* [ ] provided a link to the relevant issue or specification
* [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/main/docs/docs/building-modules)
* [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#testing)
* [ ] added a changelog entry to `CHANGELOG.md`
* [ ] included comments for [documenting Go code](https://blog.golang.org/godoc)
* [ ] updated the relevant documentation or specification
* [ ] reviewed "Files changed" and left comments if necessary
* [ ] confirmed all CI checks have passed

*All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.*

I have...

* [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
* [ ] confirmed `!` in the type prefix if API or client breaking change
* [ ] confirmed all author checklist items have been addressed
* [ ] reviewed state machine logic
* [ ] reviewed API design and naming
* [ ] reviewed documentation is accurate
* [ ] reviewed tests and test coverage
* [ ] manually tested (if applicable)

* fix merge conflict
@mpoke
Copy link
Contributor

mpoke commented Sep 7, 2023

We're planing to upgrade Gaia to SDK 0.47 in the following months. In the meantime, please use the suggested fix.

@mpoke mpoke closed this as completed Sep 7, 2023
@gilxgil
Copy link

gilxgil commented Sep 15, 2023

  1. Reopening, this is still a major issue for archive nodes.
  2. The fix is now broken with v12.0.0. Any suggestions from the team?

@Jacoblava
Copy link

We're planing to upgrade Gaia to SDK 0.47 in the following months. In the meantime, please use the suggested fix.

The suggested fix doesn't work anymore.
Latest upgrade to v12.0.0 broke it.
No more support for archive nodes.
@mpoke can you please re-open the issue?

@mpoke mpoke reopened this Sep 18, 2023
@mmulji-ic
Copy link
Contributor

Hi @Jacoblava @gilxgil we'll take a look to see if we can adapt that fix or suggestion another mitigation.

@Jacoblava
Copy link

@mmulji-ic Is there any update / fix / patch for it?
This is pretty critical for all archive node runners.
No user can retrieve archive data from GAIA for over 2 weeks now.

@Reecepbcups
Copy link
Member

Reecepbcups commented Oct 2, 2023

@Jacoblava I have updated my unofficial patch

https://github.com/Reecepbcups/cosmos-sdk/releases/tag/v0.45.16-ics-lsm-iavlpatch.1

replace github.com/cosmos/cosmos-sdk => github.com/reecepbcups/cosmos-sdk v0.45.16-ics-lsm-iavlpatch.1

also cc @mmulji-ic

@Reecepbcups
Copy link
Member

oh and also @gilxgil ^

@Jacoblava
Copy link

Jacoblava commented Oct 3, 2023 via email

@Reecepbcups
Copy link
Member

@ScepticMatt Yea this lsm patch is only for Gaia.

Axelar probably has their own fork then or they use Ethermint? My patches do not support this

@ScepticMatt
Copy link

@ScepticMatt Yea this lsm patch is only for Gaia.

Axelar probably has their own fork then or they use Ethermint? My patches do not support this

Thanks I saw that they forked it. I forked their fork and will try if I can bypass this

@mmulji-ic
Copy link
Contributor

@Jacoblava @ScepticMatt please repoen if your forked forks do not work.
Otherwise, closing as this issue seems temporarily resolved. We hope to move to sdk47 soon and hopefully this forking will not be needed. Thanks @Reecepbcups for updating.

bsrinivas8687 pushed a commit to sentinel-official/cosmos-sdk that referenced this issue Mar 4, 2024
Closes: [cosmos/gaia#2313](cosmos/gaia#2313)

The gaia v9 upgrade causes node to errors out when querying historical states before the upgrade. This is because we added a new module store in the upgrade which is the `provider module store`. Thus, every times we load the multi store for querying at a height prior to the upgrade, we try to load the `provider module store` and fail since it has not existed before the upgrade.

I make changes to `CacheMultiStoreWithVersion` so that if we fail to load any module store at a given height, we check if that module store has existed at that height or not (using `getCommitInfo()`), we can ignore the error if the store hasn't existed at that given height.

---

*All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.*

I have...

* [ ] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
* [ ] added `!` to the type prefix if API or client breaking change
* [ ] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#pr-targeting))
* [ ] provided a link to the relevant issue or specification
* [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/main/docs/docs/building-modules)
* [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#testing)
* [ ] added a changelog entry to `CHANGELOG.md`
* [ ] included comments for [documenting Go code](https://blog.golang.org/godoc)
* [ ] updated the relevant documentation or specification
* [ ] reviewed "Files changed" and left comments if necessary
* [ ] confirmed all CI checks have passed

*All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.*

I have...

* [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
* [ ] confirmed `!` in the type prefix if API or client breaking change
* [ ] confirmed all author checklist items have been addressed
* [ ] reviewed state machine logic
* [ ] reviewed API design and naming
* [ ] reviewed documentation is accurate
* [ ] reviewed tests and test coverage
* [ ] manually tested (if applicable)
haiyizxx pushed a commit to axelarnetwork/cosmos-sdk that referenced this issue Mar 21, 2024
Closes: [cosmos/gaia#2313](cosmos/gaia#2313)

The gaia v9 upgrade causes node to errors out when querying historical states before the upgrade. This is because we added a new module store in the upgrade which is the `provider module store`. Thus, every times we load the multi store for querying at a height prior to the upgrade, we try to load the `provider module store` and fail since it has not existed before the upgrade.

I make changes to `CacheMultiStoreWithVersion` so that if we fail to load any module store at a given height, we check if that module store has existed at that height or not (using `getCommitInfo()`), we can ignore the error if the store hasn't existed at that given height.

---

*All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.*

I have...

* [ ] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
* [ ] added `!` to the type prefix if API or client breaking change
* [ ] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#pr-targeting))
* [ ] provided a link to the relevant issue or specification
* [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/main/docs/docs/building-modules)
* [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#testing)
* [ ] added a changelog entry to `CHANGELOG.md`
* [ ] included comments for [documenting Go code](https://blog.golang.org/godoc)
* [ ] updated the relevant documentation or specification
* [ ] reviewed "Files changed" and left comments if necessary
* [ ] confirmed all CI checks have passed

*All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.*

I have...

* [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
* [ ] confirmed `!` in the type prefix if API or client breaking change
* [ ] confirmed all author checklist items have been addressed
* [ ] reviewed state machine logic
* [ ] reviewed API design and naming
* [ ] reviewed documentation is accurate
* [ ] reviewed tests and test coverage
* [ ] manually tested (if applicable)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: bug Issues that need priority attention -- something isn't working
Projects
Status: ✅ Done