[Bug]: Certain calls with x-cosmos-block-height are broken

[evyatar-tres]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

A bug happened!

General
Until recently, across all cosmos blockchains, I was able to query historical rewards using the "x-cosmos-block-height", this was done in correspondence to the RPC's sync_info and earliest_block_height on that specific node

Starting last week (probably related to some versioning upgrade), this does not work anymore

Usecase
RPC URL: https://rpc-us-1.kyve.network/status?
earliest_block: "1"
version: "0.37.2

SDK URL:
https://api-us-1.kyve.network/cosmos/distribution/v1beta1/delegators/kyve13jgzfjfwp2mmfrmegvmulrrmkm4fqectts76uq/rewards

SDK Version: "v0.47.6"

Using headers + error:
code 2
message "codespace sdk code 18: invalid request: failed to load state at height 100000; version mismatch on immutable IAVL tree; version does not exist. Version has either been pruned, or is for a future block height (latest height: 5457144)"
details []

Note, until last week this worked for all block heights

Cosmos SDK Version

"v0.47.6"

How to reproduce?

https://api-us-1.kyve.network/cosmos/distribution/v1beta1/delegators/kyve13jgzfjfwp2mmfrmegvmulrrmkm4fqectts76uq/rewards

with x-cosmos-block-height, using this value: 100000

[chriscollinsgit]

We are experiencing a similar issue with supply metrics. Historical supply values are not being returned correct using the x-cosmos-block-height parameter.

Here is the current supply.
https://cosmos-lcd.quickapi.com/cosmos/bank/v1beta1/supply/by_denom?denom=uatom&x-cosmos-block-height=19653629

The following value should be roughly 324240612.620618 however Atom supply returned is almost identical to the current supply.
https://cosmos-lcd.quickapi.com/cosmos/bank/v1beta1/supply/by_denom?denom=uatom&x-cosmos-block-height=13653629

[alexanderbez]

Hello, personally, I think the error is pretty self-explanatory. It seems like the historical height you're querying against is pruned or no longer exists. What are you pruning settings on these node(s)?

[evyatar-tres]

Hi @alexanderbez
if you look here: https://rpc-us-1.kyve.network/status?

you can see this:

"sync_info":{"latest_block_hash":"ADDC54434F12E342BB95F3330D5D7D9E726478D9564F7B68FA88C6065FB7E475","latest_app_hash":"D7B588882E069991CA9A69B5EE86E060CC01AF0F112C3A22BD20988310D3384E","latest_block_height":"5459012","latest_block_time":"2024-03-21T15:38:14.316942125Z","earliest_block_hash":"144269EC6B36319FCB480E27420D59E487709CEF784EC37E39A396E05E9DC437","earliest_app_hash":"E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855","earliest_block_height":"1","earliest_block_time":"2023-03-14T14:03:14Z","catching_up":false}

which states the earliest block is 1,

this call also used to work until this week, and i have ran into it across alot of different archive nodes, implying that this has something to do with a version upgrade

[alexanderbez]

@evyatar-tres That's CometBFT's perspective on state, not the application. You're querying the application state. CometBFT pruning and application pruning are handled independently.

[chriscollinsgit]

In our case, we have been querying this node and another node from block 5200791 successfully everyday for months until this morning.

[chriscollinsgit]

@alexanderbez from looking at the following endpoint

https://cosmoshub-api.cosmosrescue.dev/cosmos/base/tendermint/v1beta1/blocks/5200791

I can see that you are correct about the nodes now being pruned. Is this expected after the upgrade. Is it likely that a period of resyncing is required for nodes after the upgrade?

[evyatar-tres]

@alexanderbez
any update on this?

[evyatar-tres]

Also, might be related to this?

cosmos/gaia#2313

[tac0turtle]

as reece mentioned in his patch this is on purpose to prevent other bugs, We can look into it in the future but for we recommend running a patch until we have the bandwidth to fix this. If anyone here wants to look into it we are happy to review prs

[tac0turtle]

closing this as it seems a resolution was met. We cant gurantee queries cross upgrades due to versions breaking. Best chance is to query the block using the old binary instead of the latest