New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gaia-9002 Crash log #3019

Closed
Slamper opened this Issue Dec 6, 2018 · 6 comments

Comments

Projects
None yet
4 participants
@Slamper
Copy link
Contributor

Slamper commented Dec 6, 2018

invariant broken: loose token invariance:
	pool.LooseTokens: 4409145988819987660.4767998960
	sum of account tokens: 4409145988819987660.4768000000" stack="goroutine 1201 [running]:
runtime/debug.Stack(0xc00a433ac0, 0xf38d60, 0xc003f91540)
	/usr/lib/golang/src/runtime/debug/stack.go:24 +0xa7
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).receiveRoutine.func2(0xc000497c00, 0x1217900)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:576 +0x57
panic(0xf38d60, 0xc003f91540)
	/usr/lib/golang/src/runtime/panic.go:513 +0x1b9
github.com/cosmos/cosmos-sdk/cmd/gaia/app.(*GaiaApp).assertRuntimeInvariants(0xc000af1000)
	src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/invariants.go:30 +0x453
github.com/cosmos/cosmos-sdk/cmd/gaia/app.(*GaiaApp).EndBlocker(0xc000af1000, 0x1315a80, 0xc00f0eeb10, 0xc00c567ac0, 0xd, 0x17b09, 0x0, 0x0, 0x0, 0x0, ...)
	src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/app.go:218 +0x241
github.com/cosmos/cosmos-sdk/cmd/gaia/app.(*GaiaApp).EndBlocker-fm(0x1315a80, 0xc00f0eeb10, 0xc00c567ac0, 0xd, 0x17b09, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/app.go:165 +0xc8
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).EndBlock(0xc00018a700, 0x17b09, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	src/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:794 +0x105
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/abci/client.(*localClient).EndBlockSync(0xc0001b4cc0, 0x17b09, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/abci/client/local_client.go:239 +0xea
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/proxy.(*appConnConsensus).EndBlockSync(0xc000978d40, 0x17b09, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/proxy/app_conn.go:77 +0x51
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state.execBlockOnProxyApp(0x1316900, 0xc00086a500, 0x131ba80, 0xc000978d40, 0xc002e3cd20, 0xc00f0ee570, 0x1320540, 0xc0001c46b0, 0x24ca7, 0xc009c7e560, ...)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state/execution.go:265 +0x701
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(0xc000112700, 0x7, 0x0, 0xc000b2ca20, 0x6, 0xc000b2ca30, 0x9, 0x17b08, 0x24ca7, 0xc009c7e560, ...)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state/execution.go:96 +0x15f
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).finalizeCommit(0xc000497c00, 0x17b09)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1290 +0xa8c
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).tryFinalizeCommit(0xc000497c00, 0x17b09)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1221 +0x451
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).addProposalBlockPart(0xc000497c00, 0xc00016dd40, 0xc00417e0f0, 0x28, 0xc00c3b3000, 0x2d9d, 0x2d9d)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1460 +0x4d1
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).handleMsg(0xc000497c00, 0x13084a0, 0xc00016dd40, 0xc00417e0f0, 0x28)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:638 +0x7a0
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).receiveRoutine(0xc000497c00, 0x0)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:607 +0x670
created by github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).OnStart
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:300 +0x132
invariant broken: loose token invariance:
	pool.LooseTokens: 4409145988819987660.4767998960
	sum of account tokens: 4409145988819987660.4768000000" stack="goroutine 1201 [running]:
runtime/debug.Stack(0xc00a433ac0, 0xf38d60, 0xc003f91540)
	/usr/lib/golang/src/runtime/debug/stack.go:24 +0xa7
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).receiveRoutine.func2(0xc000497c00, 0x1217900)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:576 +0x57
panic(0xf38d60, 0xc003f91540)
	/usr/lib/golang/src/runtime/panic.go:513 +0x1b9
github.com/cosmos/cosmos-sdk/cmd/gaia/app.(*GaiaApp).assertRuntimeInvariants(0xc000af1000)
	src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/invariants.go:30 +0x453
github.com/cosmos/cosmos-sdk/cmd/gaia/app.(*GaiaApp).EndBlocker(0xc000af1000, 0x1315a80, 0xc00f0eeb10, 0xc00c567ac0, 0xd, 0x17b09, 0x0, 0x0, 0x0, 0x0, ...)
	src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/app.go:218 +0x241
github.com/cosmos/cosmos-sdk/cmd/gaia/app.(*GaiaApp).EndBlocker-fm(0x1315a80, 0xc00f0eeb10, 0xc00c567ac0, 0xd, 0x17b09, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/app.go:165 +0xc8
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).EndBlock(0xc00018a700, 0x17b09, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	src/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:794 +0x105
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/abci/client.(*localClient).EndBlockSync(0xc0001b4cc0, 0x17b09, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/abci/client/local_client.go:239 +0xea
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/proxy.(*appConnConsensus).EndBlockSync(0xc000978d40, 0x17b09, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/proxy/app_conn.go:77 +0x51
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state.execBlockOnProxyApp(0x1316900, 0xc00086a500, 0x131ba80, 0xc000978d40, 0xc002e3cd20, 0xc00f0ee570, 0x1320540, 0xc0001c46b0, 0x24ca7, 0xc009c7e560, ...)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state/execution.go:265 +0x701
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(0xc000112700, 0x7, 0x0, 0xc000b2ca20, 0x6, 0xc000b2ca30, 0x9, 0x17b08, 0x24ca7, 0xc009c7e560, ...)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state/execution.go:96 +0x15f
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).finalizeCommit(0xc000497c00, 0x17b09)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1290 +0xa8c
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).tryFinalizeCommit(0xc000497c00, 0x17b09)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1221 +0x451
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).addProposalBlockPart(0xc000497c00, 0xc00016dd40, 0xc00417e0f0, 0x28, 0xc00c3b3000, 0x2d9d, 0x2d9d)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1460 +0x4d1
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).handleMsg(0xc000497c00, 0x13084a0, 0xc00016dd40, 0xc00417e0f0, 0x28)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:638 +0x7a0
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).receiveRoutine(0xc000497c00, 0x0)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:607 +0x670
created by github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).OnStart
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:300 +0x132


For Admin Use

  • Not duplicate issue
  • Appropriate labels applied
  • Appropriate contributors tagged
  • Contributor assigned/self-assigned
@cwgoes

This comment has been minimized.

Copy link
Contributor

cwgoes commented Dec 6, 2018

Runtime invariants are asserted at the end of each block, so this discrepancy was caused by a transaction (or begin block / end block operation) in the block we were attempting to execute.

@Slamper

This comment has been minimized.

Copy link
Contributor

Slamper commented Dec 6, 2018

{
        "Height": 97032,
        "Hash": "3FF40CAD512D1379EE70E2FCD274C7D14F531B5CD62BADDBF391AC8F1EB863C0",
        "Fee": {
            "gas": 200000,
            "amount": [
                {
                    "denom": "",
                    "amount": "0"
                }
            ]
        },
        "Memo": "",
        "IsOK": true,
        "ValidFormat": true,
        "Result": {
            "log": "Msg 0: ",
            "tags": [
                {
                    "key": "action",
                    "value": "withdraw_validator_rewards_all"
                },
                {
                    "key": "source-validator",
                    "value": "cosmosvaloper1shuqhpl273t96yg6nnqvyfeewj3ew3md6uy7uj"
                }
            ],
            "gas_used": 8082,
            "gas_wanted": 200000
        },
        "Messages": [
            {
                "data": {
                    "validator_addr": "cosmosvaloper1shuqhpl273t96yg6nnqvyfeewj3ew3md6uy7uj"
                },
                "type": "withdraw_validator_rewards_all"
            }
        ],
        "Signatures": [
            {
                "address": "85f80b87eaf4565d111a9cc0c2273974a397476d",
                "sequence": 3107
            }
        ]
    },
    {
        "Height": 97032,
        "Hash": "3F04C45EAD604CFF00BD4746ED015A64F617E6F2AEB3D11E03553D1F2E2CE7D0",
        "Fee": {
            "gas": 200000,
            "amount": [
                {
                    "denom": "",
                    "amount": "0"
                }
            ]
        },
        "Memo": "",
        "IsOK": true,
        "ValidFormat": true,
        "Result": {
            "log": "Msg 0: ",
            "tags": [
                {
                    "key": "action",
                    "value": "withdraw_delegation_rewards_all"
                },
                {
                    "key": "delegator",
                    "value": "cosmos1vkggdmhuacs2306uasqw3au8a8lmcmqhqngtj4"
                }
            ],
            "gas_used": 7498,
            "gas_wanted": 200000
        },
        "Messages": [
            {
                "data": {
                    "delegator_addr": "cosmos1vkggdmhuacs2306uasqw3au8a8lmcmqhqngtj4"
                },
                "type": "withdraw_delegation_rewards_all"
            }
        ],
        "Signatures": [
            {
                "address": "659086eefcee20a8bf5cec00e8f787e9ffbc6c17",
                "sequence": 3678
            }
        ]
    },
    {
        "Height": 97032,
        "Hash": "4610B78843D9AD031F3A72745E372C8AA868CB2A408095D30A7C2D3676448EB5",
        "Fee": {
            "gas": 200000,
            "amount": [
                {
                    "denom": "",
                    "amount": "0"
                }
            ]
        },
        "Memo": "",
        "IsOK": true,
        "ValidFormat": true,
        "Result": {
            "log": "Msg 0: ",
            "tags": [
                {
                    "key": "action",
                    "value": "withdraw_validator_rewards_all"
                },
                {
                    "key": "source-validator",
                    "value": "cosmosvaloper10505nl7yftsme9jk2glhjhta7w0475uvl4k8ju"
                }
            ],
            "gas_used": 12682,
            "gas_wanted": 200000
        },
        "Messages": [
            {
                "data": {
                    "validator_addr": "cosmosvaloper10505nl7yftsme9jk2glhjhta7w0475uvl4k8ju"
                },
                "type": "withdraw_validator_rewards_all"
            }
        ],
        "Signatures": [
            {
                "address": "7d1f49ffc44ae1bc9656523f795d7df39f5f538c",
                "sequence": 12164
            }
        ]
    },

These are the transactions of the block

@rigelrozanski

This comment has been minimized.

Copy link
Member

rigelrozanski commented Dec 6, 2018

I bet that #2958 resolved this issue... wish there was a way to simulate the blockchain easily to test if this is true of not

@rigelrozanski

This comment has been minimized.

Copy link
Member

rigelrozanski commented Dec 7, 2018

relevant now merged in (but wasn't for this testnet) #2982 (comment)

@cwgoes

This comment has been minimized.

Copy link
Contributor

cwgoes commented Dec 7, 2018

relevant now merged in (but wasn't for this testnet) #2982 (comment)

Hmm, maybe - #2982 only changed the logic in state export. Maybe we're leaving rewards in a dist info before deleting it here, though, and that's causing the problem (which we would need to fix separately).

jaekwon added a commit that referenced this issue Dec 8, 2018

Fix updateValidatorDistInfoFromPool (#3046)
Fixes regression introduced by #2984.
Continuiation of #3033 , which didn't fix the simulation issues.
(candidate) Complete solution for #3019, 9002 halt bug.

From #2984, it isn't sufficient to take the fee pool rewards of a validator. Since we don't track delegator accums (as we do with validator accums), and because onValidatorModified >updateValidatorDistInfoFromPool is also being called upon delegation updates (or at least I believe this is the reason), it is necessary to also withdraw self delegation.

TODO: I don't think self-delegation should be required to be modified here... consider using a delegation hook to do the self-delegation withdraw part instead, e.g. splitting the updateValidatorDistInfoFromPool function into two. It might not result in cleaner code, however. Think hard.
@zmanian

This comment has been minimized.

Copy link
Member

zmanian commented Jan 5, 2019

Pretty sure this can be closed.

@zmanian zmanian closed this Jan 5, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment