Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Issues with fc3b42dfb6ed25676 #1539

Closed
arajasek opened this issue Apr 13, 2020 · 4 comments
Closed

Issues with fc3b42dfb6ed25676 #1539

arajasek opened this issue Apr 13, 2020 · 4 comments

Comments

@arajasek
Copy link
Contributor

arajasek commented Apr 13, 2020 •
edited

Here are some observed invalid messages when syncing the "current" interop network (will be exported and saved when it's no lunger current).

  1. "failed to withdraw reward: failed to remove rewards: failed to delete multimap key"

Probably the most seen error, seems to come down to a lot of requests being issues to withdraw rewards for owners that aren't in reward state's st.RewardMap

2020-04-13T15:52:20.916-0400	ERROR	vm	vm/runtime.go:96	VM.Call failure: failed to withdraw reward: failed to remove rewards: failed to delete multimap key {?
                                                                                                                                                                              } root bafy2bzaceblwk4xkm4xq2plnncu4uk4fvpmc23izeepmh272cp2j76pvwtnau: map delete failed in node bafy2bzaceblwk4xkm4xq2plnncu4uk4fvpmc23izeepmh272cp2j76pvwtnau key ?
                                                                                                                                                     : not found (RetCode=20):
    github.com/filecoin-project/specs-actors/actors/builtin/reward.Actor.WithdrawReward.func1
        /Users/raj/go/pkg/mod/github.com/filecoin-project/specs-actors@v0.0.0-20200324235424-aef9b20a9fb1/actors/builtin/reward/reward_actor.go:135
github.com/filecoin-project/lotus/chain/vm.(*Runtime).shimCall.func1
	/Users/raj/lotus/chain/vm/runtime.go:96
runtime.gopanic
	/usr/local/Cellar/go/1.14/libexec/src/runtime/panic.go:967
github.com/filecoin-project/lotus/chain/vm.(*Runtime).Abortf
	/Users/raj/lotus/chain/vm/runtime.go:248
github.com/filecoin-project/specs-actors/actors/builtin/reward.Actor.WithdrawReward.func1
	/Users/raj/go/pkg/mod/github.com/filecoin-project/specs-actors@v0.0.0-20200324235424-aef9b20a9fb1/actors/builtin/reward/reward_actor.go:135
github.com/filecoin-project/lotus/chain/vm.(*shimStateHandle).Transaction
	/Users/raj/lotus/chain/vm/runtime.go:386
github.com/filecoin-project/specs-actors/actors/builtin/reward.Actor.WithdrawReward
	/Users/raj/go/pkg/mod/github.com/filecoin-project/specs-actors@v0.0.0-20200324235424-aef9b20a9fb1/actors/builtin/reward/reward_actor.go:131
reflect.Value.call
	/usr/local/Cellar/go/1.14/libexec/src/reflect/value.go:460
reflect.Value.Call
	/usr/local/Cellar/go/1.14/libexec/src/reflect/value.go:321
github.com/filecoin-project/lotus/chain/vm.(*invoker).transform.func2.1
	/Users/raj/lotus/chain/vm/invoker.go:151
github.com/filecoin-project/lotus/chain/vm.(*Runtime).shimCall
	/Users/raj/lotus/chain/vm/runtime.go:105
github.com/filecoin-project/lotus/chain/vm.(*invoker).transform.func2
	/Users/raj/lotus/chain/vm/invoker.go:150
reflect.callReflect
	/usr/local/Cellar/go/1.14/libexec/src/reflect/value.go:549
reflect.makeFuncStub
	/usr/local/Cellar/go/1.14/libexec/src/reflect/asm_amd64.s:20
github.com/filecoin-project/lotus/chain/vm.(*invoker).Invoke
	/Users/raj/lotus/chain/vm/invoker.go:73
github.com/filecoin-project/lotus/chain/vm.(*VM).Invoke
	/Users/raj/lotus/chain/vm/vm.go:611
github.com/filecoin-project/lotus/chain/vm.(*VM).send
	/Users/raj/lotus/chain/vm/vm.go:248
github.com/filecoin-project/lotus/chain/vm.(*VM).ApplyMessage
	/Users/raj/lotus/chain/vm/vm.go:388
github.com/filecoin-project/lotus/chain/stmgr.(*StateManager).ApplyBlocks
	/Users/raj/lotus/chain/stmgr/stmgr.go:171
github.com/filecoin-project/lotus/chain/stmgr.(*StateManager).computeTipSetState
	/Users/raj/lotus/chain/stmgr/stmgr.go:332
github.com/filecoin-project/lotus/chain/stmgr.(*StateManager).TipSetState
	/Users/raj/lotus/chain/stmgr/stmgr.go:112
github.com/filecoin-project/lotus/chain.(*Syncer).ValidateBlock
	/Users/raj/lotus/chain/sync.go:584
github.com/filecoin-project/lotus/chain.(*Syncer).ValidateTipSet
	/Users/raj/lotus/chain/sync.go:459
github.com/filecoin-project/lotus/chain.(*Syncer).syncMessagesAndCheckState.func1
	/Users/raj/lotus/chain/sync.go:1103
github.com/filecoin-project/lotus/chain.(*Syncer).iterFullTipsets
	/Users/raj/lotus/chain/sync.go:1173
github.com/filecoin-project/lotus/chain.(*Syncer).syncMessagesAndCheckState
	/Users/raj/lotus/chain/sync.go:1101
github.com/filecoin-project/lotus/chain.(*Syncer).collectChain
	/Users/raj/lotus/chain/sync.go:1247
github.com/filecoin-project/lotus/chain.(*Syncer).Sync
	/Users/raj/lotus/chain/sync.go:413
github.com/filecoin-project/lotus/chain.(*SyncManager).syncWorker
	/Users/raj/lotus/chain/sync_manager.go:386
2020-04-13T15:52:20.916-0400	WARN	vm	vm/vm.go:405	Send actor error	{"from": "t3slerb56clb4lywdk2d75y3fmcftx7ml2jdyf26mffdf2ae5gafsj5okmbdfjdpf6oip6kpt4jhkymn3rrpoa", "to": "t02", "nonce": 225, "method": "3", "height": "1612", "error": "failed to withdraw reward: failed to remove rewards: failed to delete multimap key {\u0000\ufffd\u000c} root bafy2bzaceblwk4xkm4xq2plnncu4uk4fvpmc23izeepmh272cp2j76pvwtnau: map delete failed in node bafy2bzaceblwk4xkm4xq2plnncu4uk4fvpmc23izeepmh272cp2j76pvwtnau key \u0000\ufffd\u000c: not found (RetCode=20):\n    github.com/filecoin-project/specs-actors/actors/builtin/reward.Actor.WithdrawReward.func1\n        /Users/raj/go/pkg/mod/github.com/filecoin-project/specs-actors@v0.0.0-20200324235424-aef9b20a9fb1/actors/builtin/reward/reward_actor.go:135"}
  1. "PoSt submission too late"
2020-04-13T15:52:20.916-0400	ERROR	vm	vm/runtime.go:96	VM.Call failure: PoSt submission too late (RetCode=20):
    github.com/filecoin-project/specs-actors/actors/builtin/miner.Actor.SubmitWindowedPoSt.func1
        /Users/raj/go/pkg/mod/github.com/filecoin-project/specs-actors@v0.0.0-20200324235424-aef9b20a9fb1/actors/builtin/miner/miner_actor.go:167
github.com/filecoin-project/lotus/chain/vm.(*Runtime).shimCall.func1
	/Users/raj/lotus/chain/vm/runtime.go:96
runtime.gopanic
	/usr/local/Cellar/go/1.14/libexec/src/runtime/panic.go:967
github.com/filecoin-project/lotus/chain/vm.(*Runtime).Abortf
	/Users/raj/lotus/chain/vm/runtime.go:248
github.com/filecoin-project/specs-actors/actors/builtin/miner.Actor.SubmitWindowedPoSt.func1
	/Users/raj/go/pkg/mod/github.com/filecoin-project/specs-actors@v0.0.0-20200324235424-aef9b20a9fb1/actors/builtin/miner/miner_actor.go:167
github.com/filecoin-project/lotus/chain/vm.(*shimStateHandle).Transaction
	/Users/raj/lotus/chain/vm/runtime.go:386
github.com/filecoin-project/specs-actors/actors/builtin/miner.Actor.SubmitWindowedPoSt
	/Users/raj/go/pkg/mod/github.com/filecoin-project/specs-actors@v0.0.0-20200324235424-aef9b20a9fb1/actors/builtin/miner/miner_actor.go:163
reflect.Value.call
	/usr/local/Cellar/go/1.14/libexec/src/reflect/value.go:460
reflect.Value.Call
	/usr/local/Cellar/go/1.14/libexec/src/reflect/value.go:321
github.com/filecoin-project/lotus/chain/vm.(*invoker).transform.func2.1
	/Users/raj/lotus/chain/vm/invoker.go:151
github.com/filecoin-project/lotus/chain/vm.(*Runtime).shimCall
	/Users/raj/lotus/chain/vm/runtime.go:105
github.com/filecoin-project/lotus/chain/vm.(*invoker).transform.func2
	/Users/raj/lotus/chain/vm/invoker.go:150
reflect.callReflect
	/usr/local/Cellar/go/1.14/libexec/src/reflect/value.go:549
reflect.makeFuncStub
	/usr/local/Cellar/go/1.14/libexec/src/reflect/asm_amd64.s:20
github.com/filecoin-project/lotus/chain/vm.(*invoker).Invoke
	/Users/raj/lotus/chain/vm/invoker.go:73
github.com/filecoin-project/lotus/chain/vm.(*VM).Invoke
	/Users/raj/lotus/chain/vm/vm.go:611
github.com/filecoin-project/lotus/chain/vm.(*VM).send
	/Users/raj/lotus/chain/vm/vm.go:248
github.com/filecoin-project/lotus/chain/vm.(*VM).ApplyMessage
	/Users/raj/lotus/chain/vm/vm.go:388
github.com/filecoin-project/lotus/chain/stmgr.(*StateManager).ApplyBlocks
	/Users/raj/lotus/chain/stmgr/stmgr.go:171
github.com/filecoin-project/lotus/chain/stmgr.(*StateManager).computeTipSetState
	/Users/raj/lotus/chain/stmgr/stmgr.go:332
github.com/filecoin-project/lotus/chain/stmgr.(*StateManager).TipSetState
	/Users/raj/lotus/chain/stmgr/stmgr.go:112
github.com/filecoin-project/lotus/chain.(*Syncer).ValidateBlock
	/Users/raj/lotus/chain/sync.go:584
github.com/filecoin-project/lotus/chain.(*Syncer).ValidateTipSet
	/Users/raj/lotus/chain/sync.go:459
github.com/filecoin-project/lotus/chain.(*Syncer).syncMessagesAndCheckState.func1
	/Users/raj/lotus/chain/sync.go:1103
github.com/filecoin-project/lotus/chain.(*Syncer).iterFullTipsets
	/Users/raj/lotus/chain/sync.go:1173
github.com/filecoin-project/lotus/chain.(*Syncer).syncMessagesAndCheckState
	/Users/raj/lotus/chain/sync.go:1101
github.com/filecoin-project/lotus/chain.(*Syncer).collectChain
	/Users/raj/lotus/chain/sync.go:1247
github.com/filecoin-project/lotus/chain.(*Syncer).Sync
	/Users/raj/lotus/chain/sync.go:413
github.com/filecoin-project/lotus/chain.(*SyncManager).syncWorker
	/Users/raj/lotus/chain/sync_manager.go:386
2020-04-13T15:52:20.916-0400	WARN	vm	vm/vm.go:405	Send actor error	{"from": "t3ujloej6enllrnbxn7nsm7var4erzid34pq4ocbei5muzzew7h3ucffbqenb2fryhm47m4k5o7ago2pc4xfha", "to": "t01735", "nonce": 795, "method": "5", "height": "1612", "error": "PoSt submission too late (RetCode=20):\n    github.com/filecoin-project/specs-actors/actors/builtin/miner.Actor.SubmitWindowedPoSt.func1\n        /Users/raj/go/pkg/mod/github.com/filecoin-project/specs-actors@v0.0.0-20200324235424-aef9b20a9fb1/actors/builtin/miner/miner_actor.go:167"}
  1. invalid seals
2020-04-13T15:44:59.376-0400	ERROR	vm	vm/runtime.go:96	VM.Call failure: invalid seal {SectorID:{Miner:1851 Number:1398} OnChain:1398 Randomness:[93 223 69 241 133 191 41 197 148 215 62 172 182 222 48 99 237 9 127 252 243 170 148 108 220 28 62 197 157 94 72 80] InteractiveRandomness:[159 35 84 14 68 200 157 186 184 73 148 57 225 3 184 135 42 34 124 162 47 252 52 132 249 51 224 179 220 186 46 51] UnsealedCID:bafk4chzahfla46ytve5qpisd7utsb75hzm7b2lsqlkzwfhtz6rrrgujm3ida}: invalid proof (RetCode=20):
    github.com/filecoin-project/specs-actors/actors/builtin/miner.Actor.verifySeal
        /Users/raj/go/pkg/mod/github.com/filecoin-project/specs-actors@v0.0.0-20200324235424-aef9b20a9fb1/actors/builtin/miner/miner_actor.go:1020
github.com/filecoin-project/lotus/chain/vm.(*Runtime).shimCall.func1
	/Users/raj/lotus/chain/vm/runtime.go:96
runtime.gopanic
	/usr/local/Cellar/go/1.14/libexec/src/runtime/panic.go:967
github.com/filecoin-project/lotus/chain/vm.(*Runtime).Abortf
	/Users/raj/lotus/chain/vm/runtime.go:248
github.com/filecoin-project/specs-actors/actors/builtin/miner.Actor.verifySeal
	/Users/raj/go/pkg/mod/github.com/filecoin-project/specs-actors@v0.0.0-20200324235424-aef9b20a9fb1/actors/builtin/miner/miner_actor.go:1020
github.com/filecoin-project/specs-actors/actors/builtin/miner.Actor.ProveCommitSector
	/Users/raj/go/pkg/mod/github.com/filecoin-project/specs-actors@v0.0.0-20200324235424-aef9b20a9fb1/actors/builtin/miner/miner_actor.go:302
reflect.Value.call
	/usr/local/Cellar/go/1.14/libexec/src/reflect/value.go:460
reflect.Value.Call
	/usr/local/Cellar/go/1.14/libexec/src/reflect/value.go:321
github.com/filecoin-project/lotus/chain/vm.(*invoker).transform.func2.1
	/Users/raj/lotus/chain/vm/invoker.go:151
github.com/filecoin-project/lotus/chain/vm.(*Runtime).shimCall
	/Users/raj/lotus/chain/vm/runtime.go:105
github.com/filecoin-project/lotus/chain/vm.(*invoker).transform.func2
	/Users/raj/lotus/chain/vm/invoker.go:150
reflect.callReflect
	/usr/local/Cellar/go/1.14/libexec/src/reflect/value.go:549
reflect.makeFuncStub
	/usr/local/Cellar/go/1.14/libexec/src/reflect/asm_amd64.s:20
github.com/filecoin-project/lotus/chain/vm.(*invoker).Invoke
	/Users/raj/lotus/chain/vm/invoker.go:73
github.com/filecoin-project/lotus/chain/vm.(*VM).Invoke
	/Users/raj/lotus/chain/vm/vm.go:611
github.com/filecoin-project/lotus/chain/vm.(*VM).send
	/Users/raj/lotus/chain/vm/vm.go:248
github.com/filecoin-project/lotus/chain/vm.(*VM).ApplyMessage
	/Users/raj/lotus/chain/vm/vm.go:388
github.com/filecoin-project/lotus/chain/stmgr.(*StateManager).ApplyBlocks
	/Users/raj/lotus/chain/stmgr/stmgr.go:171
github.com/filecoin-project/lotus/chain/stmgr.(*StateManager).computeTipSetState
	/Users/raj/lotus/chain/stmgr/stmgr.go:332
github.com/filecoin-project/lotus/chain/stmgr.(*StateManager).TipSetState
	/Users/raj/lotus/chain/stmgr/stmgr.go:112
github.com/filecoin-project/lotus/chain.(*Syncer).ValidateBlock
	/Users/raj/lotus/chain/sync.go:584
github.com/filecoin-project/lotus/chain.(*Syncer).ValidateTipSet
	/Users/raj/lotus/chain/sync.go:459
github.com/filecoin-project/lotus/chain.(*Syncer).syncMessagesAndCheckState.func1
	/Users/raj/lotus/chain/sync.go:1103
github.com/filecoin-project/lotus/chain.(*Syncer).iterFullTipsets
	/Users/raj/lotus/chain/sync.go:1173
github.com/filecoin-project/lotus/chain.(*Syncer).syncMessagesAndCheckState
	/Users/raj/lotus/chain/sync.go:1101
github.com/filecoin-project/lotus/chain.(*Syncer).collectChain
	/Users/raj/lotus/chain/sync.go:1247
github.com/filecoin-project/lotus/chain.(*Syncer).Sync
	/Users/raj/lotus/chain/sync.go:413
github.com/filecoin-project/lotus/chain.(*SyncManager).syncWorker
	/Users/raj/lotus/chain/sync_manager.go:386
2020-04-13T15:44:59.377-0400	WARN	vm	vm/vm.go:405	Send actor error	{"from": "t3v2yprbneuleld2h7xddkegrqbga3gwdgpfy7ny56ocw5e57lep2v65r6k25xv6y5p76ebtngowxyi662ypia", "to": "t01851", "nonce": 2436, "method": "8", "height": "1236", "error": "invalid seal {SectorID:{Miner:1851 Number:1398} OnChain:1398 Randomness:[93 223 69 241 133 191 41 197 148 215 62 172 182 222 48 99 237 9 127 252 243 170 148 108 220 28 62 197 157 94 72 80] InteractiveRandomness:[159 35 84 14 68 200 157 186 184 73 148 57 225 3 184 135 42 34 124 162 47 252 52 132 249 51 224 179 220 186 46 51] UnsealedCID:bafk4chzahfla46ytve5qpisd7utsb75hzm7b2lsqlkzwfhtz6rrrgujm3ida}: invalid proof (RetCode=20):\n    github.com/filecoin-project/specs-actors/actors/builtin/miner.Actor.verifySeal\n        /Users/raj/go/pkg/mod/github.com/filecoin-project/specs-actors@v0.0.0-20200324235424-aef9b20a9fb1/actors/builtin/miner/miner_actor.go:1020"}
@arajasek
Copy link
Contributor Author

  1. sector already committed
2020-04-13T16:53:05.717-0400	ERROR	vm	vm/runtime.go:96	VM.Call failure: sector 24 already committed (RetCode=16):
    github.com/filecoin-project/specs-actors/actors/builtin/miner.Actor.PreCommitSector
        /Users/raj/go/pkg/mod/github.com/filecoin-project/specs-actors@v0.0.0-20200324235424-aef9b20a9fb1/actors/builtin/miner/miner_actor.go:230
github.com/filecoin-project/lotus/chain/vm.(*Runtime).shimCall.func1
	/Users/raj/lotus/chain/vm/runtime.go:96
runtime.gopanic
	/usr/local/Cellar/go/1.14/libexec/src/runtime/panic.go:967
github.com/filecoin-project/lotus/chain/vm.(*Runtime).Abortf
	/Users/raj/lotus/chain/vm/runtime.go:248
github.com/filecoin-project/specs-actors/actors/builtin/miner.Actor.PreCommitSector
	/Users/raj/go/pkg/mod/github.com/filecoin-project/specs-actors@v0.0.0-20200324235424-aef9b20a9fb1/actors/builtin/miner/miner_actor.go:230
reflect.Value.call
	/usr/local/Cellar/go/1.14/libexec/src/reflect/value.go:460
reflect.Value.Call
	/usr/local/Cellar/go/1.14/libexec/src/reflect/value.go:321
github.com/filecoin-project/lotus/chain/vm.(*invoker).transform.func2.1
	/Users/raj/lotus/chain/vm/invoker.go:151
github.com/filecoin-project/lotus/chain/vm.(*Runtime).shimCall
	/Users/raj/lotus/chain/vm/runtime.go:105
github.com/filecoin-project/lotus/chain/vm.(*invoker).transform.func2
	/Users/raj/lotus/chain/vm/invoker.go:150
reflect.callReflect
	/usr/local/Cellar/go/1.14/libexec/src/reflect/value.go:549
reflect.makeFuncStub
	/usr/local/Cellar/go/1.14/libexec/src/reflect/asm_amd64.s:20
github.com/filecoin-project/lotus/chain/vm.(*invoker).Invoke
	/Users/raj/lotus/chain/vm/invoker.go:73
github.com/filecoin-project/lotus/chain/vm.(*VM).Invoke
	/Users/raj/lotus/chain/vm/vm.go:611
github.com/filecoin-project/lotus/chain/vm.(*VM).send
	/Users/raj/lotus/chain/vm/vm.go:248
github.com/filecoin-project/lotus/chain/vm.(*VM).ApplyMessage
	/Users/raj/lotus/chain/vm/vm.go:388
github.com/filecoin-project/lotus/chain/stmgr.(*StateManager).ApplyBlocks
	/Users/raj/lotus/chain/stmgr/stmgr.go:171
github.com/filecoin-project/lotus/chain/stmgr.(*StateManager).computeTipSetState
	/Users/raj/lotus/chain/stmgr/stmgr.go:332
github.com/filecoin-project/lotus/chain/stmgr.(*StateManager).TipSetState
	/Users/raj/lotus/chain/stmgr/stmgr.go:112
github.com/filecoin-project/lotus/chain.(*Syncer).ValidateBlock
	/Users/raj/lotus/chain/sync.go:584
github.com/filecoin-project/lotus/chain.(*Syncer).ValidateTipSet
	/Users/raj/lotus/chain/sync.go:459
github.com/filecoin-project/lotus/chain.(*Syncer).syncMessagesAndCheckState.func1
	/Users/raj/lotus/chain/sync.go:1103
github.com/filecoin-project/lotus/chain.(*Syncer).iterFullTipsets
	/Users/raj/lotus/chain/sync.go:1173
github.com/filecoin-project/lotus/chain.(*Syncer).syncMessagesAndCheckState
	/Users/raj/lotus/chain/sync.go:1101
github.com/filecoin-project/lotus/chain.(*Syncer).collectChain
	/Users/raj/lotus/chain/sync.go:1247
github.com/filecoin-project/lotus/chain.(*Syncer).Sync
	/Users/raj/lotus/chain/sync.go:413
github.com/filecoin-project/lotus/chain.(*SyncManager).syncWorker
	/Users/raj/lotus/chain/sync_manager.go:386
2020-04-13T16:53:05.717-0400	WARN	vm	vm/vm.go:405	Send actor error	{"from": "t3qgcppefjdey7gq6hfyrokiceugzdjqlogsymdbpfkdr5cqbrmvejdnjwvqi5tbjrgvuy7l3372ykzdhwxh2q", "to": "t01632", "nonce": 418, "method": "7", "height": "3388", "error": "sector 24 already committed (RetCode=16):\n    github.com/filecoin-project/specs-actors/actors/builtin/miner.Actor.PreCommitSector\n        /Users/raj/go/pkg/mod/github.com/filecoin-project/specs-actors@v0.0.0-20200324235424-aef9b20a9fb1/actors/builtin/miner/miner_actor.go:230"}

@arajasek
Copy link
Contributor Author 

lotus(2948,0x700007c4c000) malloc: can't allocate region
:*** mach_vm_map(size=3458764513903869952, flags: 60000100) failed (error code=3)
lotus(2948,0x700007c4c000) malloc: *** set a breakpoint in malloc_error_break to debug
memory allocation of 3458764513903867612 bytes failedSIGABRT: abort
PC=0x7fff6ed6a33a m=23 sigcode=0

goroutine 0 [idle]:
runtime: unknown pc 0x7fff6ed6a33a
stack: frame={sp:0x700007c4b8a8, fp:0x0} stack=[0x700007bcc350,0x700007c4bf50)
0000700007c4b7a8:  0000000000000000  0000000000000000 
0000700007c4b7b8:  0000000000000000  0000000000000000 
0000700007c4b7c8:  0000000000000000  0000000000000000 
0000700007c4b7d8:  0000000000000000  0000000000000000 
0000700007c4b7e8:  0000000000000000  0000000000000000 
0000700007c4b7f8:  0000000000000000  000000004d8810b0 
0000700007c4b808:  000000c01ec00002  0000700007c4b8e0 
0000700007c4b818:  0000000000000000  0000700007c4b8a0 
0000700007c4b828:  000000000554df2b  0003000000004f78 
0000700007c4b838:  00000000066ab048  0000000000000002 
0000700007c4b848:  0000000000000000  00007fff6edda16f 
0000700007c4b858:  0000700007c4b8f0  0000000000000001 
0000700007c4b868:  0000700007c4b908  0000000000003103 
0000700007c4b878:  0000000000000000  0000000000000000 
0000700007c4b888:  3000000004f776dc  000000004cab2d10 
0000700007c4b898:  0000000000000000  0000700007c4b920 
0000700007c4b8a8: <00007fff6ee26e60  0000000000000003 
0000700007c4b8b8:  000000004d8810b0  0000700007c4b8f0 
0000700007c4b8c8:  0000000000000000  0000700007c4b910 
0000700007c4b8d8:  00007fff6ecf1808  0000000000000000 
0000700007c4b8e8:  0000000000000000  00000000fffff9df 
0000700007c4b8f8:  ffffffff00000002  3000000004f776dc 
0000700007c4b908:  000000004cab2d10  0000700007c4b920 
0000700007c4b918:  000000000556da99  0000700007c4b930 
0000700007c4b928:  000000000555de90  0000700007c4b940 
0000700007c4b938:  0000000005581399  0000700007c4b9d0 
0000700007c4b948:  0000000004e77b2f  0000700007c4b980 
0000700007c4b958:  00007fff6edd8d7a  0000000000000020 
0000700007c4b968:  0000000000000001  000000004d8810c8 
0000700007c4b978:  000000004cab2d10  0000700007c4b9f8 
0000700007c4b988:  0000000000000000  0000000000000020 
0000700007c4b998:  0000000000000001  0000700007c4b9d0 
runtime: unknown pc 0x7fff6ed6a33a
stack: frame={sp:0x700007c4b8a8, fp:0x0} stack=[0x700007bcc350,0x700007c4bf50)
0000700007c4b7a8:  0000000000000000  0000000000000000 
0000700007c4b7b8:  0000000000000000  0000000000000000 
0000700007c4b7c8:  0000000000000000  0000000000000000 
0000700007c4b7d8:  0000000000000000  0000000000000000 
0000700007c4b7e8:  0000000000000000  0000000000000000 
0000700007c4b7f8:  0000000000000000  000000004d8810b0 
0000700007c4b808:  000000c01ec00002  0000700007c4b8e0 
0000700007c4b818:  0000000000000000  0000700007c4b8a0 
0000700007c4b828:  000000000554df2b  0003000000004f78 
0000700007c4b838:  00000000066ab048  0000000000000002 
0000700007c4b848:  0000000000000000  00007fff6edda16f 
0000700007c4b858:  0000700007c4b8f0  0000000000000001 
0000700007c4b868:  0000700007c4b908  0000000000003103 
0000700007c4b878:  0000000000000000  0000000000000000 
0000700007c4b888:  3000000004f776dc  000000004cab2d10 
0000700007c4b898:  0000000000000000  0000700007c4b920 
0000700007c4b8a8: <00007fff6ee26e60  0000000000000003 
0000700007c4b8b8:  000000004d8810b0  0000700007c4b8f0 
0000700007c4b8c8:  0000000000000000  0000700007c4b910 
0000700007c4b8d8:  00007fff6ecf1808  0000000000000000 
0000700007c4b8e8:  0000000000000000  00000000fffff9df 
0000700007c4b8f8:  ffffffff00000002  3000000004f776dc 
0000700007c4b908:  000000004cab2d10  0000700007c4b920 
0000700007c4b918:  000000000556da99  0000700007c4b930 
0000700007c4b928:  000000000555de90  0000700007c4b940 
0000700007c4b938:  0000000005581399  0000700007c4b9d0 
0000700007c4b948:  0000000004e77b2f  0000700007c4b980 
0000700007c4b958:  00007fff6edd8d7a  0000000000000020 
0000700007c4b968:  0000000000000001  000000004d8810c8 
0000700007c4b978:  000000004cab2d10  0000700007c4b9f8 
0000700007c4b988:  0000000000000000  0000000000000020 
0000700007c4b998:  0000000000000001  0000700007c4b9d0 

goroutine 2027946 [syscall]:
runtime.cgocall(0x4ca26d0, 0xc0203a96d0, 0x56521c0)
	/usr/local/Cellar/go/1.14/libexec/src/runtime/cgocall.go:133 +0x5b fp=0xc0203a9678 sp=0xc0203a9640 pc=0x40056ab
github.com/filecoin-project/filecoin-ffi/generated._Cfunc_fil_verify_post(0xbeae11b7bc7f4888, 0xd9af726b9dd97231, 0x84b1e859f7545c11, 0x6b41b709afa4f8, 0x1103, 0x6b3b4000, 0x1a93f, 0x4d8810b0, 0x1, 0x4d896280, ...)
	_cgo_gotypes.go:1666 +0x51 fp=0xc0203a96d0 sp=0xc0203a9678 pc=0x4614191
github.com/filecoin-project/filecoin-ffi/generated.FilVerifyPost.func1(0xbeae11b7bc7f4888, 0xd9af726b9dd97231, 0x84b1e859f7545c11, 0x6b41b709afa4f8, 0x1103, 0x6b3b4000, 0x1a93f, 0x4d8810b0, 0x1, 0x4d896280, ...)
	/Users/raj/lotus/extern/filecoin-ffi/generated/generated.go:559 +0x127 fp=0xc0203a97a0 sp=0xc0203a96d0 pc=0x4626537
github.com/filecoin-project/filecoin-ffi/generated.FilVerifyPost(0xbeae11b7bc7f4888, 0xd9af726b9dd97231, 0x84b1e859f7545c11, 0x6b41b709afa4f8, 0x0, 0x0, 0x0, 0x1103, 0xc05288c000, 0x1a93f, ...)
	/Users/raj/lotus/extern/filecoin-ffi/generated/generated.go:559 +0x20e fp=0xc0203a9890 sp=0xc0203a97a0 pc=0x46252de
github.com/filecoin-project/filecoin-ffi.VerifyPoSt(0xc0356abd60, 0x20, 0x20, 0xc01be6dda0, 0x1, 0x1, 0xc0215f5ee0, 0x1, 0x1, 0xc05193e000, ...)
	/Users/raj/lotus/extern/filecoin-ffi/proofs.go:78 +0x336 fp=0xc0203a9a88 sp=0xc0203a9890 pc=0x462d076
github.com/filecoin-project/sector-storage/ffiwrapper.verifyPost(0x5c3e880, 0xc020a29c20, 0xc0356abd60, 0x20, 0x20, 0xc01be6dda0, 0x1, 0x1, 0xc0215f5ee0, 0x1, ...)
	/Users/raj/go/pkg/mod/github.com/filecoin-project/sector-storage@v0.0.0-20200406195014-a6d093838576/ffiwrapper/verifier_cgo.go:142 +0xf7 fp=0xc0203a9b38 sp=0xc0203a9a88 pc=0x4667397
github.com/filecoin-project/sector-storage/ffiwrapper.proofVerifier.VerifyElectionPost(...)
	/Users/raj/go/pkg/mod/github.com/filecoin-project/sector-storage@v0.0.0-20200406195014-a6d093838576/ffiwrapper/verifier_cgo.go:129
github.com/filecoin-project/lotus/chain.(*Syncer).VerifyElectionPoStProof(0xc0003ed1a0, 0x5c3e880, 0xc020a29c20, 0xc0215c9340, 0xc02556ef80, 0x31, 0x4076dd4, 0xc022b7f2a0)
	/Users/raj/lotus/chain/sync.go:769 +0xcaf fp=0xc0203a9f30 sp=0xc0203a9b38 pc=0x49657ff
github.com/filecoin-project/lotus/chain.(*Syncer).ValidateBlock.func6(0xc000418af0, 0x403f8b6)
	/Users/raj/lotus/chain/sync.go:638 +0x5b fp=0xc0203a9f90 sp=0xc0203a9f30 pc=0x4970a4b
github.com/Gurpartap/async.Err.func1(0xc02dfac420, 0xc03647d680, 0xc01ec3b900)
	/Users/raj/go/pkg/mod/github.com/!gurpartap/async@v0.0.0-20180927173644-4f7f499dd9ee/error.go:29 +0x53 fp=0xc0203a9fc8 sp=0xc0203a9f90 pc=0x48284e3
runtime.goexit()
	/usr/local/Cellar/go/1.14/libexec/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0203a9fd0 sp=0xc0203a9fc8 pc=0x4069431
created by github.com/Gurpartap/async.Err
	/Users/raj/go/pkg/mod/github.com/!gurpartap/async@v0.0.0-20180927173644-4f7f499dd9ee/error.go:27 +0x8a

goroutine 1 [IO wait, 71 minutes]:
internal/poll.runtime_pollWait(0x8655b58, 0x72, 0x0)
	/usr/local/Cellar/go/1.14/libexec/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc000129d18, 0x72, 0x0, 0x0, 0x58fb7cb)
	/usr/local/Cellar/go/1.14/libexec/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/Cellar/go/1.14/libexec/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc000129d00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/Cellar/go/1.14/libexec/src/internal/poll/fd_unix.go:384 +0x1d4
net.(*netFD).accept(0xc000129d00, 0x203000, 0xc000544f20, 0xc0001f7720)
	/usr/local/Cellar/go/1.14/libexec/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc000597d00, 0x50, 0x50, 0x57e9a60)
	/usr/local/Cellar/go/1.14/libexec/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc000597d00, 0x582ff60, 0x757e560, 0xc000064c00, 0x30)
	/usr/local/Cellar/go/1.14/libexec/src/net/tcpsock.go:261 +0x64
github.com/multiformats/go-multiaddr-net.(*maListener).Accept(0xc000597d80, 0xc0005188c0, 0xc0001f7800, 0x400f648, 0x30)
	/Users/raj/go/pkg/mod/github.com/multiformats/go-multiaddr-net@v0.1.3/net.go:243 +0x37
github.com/multiformats/go-multiaddr-net.(*netListenerAdapter).Accept(0xc000544ed0, 0x582ff60, 0xc0004dbdd0, 0x5709580, 0x67888f0)
	/Users/raj/go/pkg/mod/github.com/multiformats/go-multiaddr-net@v0.1.3/net.go:222 +0x33
net/http.(*Server).Serve(0xc00057a0e0, 0x5c36e80, 0xc000544ed0, 0x0, 0x0)
	/usr/local/Cellar/go/1.14/libexec/src/net/http/server.go:2901 +0x25d
main.serveRPC(0x5c66e20, 0xc0006c0400, 0xc00011fd50, 0x5c56320, 0xc00055fd20, 0xc00011fd50, 0x0)
	/Users/raj/lotus/cmd/lotus/rpc.go:77 +0x5bb
main.glob..func1(0xc000481080, 0x0, 0x0)
	/Users/raj/lotus/cmd/lotus/daemon.go:207 +0xec2
gopkg.in/urfave/cli%2ev2.(*Command).Run(0x67a0720, 0xc000481020, 0x0, 0x0)
	/Users/raj/go/pkg/mod/gopkg.in/urfave/cli.v2@v2.0.0-20180128182452-d3ae77c26ac8/command.go:167 +0x6f8
gopkg.in/urfave/cli%2ev2.(*App).Run(0xc000490000, 0xc00000e080, 0x2, 0x2, 0x0, 0x0)
	/Users/raj/go/pkg/mod/gopkg.in/urfave/cli.v2@v2.0.0-20180128182452-d3ae77c26ac8/app.go:261 +0x6e3
main.main()
	/Users/raj/lotus/cmd/lotus/main.go:71 +0x5e4

goroutine 6 [select, 71 minutes]:
github.com/ipfs/go-log/writer.(*MirrorWriter).logRoutine(0xc0000a11d0)
	/Users/raj/go/pkg/mod/github.com/ipfs/go-log@v1.0.3/writer/writer.go:71 +0x128
created by github.com/ipfs/go-log/writer.NewMirrorWriter
	/Users/raj/go/pkg/mod/github.com/ipfs/go-log@v1.0.3/writer/writer.go:36 +0xb9

goroutine 7 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0000948c0)
	/Users/raj/go/pkg/mod/go.opencensus.io@v0.22.3/stats/view/worker.go:154 +0x100
created by go.opencensus.io/stats/view.init.0
	/Users/raj/go/pkg/mod/go.opencensus.io@v0.22.3/stats/view/worker.go:32 +0x57

goroutine 971 [select, 71 minutes]:
github.com/ipfs/go-bitswap/sessionrequestsplitter.(*SessionRequestSplitter).run(0xc01ed12750)
	/Users/raj/go/pkg/mod/github.com/ipfs/go-bitswap@v0.1.8/sessionrequestsplitter/sessionrequestsplitter.go:86 +0xf4
created by github.com/ipfs/go-bitswap/sessionrequestsplitter.New
	/Users/raj/go/pkg/mod/github.com/ipfs/go-bitswap@v0.1.8/sessionrequestsplitter/sessionrequestsplitter.go:43 +0xa6

goroutine 10 [select, 11 minutes]:
github.com/libp2p/go-libp2p-peerstore/pstoremem.(*memoryAddrBook).background(0xc00043b200)
	/Users/raj/go/pkg/mod/github.com/libp2p/go-libp2p-peerstore@v0.2.0/pstoremem/addr_book.go:93 +0x126
created by github.com/libp2p/go-libp2p-peerstore/pstoremem.NewAddrBook
	/Users/raj/go/pkg/mod/github.com/libp2p/go-libp

@arajasek
Copy link
Contributor Author 

2020-04-13T21:53:28.001-0400	ERROR	vm	vm/runtime.go:96	VM.Call failure: no precommitted sector 9005419 (RetCode=17):
    github.com/filecoin-project/specs-actors/actors/builtin/miner.Actor.ProveCommitSector
        /Users/raj/go/pkg/mod/github.com/filecoin-project/specs-actors@v0.0.0-20200324235424-aef9b20a9fb1/actors/builtin/miner/miner_actor.go:290
github.com/filecoin-project/lotus/chain/vm.(*Runtime).shimCall.func1
	/Users/raj/lotus/chain/vm/runtime.go:96
runtime.gopanic
	/usr/local/Cellar/go/1.14/libexec/src/runtime/panic.go:967
github.com/filecoin-project/lotus/chain/vm.(*Runtime).Abortf
	/Users/raj/lotus/chain/vm/runtime.go:248
github.com/filecoin-project/specs-actors/actors/builtin/miner.Actor.ProveCommitSector
	/Users/raj/go/pkg/mod/github.com/filecoin-project/specs-actors@v0.0.0-20200324235424-aef9b20a9fb1/actors/builtin/miner/miner_actor.go:290
reflect.Value.call
	/usr/local/Cellar/go/1.14/libexec/src/reflect/value.go:460
reflect.Value.Call
	/usr/local/Cellar/go/1.14/libexec/src/reflect/value.go:321
github.com/filecoin-project/lotus/chain/vm.(*invoker).transform.func2.1
	/Users/raj/lotus/chain/vm/invoker.go:151
github.com/filecoin-project/lotus/chain/vm.(*Runtime).shimCall
	/Users/raj/lotus/chain/vm/runtime.go:105
github.com/filecoin-project/lotus/chain/vm.(*invoker).transform.func2
	/Users/raj/lotus/chain/vm/invoker.go:150
reflect.callReflect
	/usr/local/Cellar/go/1.14/libexec/src/reflect/value.go:549
reflect.makeFuncStub
	/usr/local/Cellar/go/1.14/libexec/src/reflect/asm_amd64.s:20
github.com/filecoin-project/lotus/chain/vm.(*invoker).Invoke
	/Users/raj/lotus/chain/vm/invoker.go:73
github.com/filecoin-project/lotus/chain/vm.(*VM).Invoke
	/Users/raj/lotus/chain/vm/vm.go:611
github.com/filecoin-project/lotus/chain/vm.(*VM).send
	/Users/raj/lotus/chain/vm/vm.go:248
github.com/filecoin-project/lotus/chain/vm.(*VM).ApplyMessage
	/Users/raj/lotus/chain/vm/vm.go:388
github.com/filecoin-project/lotus/chain/stmgr.(*StateManager).ApplyBlocks
	/Users/raj/lotus/chain/stmgr/stmgr.go:171
github.com/filecoin-project/lotus/chain/stmgr.(*StateManager).computeTipSetState
	/Users/raj/lotus/chain/stmgr/stmgr.go:332
github.com/filecoin-project/lotus/chain/stmgr.(*StateManager).TipSetState
	/Users/raj/lotus/chain/stmgr/stmgr.go:112
github.com/filecoin-project/lotus/chain.(*Syncer).ValidateBlock
	/Users/raj/lotus/chain/sync.go:584
github.com/filecoin-project/lotus/chain.(*Syncer).ValidateTipSet
	/Users/raj/lotus/chain/sync.go:459
github.com/filecoin-project/lotus/chain.(*Syncer).syncMessagesAndCheckState.func1
	/Users/raj/lotus/chain/sync.go:1103
github.com/filecoin-project/lotus/chain.(*Syncer).iterFullTipsets
	/Users/raj/lotus/chain/sync.go:1173
github.com/filecoin-project/lotus/chain.(*Syncer).syncMessagesAndCheckState
	/Users/raj/lotus/chain/sync.go:1101
github.com/filecoin-project/lotus/chain.(*Syncer).collectChain
	/Users/raj/lotus/chain/sync.go:1247
github.com/filecoin-project/lotus/chain.(*Syncer).Sync
	/Users/raj/lotus/chain/sync.go:413
github.com/filecoin-project/lotus/chain.(*SyncManager).syncWorker
	/Users/raj/lotus/chain/sync_manager.go:386
2020-04-13T21:53:28.001-0400	WARN	vm	vm/vm.go:405	Send actor error	{"from": "t3sqvsu2ftsijlkvfoteswqs6fwvejndlaxwj5btmuyheuxfekmeczjlsdtanbhgwaqjbsncpmbugit22uybjq", "to": "t01969", "nonce": 281253, "method": "8", "height": "4568", "error": "no precommitted sector 9005419 (RetCode=17):\n    github.com/filecoin-project/specs-actors/actors/builtin/miner.Actor.ProveCommitSector\n        /Users/raj/go/pkg/mod/github.com/filecoin-project/specs-actors@v0.0.0-20200324235424-aef9b20a9fb1/actors/builtin/miner/miner_actor.go:290"}

@magik6k
Copy link
Contributor

magik6k commented Jul 1, 2020

I feel like all of those are addressed, if not, they should have separate issues to make tracking easier

@magik6k magik6k closed this as completed Jul 1, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@whyrusleeping @magik6k @arajasek