New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to invalidate certain transaction? As node seems stuck with one. #1568

Closed
liukun opened this Issue Jan 18, 2019 · 20 comments

Comments

Projects
None yet
3 participants
@liukun
Copy link

liukun commented Jan 18, 2019

2019-01-18 09:15:02.029 [ERR] RPCS: Failed to create new block template: failed to do final check for check connect block when making new block template: output 942d8d10866af14d660b5175ed4b9f9807e73eb96b1eea369ecae7291f5698ce:0 referenced from transaction 87c0cd6b139421108605cf1590f4981abda99b8befbcb3df065df7f7db336112:0 either does not exist or has already been spent

Cannot create new block template as node (v1.2) stucking at this message. Is newer version resolved this?

@davecgh

This comment has been minimized.

Copy link
Member

davecgh commented Jan 18, 2019

This is resolved in 1.4.0 for which there is currently an rc1 candidate released. Please update.

EDIT: As noted elsewhere, a bug has been discovered in Release Candidate 1 of 1.4.0, so miners should not update until there is a new release candidate to address the issue.

EDIT 2: See the patch below for a fix to 1.3.0 in the mean time.

@davecgh davecgh closed this Jan 18, 2019

@liukun

This comment has been minimized.

Copy link
Author

liukun commented Jan 18, 2019

Great! Thanks.

@liukun

This comment has been minimized.

Copy link
Author

liukun commented Jan 18, 2019

Oops, we update to 1.4.0 but the wrong transaction packed into an block which not accept by 1.2.0 staking nodes! So will not collect enough votes for new blocks! @davecgh

@davecgh

This comment has been minimized.

Copy link
Member

davecgh commented Jan 18, 2019

Can you provide more details on the exact error you're seeing? I fired up an old 1.2.0 node and am syncing it to the main chain currently to see if I encounter any issues.

@liukun

This comment has been minimized.

Copy link
Author

liukun commented Jan 18, 2019

@davecgh The new 1.4.0 node will generate block that not acceptable by 1.2.0 since included the mentioned transaction.

@liukun

This comment has been minimized.

Copy link
Author

liukun commented Jan 18, 2019

2019-01-18 18:59:23.522 [INF] CHAN: FORK: Block 000000000000000044256f2027b443fdaab73c6a92a703189c87a695b98c9c76 (height 310718) forks the chain at height 310717/block 0000000000000000439620a00ab281353672c842058011371e60d5b95fd6c145, but does not cause a reorganize
2019-01-18 18:59:23.522 [INF] BMGR: Processed 1 block in the last 2m13.37s (3 transactions, 2 tickets, 5 votes, 0 revocations, height 310718, 2019-01-18 18:57:14 +0800 CST)
2019-01-18 18:59:25.089 [WRN] FEES: Trying to process mined transactions at block 310718 when previous best block was at height 310718
2019-01-18 18:59:25.089 [INF] CHAN: REORGANIZE: Chain forks at 0000000000000000439620a00ab281353672c842058011371e60d5b95fd6c145 (height 310717)
2019-01-18 18:59:25.089 [INF] CHAN: REORGANIZE: Old best chain tip was 00000000000000001d845f993fa4683dcc3a93a293dff656fb53b93077dc237c (height 310718)
2019-01-18 18:59:25.089 [INF] CHAN: REORGANIZE: New best chain tip is 000000000000000044256f2027b443fdaab73c6a92a703189c87a695b98c9c76 (height 310718)
2019-01-18 18:59:25.104 [WRN] MINR: incongruent number of voters in mempool vs mempool.voters; not enough voters found

Example that new block not acceptable but wait the old-node-chain to catch up then orphan...

@davecgh

This comment has been minimized.

Copy link
Member

davecgh commented Jan 18, 2019

Alright, thanks. So, you guys went back to 1.3.0 for now, right? We'll dig into what's going on the release candidate and get it fixed to put out a new RC.

@liukun

This comment has been minimized.

Copy link
Author

liukun commented Jan 18, 2019

another

2019-01-18 19:06:12.899 [INF] CHAN: FORK: Block 0000000000000000025eb957f96d7deb064955725af579bff6adc4d505786196 (height 310719) forks the chain at height 310718/block 000000000000000044256f2027b443fdaab73c6a92a703189c87a695b98c9c76, but does not cause a reorganize
2019-01-18 19:06:12.899 [INF] RPCS: Block submitted via getwork accepted: 0000000000000000025eb957f96d7deb064955725af579bff6adc4d505786196
2019-01-18 19:06:16.013 [WRN] FEES: Trying to process mined transactions at block 310719 when previous best block was at height 310719
2019-01-18 19:06:16.013 [INF] CHAN: REORGANIZE: Chain forks at 000000000000000044256f2027b443fdaab73c6a92a703189c87a695b98c9c76 (height 310718)
2019-01-18 19:06:16.013 [INF] CHAN: REORGANIZE: Old best chain tip was 000000000000000016a4b809db6e52d612387d3f87225bdcab50899ce70936b0 (height 310719)
2019-01-18 19:06:16.013 [INF] CHAN: REORGANIZE: New best chain tip is 0000000000000000025eb957f96d7deb064955725af579bff6adc4d505786196 (height 310719)
2019-01-18 19:06:16.028 [WRN] MINR: incongruent number of voters in mempool vs mempool.voters; not enough voters found

@davecgh davecgh reopened this Jan 18, 2019

@liukun

This comment has been minimized.

Copy link
Author

liukun commented Jan 18, 2019

Yes, back to 1.3.0 for now (after resync the old chain T_T ).

@liukun

This comment has been minimized.

Copy link
Author

liukun commented Jan 18, 2019

2019-01-18 20:00:38.531 [INF] BMGR: Rejected block 00000000000000001905d9167008829dc055b7320808a788549ea6cc3f8092df from 192.99.56.213:9108 (outbound): output 2ebfc374f31ceb18343619e028cc0924246ca5f597bc9408ff10cd8d82ea2f7b:0 referenced from transaction 495559c93da67b88b3ad92d8ecc4ec54df47fa441fdc56dc4f21728ee2782fe7:0 either does not exist or has already been spent

Rejected message on v1.3.0

@davecgh

This comment has been minimized.

Copy link
Member

davecgh commented Jan 18, 2019

Is that on a freshly synced v1.3.0? I just created a newly synced 1.3.0 and manually submitted the block to it:

2019-01-18 06:13:58.400 [INF] RPCS: Accepted block 00000000000000001905d9167008829dc055b7320808a788549ea6cc3f8092df via submitblock
$ dcrctl version
{
  "dcrd": {
    "versionstring": "1.3.0+release",
@liukun

This comment has been minimized.

Copy link
Author

liukun commented Jan 18, 2019

2019-01-18 18:00:27.678 [INF] DCRD: Version 1.3.0+dev (Go version go1.11.4)
2019-01-18 18:00:27.678 [INF] DCRD: Home dir: .
2019-01-18 19:30:59.080 [INF] BMGR: Syncing to block height 310724 from peer 192.99.56.213:9108
2019-01-18 19:30:59.613 [INF] BMGR: New valid peer 23.20.248.113:9108 (outbound) (/dcrwire:0.3.0/dcrd:1.3.0/)
2019-01-18 19:31:27.041 [INF] BMGR: Processed 1 block in the last 1m47.87s (1 transaction, 4 tickets, 5 votes, 0 revocations, height 310724, 2019-01-18 19:30:46 +0800 CST)
2019-01-18 19:31:54.875 [INF] BMGR: New valid peer 107.170.230.148:9108 (outbound) (/dcrwire:0.3.0/dcrd:1.3.0/)
2019-01-18 19:35:41.535 [INF] BMGR: Processed 1 block in the last 4m14.49s (11 transactions, 0 tickets, 5 votes, 0 revocations, height 310725, 2019-01-18 19:06:14 +0800 CST)
2019-01-18 19:37:14.618 [INF] BMGR: Processed 1 block in the last 1m33.08s (6 transactions, 7 tickets, 5 votes, 0 revocations, height 310726, 2019-01-18 19:07:48 +0800 CST)
2019-01-18 20:00:38.531 [INF] BMGR: Rejected block 00000000000000001905d9167008829dc055b7320808a788549ea6cc3f8092df from 192.99.56.213:9108 (outbound): output 2ebfc374f31ceb18343619e028cc0924246ca5f597bc9408ff10cd8d82ea2f7b:0 referenced from transaction 495559c93da67b88b3ad92d8ecc4ec54df47fa441fdc56dc4f21728ee2782fe7:0 either does not exist or has already been spent
2019-01-18 20:28:28.564 [INF] BMGR: Processed 1 block in the last 8m24.2s (9 transactions, 4 tickets, 5 votes, 0 revocations, height 310733, 2019-01-18 19:58:52 +0800 CST)
2019-01-18 20:30:32.376 [INF] BMGR: Processed 1 block in the last 2m3.81s (3 transactions, 7 tickets, 5 votes, 0 revocations, height 310734, 2019-01-18 20:01:15 +0800 CST)
2019-01-18 20:31:14.373 [INF] BMGR: Lost peer 207.154.229.177:9108 (outbound)
2019-01-18 20:31:16.584 [INF] BMGR: New valid peer 5.135.183.154:9108 (outbound) (/dcrwire:0.3.0/dcrd:1.2.0/)
2019-01-18 20:35:53.260 [INF] BMGR: Processed 1 block in the last 5m20.88s (3 transactions, 1 ticket, 5 votes, 0 revocations, height 310735, 2019-01-18 20:06:01 +0800 CST)
2019-01-18 20:46:54.879 [INF] BMGR: Processed 1 block in the last 11m1.61s (7 transactions, 5 tickets, 5 votes, 0 revocations, height 310737, 2019-01-18 20:17:07 +0800 CST)
2019-01-18 20:49:14.655 [INF] BMGR: Lost peer 209.250.242.111:9108 (outbound)
2019-01-18 20:51:14.159 [INF] BMGR: Processed 1 block in the last 4m19.28s (3 transactions, 6 tickets, 5 votes, 0 revocations, height 310738, 2019-01-18 20:51:00 +0800 CST)
2019-01-18 20:56:15.678 [INF] RPCS: Block submitted via getwork accepted: 00000000000000000b68e4a3d5efbf94d7682d12ed5af7f2caa4efa26513f0e6
2019-01-18 20:57:02.125 [INF] BMGR: Rejected block 00000000000000001078fc4bd0859d9f0f3fbe6768f8368eea2910edaad949cf from 192.99.56.213:9108 (outbound): output 2ebfc374f31ceb18343619e028cc0924246ca5f597bc9408ff10cd8d82ea2f7b:0 referenced from transaction 495559c93da67b88b3ad92d8ecc4ec54df47fa441fdc56dc4f21728ee2782fe7:0 either does not exist or has already been spent
2019-01-18 21:04:02.058 [INF] BMGR: Processed 1 block in the last 12m47.89s (9 transactions, 5 tickets, 5 votes, 0 revocations, height 310740, 2019-01-18 20:34:26 +0800 CST)

@liukun

This comment has been minimized.

Copy link
Author

liukun commented Jan 18, 2019

Above partial logs including two rejected blocks, all caused by "output aa:0 referenced from transaction bb:0 either does not exist or has already been spent".

@bitkevin

This comment has been minimized.

Copy link

bitkevin commented Jan 21, 2019

v1.4.0-rc1 is shit, please do NOT upgrade or you will get lots of orphan blocks.

@davecgh

This comment has been minimized.

Copy link
Member

davecgh commented Jan 21, 2019

As an update, we've tracked down the issue and have a PR open to address it. Thanks to those who reported the issue and helped test the release candidate to discover the issue. This is why there are release candidates.

I'll leave this issue open until a new RC is released which addresses it.

@liukun

This comment has been minimized.

Copy link
Author

liukun commented Jan 22, 2019

@davecgh Thanks for the update. BTW, any temporary patch available to old versions (v1.3) that prevent the node from being stuck on final checks when getwork? I have to restart node each time. Or some information about which commit introduced the rejected transaction?

Logs when stuck:

2019-01-22 13:20:02.496 [ERR] RPCS: Failed to create new block template: failed to do final check for check connect block when making new block template: output 4011a1339662a1ea970f2a1f96892d41c64a20206f8487f2e7bcb6b574fd00d1:0 referenced from transaction 16e879e24a741362da91f5
82a37889caccd14f922ce3019d6270365b7225ab0b:0 either does not exist or has already been spent
2019-01-22 13:20:02.502 [ERR] RPCS: Failed to create new block template: failed to do final check for check connect block when making new block template: output 4011a1339662a1ea970f2a1f96892d41c64a20206f8487f2e7bcb6b574fd00d1:0 referenced from transaction 16e879e24a741362da91f5
82a37889caccd14f922ce3019d6270365b7225ab0b:0 either does not exist or has already been spent
2019-01-22 13:20:02.593 [ERR] RPCS: Failed to create new block template: failed to do final check for check connect block when making new block template: output 4011a1339662a1ea970f2a1f96892d41c64a20206f8487f2e7bcb6b574fd00d1:0 referenced from transaction 16e879e24a741362da91f5
82a37889caccd14f922ce3019d6270365b7225ab0b:0 either does not exist or has already been spent
@davecgh

This comment has been minimized.

Copy link
Member

davecgh commented Jan 22, 2019

The following patch will stop that from happening on 1.3.0:

diff --git a/mempool/mempool.go b/mempool/mempool.go
index f4ac8846..4915cb73 100644
--- a/mempool/mempool.go
+++ b/mempool/mempool.go
@@ -858,8 +858,22 @@ func (mp *TxPool) maybeAcceptTransaction(tx *dcrutil.Tx, isNew, rateLimit, allow
                tx.SetTree(wire.TxTreeStake)
        }

-       // Reject votes before stake validation height.
        isVote := txType == stake.TxTypeSSGen
+       if msgTx.Version >= 2 && !isVote {
+               for _, txIn := range msgTx.TxIn {
+                       sequenceNum := txIn.Sequence
+                       if sequenceNum&wire.SequenceLockTimeDisabled != 0 {
+                               continue
+                       }
+
+                       str := "violates sequence lock consensus bug"
+                       return nil, txRuleError(wire.RejectInvalid, str)
+               }
+       }
+
+       // Reject votes before stake validation height.
        stakeValidationHeight := mp.cfg.ChainParams.StakeValidationHeight
        if isVote && nextBlockHeight < stakeValidationHeight {
                str := fmt.Sprintf("votes are not valid until block height %d (next "+
@davecgh

This comment has been minimized.

Copy link
Member

davecgh commented Jan 24, 2019

Has the patch been working for you? Given that we've tested the it pretty thoroughly, I'm confident that it works properly, but more confirmation is always a good thing.

@liukun

This comment has been minimized.

Copy link
Author

liukun commented Jan 25, 2019

@davecgh The patch works properly. The v1.3 nodes run continuously ever after. Thank you for your support.

@davecgh

This comment has been minimized.

Copy link
Member

davecgh commented Jan 29, 2019

This is resolved by 1.4.0-rc3.

@davecgh davecgh closed this Jan 29, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment