Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Network froze on 1874879 May 22, 2023 4:02:30 AM UTC / daemon can't start too #5383

Open
gades opened this issue May 22, 2023 · 42 comments
Open

Comments

@gades
Copy link

gades commented May 22, 2023

Daemon can't start in case restart - invalid blocks and after resync it stops on 1874879

@gades gades changed the title Network froze on 1874879 May 22, 2023 4:02:30 AM UTC / daemon can start too Network froze on 1874879 May 22, 2023 4:02:30 AM UTC / daemon can't start too May 22, 2023
@knst
Copy link
Collaborator

knst commented May 22, 2023

There was a problem during v19 activation of Dash Core. The chain is stalled and currently not producing blocks. We have everyone investigating the issue.

https://twitter.com/SamWestrich/status/1660513873546596354

@gades
Copy link
Author

gades commented May 22, 2023

"v19": { "status": "active", "start_time": 1682380800, "timeout": 1714003200, "since": 1874880 }

yeah, this issue is in bip9 softforks and  why was not tested it in testnet before roll out in main net?

@gituser
Copy link

gituser commented May 22, 2023

After update to v19.0.0 from v18.2.2 getting this error:

2023-05-22T10:41:54Z Using obfuscation key for /home/dash/.dash/chainstate: 19f6def09d7d0cc9
2023-05-22T10:41:55Z Loaded best chain: hashBestChain=000000000000001e28817f2896e1dcf993dfab73f4b80a64e9e3be5680d2b3d7 height=1874880 date=2023-05-22T04:13:10Z progress=0.999918
2023-05-22T10:41:55Z CDeterministicMNManager::MigrateDBIfNeeded -- upgrading DB to migrate MN type
2023-05-22T10:41:55Z CDeterministicMNManager::MigrateDBIfNeeded -- migration already done. skipping.
2023-05-22T10:41:55Z init message: Verifying blocks...
2023-05-22T10:41:55Z Verifying last 6 blocks at level 3
2023-05-22T10:41:55Z [0%]...[16%]...[33%]...ERROR: UndoSpecialTxsInBlock -- failed: UpdateMN: Can't update a masternode e01979c6ce16c95541310d879f91e2324164bab362f5d21f903f33941fa0b5e5 with a duplicate pubKeyOperator=000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

2023-05-22T10:42:01Z ERROR: VerifyDB(): *** irrecoverable inconsistency in block data at 1874878, hash=0000000000000009be0f8add92c7d1d92654806f76f983f1e089a20cc0597926
2023-05-22T10:42:01Z : Corrupted block database detected.
Please restart with -reindex or -reindex-chainstate to recover.
2023-05-22T10:42:01Z Aborted block database rebuild. Exiting.
2023-05-22T10:42:01Z PrepareShutdown: In progress...
2023-05-22T10:42:01Z cl-schdlr thread exit
2023-05-22T10:42:01Z scheduler thread exit

@gades
Copy link
Author

gades commented May 22, 2023

After update to v19.0.0 from v18.2.2 getting this error:

2023-05-22T10:41:54Z Using obfuscation key for /home/dash/.dash/chainstate: 19f6def09d7d0cc9
2023-05-22T10:41:55Z Loaded best chain: hashBestChain=000000000000001e28817f2896e1dcf993dfab73f4b80a64e9e3be5680d2b3d7 height=1874880 date=2023-05-22T04:13:10Z progress=0.999918
2023-05-22T10:41:55Z CDeterministicMNManager::MigrateDBIfNeeded -- upgrading DB to migrate MN type
2023-05-22T10:41:55Z CDeterministicMNManager::MigrateDBIfNeeded -- migration already done. skipping.
2023-05-22T10:41:55Z init message: Verifying blocks...
2023-05-22T10:41:55Z Verifying last 6 blocks at level 3
2023-05-22T10:41:55Z [0%]...[16%]...[33%]...ERROR: UndoSpecialTxsInBlock -- failed: UpdateMN: Can't update a masternode e01979c6ce16c95541310d879f91e2324164bab362f5d21f903f33941fa0b5e5 with a duplicate pubKeyOperator=000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

2023-05-22T10:42:01Z ERROR: VerifyDB(): *** irrecoverable inconsistency in block data at 1874878, hash=0000000000000009be0f8add92c7d1d92654806f76f983f1e089a20cc0597926
2023-05-22T10:42:01Z : Corrupted block database detected.
Please restart with -reindex or -reindex-chainstate to recover.
2023-05-22T10:42:01Z Aborted block database rebuild. Exiting.
2023-05-22T10:42:01Z PrepareShutdown: In progress...
2023-05-22T10:42:01Z cl-schdlr thread exit
2023-05-22T10:42:01Z scheduler thread exit

the same issue with v19 and you can fix by downloading blocks from network and it will work until daemon will up

@gituser
Copy link

gituser commented May 22, 2023

the same issue with v19 and you can fix by downloading blocks from network and it will work until daemon will up

is there any way to fix this without resync?

@qwizzie
Copy link

qwizzie commented May 22, 2023

the same issue with v19 and you can fix by downloading blocks from network and it will work until daemon will up

is there any way to fix this without resync?

Untill it becomes clear what exactly the issue is and how they plan to address the issue (hotfix ?) we should just wait.
Dash blockchain explorer (https://chainz.cryptoid.info/dash/) is also still stuck (indicating the blockchain is still stallled), so i don't think a resync will help much at this moment.

@gades
Copy link
Author

gades commented May 22, 2023

They did patch ea18452 and new version is accepting blocks from miner who is using last one ...

@Har01d
Copy link

Har01d commented May 22, 2023

Upgrading to 19.1.0 doesn't help, the node is stuck at height=1874880
invalidateblock 0000000000000013cdc708723f111c6d34effd9ea663e62eaaa5c9ff299800cb and reconsiderblock 0000000000000013cdc708723f111c6d34effd9ea663e62eaaa5c9ff299800cb don't help either.

@qwizzie
Copy link

qwizzie commented May 22, 2023

Upgrading to 19.1.0 doesn't help, the node is stuck at height=1874880 invalidateblock 0000000000000013cdc708723f111c6d34effd9ea663e62eaaa5c9ff299800cb and reconsiderblock 0000000000000013cdc708723f111c6d34effd9ea663e62eaaa5c9ff299800cb don't help either.

Yeah, i ended up giving my masternodes a reindex command after updating to v19.1
Spork 23 (Quorum PoSe) is disabled, so no worries about getting PoSe scored during reindex with regards to masternodes.

@PastaPastaPasta
Copy link
Member

current VALID height and hash I see is / are
{
"height": 1874884,
"hash": "000000000000005c3c3370b96c2a4f58e7062a050be45479dd58cd3a91121950",
"difficulty": 32942678.88782359,
"chainwork": "0000000000000000000000000000000000000000000084f03183f1747f340482",
"branchlen": 0,
"forkpoint": "000000000000005c3c3370b96c2a4f58e7062a050be45479dd58cd3a91121950",
"status": "active"
},
{
"height": 1874880,
"hash": "000000000000001e28817f2896e1dcf993dfab73f4b80a64e9e3be5680d2b3d7",
"difficulty": 117437257.0405482,
"chainwork": "0000000000000000000000000000000000000000000084f02826d7a9be295cf4",
"branchlen": 1,
"forkpoint": "0000000000000013cdc708723f111c6d34effd9ea663e62eaaa5c9ff299800cb",
"status": "valid-fork"
}

@qwizzie
Copy link

qwizzie commented May 22, 2023

I am seeing a lot (and i mean a lot) of 'CActiveMasternodeManager::Init -- ERROR: Local address does not match the address from ProTx' messages in my debug.log after i updated to v19.1 and issued a reindex. I checked a few masternodes of mine and those i checked all had that error message in their debug.log

Update : decided to just remove the blockchain and most of the .dat files and download and sync from scratch.

@Har01d
Copy link

Har01d commented May 22, 2023

Upgrading to 19.1.0 doesn't help, the node is stuck at height=1874880 invalidateblock 0000000000000013cdc708723f111c6d34effd9ea663e62eaaa5c9ff299800cb and reconsiderblock 0000000000000013cdc708723f111c6d34effd9ea663e62eaaa5c9ff299800cb don't help either.

It suddenly started to sync and it seems to be working fine now.

@maayank
Copy link

maayank commented May 22, 2023

With 19.1. Still froze for us. Tried to invalidate and reconsider (even using earlier blocks) and didn't help. Getblockcount, getpeerinfo and getblockchaininfo after the break. Would appreciate all help

root@dash:/# dash-cli -datadir=/data/dash getblockcount
1874880
root@dash:/# dash-cli -datadir=/data/dash getpeerinfo  
[
  {
    "id": 1,
    "addr": "82.211.25.172:9999",
    "addrlocal": "3.21.200.4:39512",
    "addrbind": "172.18.0.3:39512",
    "services": "0000000000000c05",
    "servicesnames": [
      "NETWORK",
      "BLOOM",
      "NETWORK_LIMITED",
      "HEADERS_COMPRESSED"
    ],
    "relaytxes": true,
    "lastsend": 1684799294,
    "lastrecv": 1684799339,
    "bytessent": 1488,
    "bytesrecv": 18787,
    "conntime": 1684799287,
    "timeoffset": 0,
    "pingtime": 0.229181,
    "minping": 0.229181,
    "version": 70227,
    "subver": "/Dash Core:19.0.0/",
    "inbound": false,
    "addnode": false,
    "masternode": false,
    "startingheight": 1874879,
    "banscore": 0,
    "synced_headers": -1,
    "synced_blocks": -1,
    "inflight": [
    ],
    "whitelisted": false,
    "permissions": [
    ],
    "bytessent_per_msg": {
      "getaddr": 24,
      "getheaders2": 1085,
      "getsporks": 24,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "senddsq": 25,
      "sendheaders2": 24,
      "verack": 24,
      "version": 161
    },
    "bytesrecv_per_msg": {
      "addrv2": 15645,
      "dsq": 166,
      "headers2": 25,
      "inv": 244,
      "mnauth": 152,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "senddsq": 25,
      "sendheaders2": 24,
      "spork": 2200,
      "verack": 24,
      "version": 161
    }
  },
  {
    "id": 2,
    "addr": "51.91.75.25:9999",
    "addrlocal": "3.21.200.4:47566",
    "addrbind": "172.18.0.3:47566",
    "services": "0000000000000c05",
    "servicesnames": [
      "NETWORK",
      "BLOOM",
      "NETWORK_LIMITED",
      "HEADERS_COMPRESSED"
    ],
    "relaytxes": true,
    "lastsend": 1684799294,
    "lastrecv": 1684799357,
    "bytessent": 403,
    "bytesrecv": 20077,
    "conntime": 1684799289,
    "timeoffset": 0,
    "pingtime": 0.121307,
    "minping": 0.121307,
    "version": 70227,
    "subver": "/Dash Core:19.0.0/",
    "inbound": false,
    "addnode": false,
    "masternode": false,
    "startingheight": 1874879,
    "banscore": 0,
    "synced_headers": -1,
    "synced_blocks": -1,
    "inflight": [
    ],
    "whitelisted": false,
    "permissions": [
    ],
    "bytessent_per_msg": {
      "getaddr": 24,
      "getsporks": 24,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "senddsq": 25,
      "sendheaders2": 24,
      "verack": 24,
      "version": 161
    },
    "bytesrecv_per_msg": {
      "addrv2": 17328,
      "inv": 194,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "senddsq": 25,
      "sendheaders2": 24,
      "spork": 2200,
      "verack": 24,
      "version": 161
    }
  },
  {
    "id": 3,
    "addr": "68.183.188.215:9999",
    "addrlocal": "3.21.200.4:38284",
    "addrbind": "172.18.0.3:38284",
    "services": "0000000000000c05",
    "servicesnames": [
      "NETWORK",
      "BLOOM",
      "NETWORK_LIMITED",
      "HEADERS_COMPRESSED"
    ],
    "relaytxes": true,
    "lastsend": 1684799306,
    "lastrecv": 1684799346,
    "bytessent": 403,
    "bytesrecv": 20528,
    "conntime": 1684799301,
    "timeoffset": -1,
    "pingtime": 0.47413,
    "minping": 0.47413,
    "version": 70227,
    "subver": "/Dash Core:19.1.0/",
    "inbound": false,
    "addnode": false,
    "masternode": false,
    "startingheight": 1874990,
    "banscore": 0,
    "synced_headers": -1,
    "synced_blocks": -1,
    "inflight": [
    ],
    "whitelisted": false,
    "permissions": [
    ],
    "bytessent_per_msg": {
      "getaddr": 24,
      "getsporks": 24,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "senddsq": 25,
      "sendheaders2": 24,
      "verack": 24,
      "version": 161
    },
    "bytesrecv_per_msg": {
      "addrv2": 16387,
      "dsq": 166,
      "getheaders2": 1085,
      "inv": 183,
      "mnauth": 152,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "senddsq": 25,
      "sendheaders2": 24,
      "spork": 2200,
      "verack": 24,
      "version": 161
    }
  },
  {
    "id": 4,
    "addr": "139.162.105.155:9999",
    "addrlocal": "3.21.200.4:52856",
    "addrbind": "172.18.0.3:52856",
    "services": "0000000000000c05",
    "servicesnames": [
      "NETWORK",
      "BLOOM",
      "NETWORK_LIMITED",
      "HEADERS_COMPRESSED"
    ],
    "relaytxes": true,
    "lastsend": 1684799306,
    "lastrecv": 1684799352,
    "bytessent": 403,
    "bytesrecv": 19590,
    "conntime": 1684799302,
    "timeoffset": 0,
    "pingtime": 0.300366,
    "minping": 0.300366,
    "version": 70223,
    "subver": "/Dash Core:18.0.1/",
    "inbound": false,
    "addnode": false,
    "masternode": false,
    "startingheight": 1874990,
    "banscore": 0,
    "synced_headers": -1,
    "synced_blocks": -1,
    "inflight": [
    ],
    "whitelisted": false,
    "permissions": [
    ],
    "bytessent_per_msg": {
      "getaddr": 24,
      "getsporks": 24,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "senddsq": 25,
      "sendheaders2": 24,
      "verack": 24,
      "version": 161
    },
    "bytesrecv_per_msg": {
      "addrv2": 15597,
      "dsq": 170,
      "getheaders2": 1085,
      "inv": 183,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "senddsq": 25,
      "sendheaders2": 24,
      "spork": 2200,
      "verack": 24,
      "version": 161
    }
  },
  {
    "id": 5,
    "addr": "95.183.51.141:9999",
    "addrlocal": "3.21.200.4:48508",
    "addrbind": "172.18.0.3:48508",
    "services": "0000000000000c05",
    "servicesnames": [
      "NETWORK",
      "BLOOM",
      "NETWORK_LIMITED",
      "HEADERS_COMPRESSED"
    ],
    "relaytxes": true,
    "lastsend": 1684799353,
    "lastrecv": 1684799346,
    "bytessent": 443,
    "bytesrecv": 18533,
    "conntime": 1684799303,
    "timeoffset": -1,
    "pingtime": 0.124047,
    "minping": 0.124047,
    "version": 70227,
    "subver": "/Dash Core:19.0.0/",
    "inbound": false,
    "addnode": false,
    "masternode": false,
    "startingheight": 1874879,
    "banscore": 0,
    "synced_headers": -1,
    "synced_blocks": -1,
    "inflight": [
    ],
    "whitelisted": false,
    "permissions": [
    ],
    "bytessent_per_msg": {
      "addrv2": 40,
      "getaddr": 24,
      "getsporks": 24,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "senddsq": 25,
      "sendheaders2": 24,
      "verack": 24,
      "version": 161
    },
    "bytesrecv_per_msg": {
      "addrv2": 15538,
      "dsq": 166,
      "inv": 122,
      "mnauth": 152,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "senddsq": 25,
      "sendheaders2": 24,
      "spork": 2200,
      "verack": 24,
      "version": 161
    }
  },
  {
    "id": 6,
    "addr": "195.181.211.64:9999",
    "addrbind": "172.18.0.3:34134",
    "services": "0000000000000c05",
    "servicesnames": [
      "NETWORK",
      "BLOOM",
      "NETWORK_LIMITED",
      "HEADERS_COMPRESSED"
    ],
    "relaytxes": true,
    "lastsend": 1684799306,
    "lastrecv": 1684799339,
    "bytessent": 403,
    "bytesrecv": 18367,
    "conntime": 1684799303,
    "timeoffset": 0,
    "pingtime": 0.205568,
    "minping": 0.205568,
    "version": 70227,
    "subver": "/Dash Core:19.0.0/",
    "inbound": false,
    "addnode": false,
    "masternode": false,
    "startingheight": 1874879,
    "banscore": 0,
    "synced_headers": -1,
    "synced_blocks": -1,
    "inflight": [
    ],
    "whitelisted": false,
    "permissions": [
    ],
    "bytessent_per_msg": {
      "getaddr": 24,
      "getsporks": 24,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "senddsq": 25,
      "sendheaders2": 24,
      "verack": 24,
      "version": 161
    },
    "bytesrecv_per_msg": {
      "addrv2": 15538,
      "inv": 122,
      "mnauth": 152,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "senddsq": 25,
      "sendheaders2": 24,
      "spork": 2200,
      "verack": 24,
      "version": 161
    }
  },
  {
    "id": 8,
    "addr": "150.136.103.179:9999",
    "addrlocal": "3.21.200.4:52084",
    "addrbind": "172.18.0.3:52084",
    "services": "0000000000000c05",
    "servicesnames": [
      "NETWORK",
      "BLOOM",
      "NETWORK_LIMITED",
      "HEADERS_COMPRESSED"
    ],
    "relaytxes": true,
    "lastsend": 1684799343,
    "lastrecv": 1684799339,
    "bytessent": 443,
    "bytesrecv": 17876,
    "conntime": 1684799305,
    "timeoffset": 0,
    "pingtime": 0.023654,
    "minping": 0.023654,
    "version": 70227,
    "subver": "/Dash Core:19.1.0/",
    "inbound": false,
    "addnode": false,
    "masternode": false,
    "startingheight": 1875067,
    "banscore": 0,
    "synced_headers": -1,
    "synced_blocks": -1,
    "inflight": [
    ],
    "whitelisted": false,
    "permissions": [
    ],
    "bytessent_per_msg": {
      "addrv2": 40,
      "getaddr": 24,
      "getsporks": 24,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "senddsq": 25,
      "sendheaders2": 24,
      "verack": 24,
      "version": 161
    },
    "bytesrecv_per_msg": {
      "addrv2": 15199,
      "inv": 122,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "senddsq": 25,
      "sendheaders2": 24,
      "spork": 2200,
      "verack": 24,
      "version": 161
    }
  },
  {
    "id": 10,
    "addr": "88.99.11.5:9999",
    "addrlocal": "3.21.200.4:38156",
    "addrbind": "172.18.0.3:38156",
    "services": "0000000000000c05",
    "servicesnames": [
      "NETWORK",
      "BLOOM",
      "NETWORK_LIMITED",
      "HEADERS_COMPRESSED"
    ],
    "relaytxes": true,
    "lastsend": 1684799312,
    "lastrecv": 1684799346,
    "bytessent": 403,
    "bytesrecv": 21154,
    "conntime": 1684799307,
    "timeoffset": 0,
    "pingtime": 0.200404,
    "minping": 0.200404,
    "version": 70227,
    "subver": "/Dash Core:19.1.0/",
    "inbound": false,
    "addnode": false,
    "masternode": false,
    "startingheight": 1874990,
    "banscore": 0,
    "synced_headers": -1,
    "synced_blocks": -1,
    "inflight": [
    ],
    "whitelisted": false,
    "permissions": [
    ],
    "bytessent_per_msg": {
      "getaddr": 24,
      "getsporks": 24,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "senddsq": 25,
      "sendheaders2": 24,
      "verack": 24,
      "version": 161
    },
    "bytesrecv_per_msg": {
      "addrv2": 17013,
      "dsq": 166,
      "getheaders2": 1085,
      "inv": 183,
      "mnauth": 152,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "senddsq": 25,
      "sendheaders2": 24,
      "spork": 2200,
      "verack": 24,
      "version": 161
    }
  },
  {
    "id": 11,
    "addr": "135.181.52.145:9999",
    "addrlocal": "3.21.200.4:38928",
    "addrbind": "172.18.0.3:38928",
    "services": "0000000000000c05",
    "servicesnames": [
      "NETWORK",
      "BLOOM",
      "NETWORK_LIMITED",
      "HEADERS_COMPRESSED"
    ],
    "relaytxes": false,
    "lastsend": 1684799312,
    "lastrecv": 1684799334,
    "bytessent": 354,
    "bytesrecv": 3933,
    "conntime": 1684799308,
    "timeoffset": 0,
    "pingtime": 0.237356,
    "minping": 0.237356,
    "version": 70227,
    "subver": "/Dash Core:19.1.0/",
    "inbound": false,
    "addnode": false,
    "masternode": false,
    "startingheight": 1874990,
    "banscore": 0,
    "synced_headers": -1,
    "synced_blocks": -1,
    "inflight": [
    ],
    "whitelisted": false,
    "permissions": [
    ],
    "bytessent_per_msg": {
      "getsporks": 24,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "sendheaders2": 24,
      "verack": 24,
      "version": 161
    },
    "bytesrecv_per_msg": {
      "addrv2": 80,
      "getheaders2": 1085,
      "inv": 61,
      "mnauth": 152,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "senddsq": 25,
      "sendheaders2": 24,
      "spork": 2200,
      "verack": 24,
      "version": 161
    }
  },
  {
    "id": 13,
    "addr": "188.40.163.23:9999",
    "addrlocal": "3.21.200.4:42154",
    "addrbind": "172.18.0.3:42154",
    "services": "0000000000000c05",
    "servicesnames": [
      "NETWORK",
      "BLOOM",
      "NETWORK_LIMITED",
      "HEADERS_COMPRESSED"
    ],
    "relaytxes": false,
    "lastsend": 1684799318,
    "lastrecv": 1684799334,
    "bytessent": 354,
    "bytesrecv": 3893,
    "conntime": 1684799313,
    "timeoffset": 0,
    "pingtime": 0.200535,
    "minping": 0.200535,
    "version": 70227,
    "subver": "/Dash Core:19.1.0/",
    "inbound": false,
    "addnode": false,
    "masternode": false,
    "startingheight": 1874990,
    "banscore": 0,
    "synced_headers": -1,
    "synced_blocks": -1,
    "inflight": [
    ],
    "whitelisted": false,
    "permissions": [
    ],
    "bytessent_per_msg": {
      "getsporks": 24,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "sendheaders2": 24,
      "verack": 24,
      "version": 161
    },
    "bytesrecv_per_msg": {
      "addrv2": 40,
      "getheaders2": 1085,
      "inv": 61,
      "mnauth": 152,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "senddsq": 25,
      "sendheaders2": 24,
      "spork": 2200,
      "verack": 24,
      "version": 161
    }
  }
]
root@dash:/# dash-cli -datadir=/data/dash getblockchaininfo
{
  "chain": "main",
  "blocks": 1874880,
  "headers": 1874880,
  "bestblockhash": "000000000000001e28817f2896e1dcf993dfab73f4b80a64e9e3be5680d2b3d7",
  "difficulty": 117437257.0405482,
  "mediantime": 1684727247,
  "verificationprogress": 0.9997502737722491,
  "initialblockdownload": true,
  "chainwork": "0000000000000000000000000000000000000000000084f02826d7a9be295cf4",
  "size_on_disk": 34076801844,
  "pruned": false,
  "softforks": [
    {
      "id": "bip34",
      "version": 2,
      "reject": {
        "status": true
      }
    },
    {
      "id": "bip66",
      "version": 3,
      "reject": {
        "status": true
      }
    },
    {
      "id": "bip65",
      "version": 4,
      "reject": {
        "status": true
      }
    }
  ],
  "bip9_softforks": {
    "csv": {
      "status": "active",
      "start_time": 1486252800,
      "timeout": 1517788800,
      "since": 622944
    },
    "dip0001": {
      "status": "active",
      "start_time": 1508025600,
      "timeout": 1539561600,
      "since": 782208
    },
    "bip147": {
      "status": "active",
      "start_time": 1524477600,
      "timeout": 1556013600,
      "since": 939456
    },
    "dip0003": {
      "status": "active",
      "start_time": 1546300800,
      "timeout": 1577836800,
      "since": 1028160
    },
    "dip0008": {
      "status": "active",
      "start_time": 1557878400,
      "timeout": 1589500800,
      "since": 1088640
    },
    "realloc": {
      "status": "active",
      "start_time": 1601510400,
      "timeout": 1633046400,
      "since": 1374912
    },
    "dip0020": {
      "status": "active",
      "start_time": 1625097600,
      "timeout": 1656633600,
      "since": 1516032
    },
    "dip0024": {
      "status": "active",
      "start_time": 1660521600,
      "timeout": 1692057600,
      "since": 1737792
    },
    "v19": {
      "status": "defined",
      "start_time": 1686700800,
      "timeout": 1718323200,
      "since": 0
    }
  },
  "warnings": ""
}

@UdjinM6
Copy link

UdjinM6 commented May 23, 2023

@maayank pls try disconnecting a couple of nodes so that the first node in getpeerinfo list is Dash Core 19.1.0 (e.g. do disconnectnode "" 1 and disconnectnode "" 2) and see if helps.

@PastaPastaPasta
Copy link
Member

If you are stuck on block 1874880, you can try to perform the following

grab this text here: https://gist.github.com/PastaPastaPasta/ace08090e6034b3cd1b7f1379776be7f

and run submitblock

@joslee7410
Copy link

Anyone have other solution to share?

I'm still stucking at
Block 1874879
Hash 0000000000000013cdc708723f111c6d34effd9ea663e62eaaa5c9ff299800cb

Please help, thank you.

@starwill
Copy link

Upgrading to 19.1.0 doesn't help, the node is stuck at height=1874880 invalidateblock 0000000000000013cdc708723f111c6d34effd9ea663e62eaaa5c9ff299800cb and reconsiderblock 0000000000000013cdc708723f111c6d34effd9ea663e62eaaa5c9ff299800cb don't help either.

It suddenly started to sync and it seems to be working fine now.

how long have you waited until it work normally ? I have upgrade to v19.1.0 for 2 hours but it's still stucking at 1874879. have you take any other action to recover it?

@gades
Copy link
Author

gades commented May 23, 2023

how long have you waited until it work normally ? I have upgrade to v19.1.0 for 2 hours but it's still stucking at 1874879. have you take any other action to recover it?

I did following steps:

  1. stop node
  2. update it to v19.1.0
  3. remove all files in ~/.dashcore/ except dash.conf / wallet.dat
  4. start node

other steps like invalidateblock / reconsiderblock / submitblock / disconnectnode were unsuccessful

@gkardava
Copy link

dash v19.1.0 crashed with following error.

2023-05-23T08:12:45Z [default wallet] Submitting wtx ****************************** to mempool for relay
2023-05-23T08:12:46Z [ProcessBlock] h[1874880] numCommitmentsRequired[0] numCommitmentsInNewBlock[0]
2023-05-23T08:12:47Z CDeterministicMNManager::ProcessBlock -- Wrote snapshot. nHeight=1874880, mapCurMNs.allMNsCount=4400
2023-05-23T08:12:47Z ERROR: ConnectBlock(DASH): ProcessSpecialTxsInBlock for block 000000000000001e28817f2896e1dcf993dfab73f4b80a64e9e3be5680d2b3d7 failed with bad-cbtx-mnmerkleroot (code 16)
2023-05-23T08:12:47Z InvalidChainFound: invalid block=000000000000001e28817f2896e1dcf993dfab73f4b80a64e9e3be5680d2b3d7  height=1874880  log2_work=79.05461097  date=2023-05-22T04:13:10Z
2023-05-23T08:12:47Z InvalidChainFound:  current best=0000000000000013cdc708723f111c6d34effd9ea663e62eaaa5c9ff299800cb  height=1874879  log2_work=79.05460981  date=2023-05-22T04:02:30Z
2023-05-23T08:12:47Z ERROR: ConnectTip: ConnectBlock 000000000000001e28817f2896e1dcf993dfab73f4b80a64e9e3be5680d2b3d7 failed, bad-cbtx-mnmerkleroot (code 16)
2023-05-23T08:12:47Z InvalidChainFound: invalid block=000000000000001e28817f2896e1dcf993dfab73f4b80a64e9e3be5680d2b3d7  height=1874880  log2_work=79.05461097  date=2023-05-22T04:13:10Z
2023-05-23T08:12:47Z InvalidChainFound:  current best=0000000000000013cdc708723f111c6d34effd9ea663e62eaaa5c9ff299800cb  height=1874879  log2_work=79.05460981  date=2023-05-22T04:02:30Z
2023-05-23T08:12:47Z Filling coin cache with masternode UTXOs...
2023-05-23T08:12:47Z Filling coin cache with masternode UTXOs: done in 267ms
2023-05-23T08:12:47Z [default wallet] AddToWallet ******************************
2023-05-23T08:12:49Z [default wallet] AddToWallet ******************************
2023-05-23T08:12:49Z Imported mempool transactions from disk: 342 succeeded, 1 failed, 0 expired, 1 already there, 0 waiting for initial broadcast
2023-05-23T08:12:49Z loadblk thread exit
2023-05-23T08:12:52Z New outbound peer connected: version: 70227, blocks=1875183, peer=0 (full-relay)
2023-05-23T08:12:52Z Synchronizing blockheaders, height: 1875183 (~100.00%)
2023-05-23T08:12:53Z [ProcessBlock] h[1874880] numCommitmentsRequired[0] numCommitmentsInNewBlock[0]
2023-05-23T08:12:53Z CDeterministicMNManager::ProcessBlock -- Wrote snapshot. nHeight=1874880, mapCurMNs.allMNsCount=4400
2023-05-23T08:12:53Z Posix Signal: Segmentation fault
No debug information available for stacktrace. You should add debug information and then run:

@gades
Copy link
Author

gades commented May 23, 2023

dash v19.1.0 crashed with following error.

can you try to remove all files in ~/.dashcore/ except dash.conf / wallet.dat and start again?

@starwill
Copy link

how long have you waited until it work normally ? I have upgrade to v19.1.0 for 2 hours but it's still stucking at 1874879. have you take any other action to recover it?

I did following steps:

  1. stop node
  2. update it to v19.1.0
  3. remove all files in ~/.dashcore/ except dash.conf / wallet.dat
  4. start node

other steps like invalidateblock / reconsiderblock / submitblock / disconnectnode were unsuccessful

did you mean that you clear the chaindata and resync the chain?

@gades
Copy link
Author

gades commented May 23, 2023

did you mean that you clear the chaindata and resync the chain?

yes, it's only one way which solve this issue for me

@maayank
Copy link

maayank commented May 23, 2023

Just wanted to say that by now our node resumed syncing

@Har01d
Copy link

Har01d commented May 23, 2023

Upgrading to 19.1.0 doesn't help, the node is stuck at height=1874880 invalidateblock 0000000000000013cdc708723f111c6d34effd9ea663e62eaaa5c9ff299800cb and reconsiderblock 0000000000000013cdc708723f111c6d34effd9ea663e62eaaa5c9ff299800cb don't help either.

It suddenly started to sync and it seems to be working fine now.

how long have you waited until it work normally ? I have upgrade to v19.1.0 for 2 hours but it's still stucking at 1874879. have you take any other action to recover it?

Several hours. We also tried to delete banlist.dat and peers.dat but it didn't seem to help.

@gades
Copy link
Author

gades commented May 23, 2023

Several hours. We also tried to delete banlist.dat and peers.dat but it didn't seem to help.

Resyncing node from scratch was approx 2-3 hours and our nodes are up. But strange , before we start do it the count blocks were higher than in the main net

@PastaPastaPasta will there new update because I didn't saw any fixes for v19 and only one change is reschedule start voting to 15 June ?

@Crypto2
Copy link

Crypto2 commented May 23, 2023

You should only have to launch with -reindex, not do a full resync.

@PastaPastaPasta
Copy link
Member

Yes, there will be an 19.2 once we have proper fixes for the issues we saw.

@impishian
Copy link

It's been at least 5 days, still not fixed❓

Is the dash chain dead❓

No one uses this chain anymore❓ 😴

@PastaPastaPasta
Copy link
Member

... No, the actual issues are being fixed in PRs such as #5397, #5395,
#5392 & #5386

However, the network is, and has been since v19.1 was released a week ago, completely operational.

Let's remember to keep the GitHub professional and respectful.

@impishian
Copy link

impishian commented May 30, 2023

$ dash-cli --version
Dash Core RPC client version v19.1.0

$ dash-cli getblockchaininfo|head
{
"chain": "main",
"blocks": 1874879,
"headers": 1874879,
"bestblockhash": "0000000000000013cdc708723f111c6d34effd9ea663e62eaaa5c9ff299800cb",
"difficulty": 119636794.055944,
"mediantime": 1684727208,
"verificationprogress": 0.9975710346798932,
"initialblockdownload": true,
"chainwork": "0000000000000000000000000000000000000000000084f02126dd60b97ef9c4",

remove all files in ~/.dashcore/ except dash.conf / wallet.dat

resync, three times, same result.

@Crypto2
Copy link

Crypto2 commented May 30, 2023

Make sure you reindex (-reindex or reindex=1 in your config file), then it should resume fine. You make have to do a reconsiderblock on the correct block too.

@impishian
Copy link

before "remove all files ... and resync", already tried reindex, invalidateblock/reconsiderblock, but it doesn't work.

@qwizzie
Copy link

qwizzie commented May 30, 2023

In two days a new Dash bootstrap file will be created (June 1 00:00:01 UTC 2023)
Maybe that specific June bootstrap file can help people, who are currently still struggling.

See : https://github.com/UdjinM6/dash-bootstrap

It does mean waiting for two more days.

With me the removal of all files in ~/.dashcore/ (except dash.conf / .lock / settings.json / sentinel folder) did the trick for my masternodes.
If people are not running masternodes, the removal of all files in ~/.dashcore/ (except dash.conf / wallet.dat / .lock / settings.json) should do the trick.

@petebus
Copy link

petebus commented Jun 3, 2023

I tried run invalidateblock and reconsiderblock and then i -reindex my wallet and syncrhonize was succesfully, but i get this warning:

Warning: unknown new rules activated (versionbit 8)

What does it mean? Is there cause of concern?

Dash Core 19.1.0

@knst
Copy link
Collaborator

knst commented Jun 3, 2023

Warning: unknown new rules activated (versionbit 8)

That's an artefact of postponing hard fork v19 (bit 8).
This warning would disapper in a future 19.2 release.

@petebus
Copy link

petebus commented Jun 19, 2023

That's an artefact of postponing hard fork v19 (bit 8). This warning would disapper in a future 19.2 release.

Just in case, I've updated Dash Core to 19.2 and i still see this warning.

@chostorg
Copy link

Can you guys help BTCPAYSERVER bump their version to latest? I'm stuck with this bug. According to btcpayserver/btcpayserver-docker#803 you should help them somehow.

@thephez
Copy link
Collaborator

thephez commented Jun 21, 2023

Can you guys help BTCPAYSERVER bump their version to latest? I'm stuck with this bug. According to btcpayserver/btcpayserver-docker#803 you should help them somehow.

There is an open PR there to do this. See btcpayserver/btcpayserver-docker#800

@impishian
Copy link

impishian commented Jul 6, 2023

Block sync stopped at 1899071.

reconsiderblock 0000000000000015e32e73052d663626327004c81c5c22cb8b42c361015c0eae
remove peers.dat

both, not helped!


$ dash-cli --version
Dash Core RPC client version v19.2.0

$ dash-cli getblockchaininfo|head
{
"chain": "main",
"blocks": 1899071,
"headers": 1899071,
"bestblockhash": "0000000000000022cd5c8d422c2e9b3a876dee260402e8850d95ca3bf64cb9b6",
"difficulty": 83302114.74555434,
"mediantime": 1688597376,
"verificationprogress": 0.9999679197411995,
"initialblockdownload": false,
"chainwork": "00000000000000000000000000000000000000000000877f1fb6fb18c7e7fb79",

$ dash-cli getblockhash 1899071
0000000000000022cd5c8d422c2e9b3a876dee260402e8850d95ca3bf64cb9b6

1899071 hash: https://insight.dash.org/insight/block/0000000000000022cd5c8d422c2e9b3a876dee260402e8850d95ca3bf64cb9b6

1899072 hash: https://insight.dash.org/insight/block/0000000000000015e32e73052d663626327004c81c5c22cb8b42c361015c0eae

$ grep "is marked invalid" debug.log
...
...
...
0000000000000015e32e73052d663626327004c81c5c22cb8b42c361015c0eae is marked invalid

@Scumwong
Copy link

Scumwong commented Jul 6, 2023

Block sync stopped at 1899071.

reconsiderblock 0000000000000015e32e73052d663626327004c81c5c22cb8b42c361015c0eae remove peers.dat

both, not helped!

$ dash-cli --version Dash Core RPC client version v19.2.0

$ dash-cli getblockchaininfo|head { "chain": "main", "blocks": 1899071, "headers": 1899071, "bestblockhash": "0000000000000022cd5c8d422c2e9b3a876dee260402e8850d95ca3bf64cb9b6", "difficulty": 83302114.74555434, "mediantime": 1688597376, "verificationprogress": 0.9999679197411995, "initialblockdownload": false, "chainwork": "00000000000000000000000000000000000000000000877f1fb6fb18c7e7fb79",

$ dash-cli getblockhash 1899071 0000000000000022cd5c8d422c2e9b3a876dee260402e8850d95ca3bf64cb9b6

1899071 hash: https://insight.dash.org/insight/block/0000000000000022cd5c8d422c2e9b3a876dee260402e8850d95ca3bf64cb9b6

1899072 hash: https://insight.dash.org/insight/block/0000000000000015e32e73052d663626327004c81c5c22cb8b42c361015c0eae

$ grep "is marked invalid" debug.log ... ... ... 0000000000000015e32e73052d663626327004c81c5c22cb8b42c361015c0eae is marked invalid

the same problem here .
have you resloved it?

@riongull
Copy link

riongull commented Jul 6, 2023

You need to update to v19.2 if you haven't already and then reindex your chain:

  1. Add reindex=1 to dash.conf
  2. Restart dashd and let it reindex and sync (can take several hours)
  3. Stop dashd and remove or comment out reindex=1
  4. Restart dashd

@Scumwong
Copy link

Scumwong commented Jul 6, 2023

You need to reindex your chain.

  1. Add reindex=1 to dash.conf
  2. Restart dashd and let it reindex and sync (can take several hours)
  3. Remove or comment out reindex=1
  4. Restart dashd

It works,thank you very much

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests