Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Invalid bloom error thus shutting down chain bootstrap #1114

Closed
thesoulkiller opened this issue Mar 4, 2024 · 4 comments
Closed

Invalid bloom error thus shutting down chain bootstrap #1114

thesoulkiller opened this issue Mar 4, 2024 · 4 comments
Labels
bug Something isn't working

Comments

@thesoulkiller
Copy link

thesoulkiller commented Mar 4, 2024 •
edited

Short story: We recently missed the the major Warp messaging (Fuji) update, the subnet stopped 3 days, after we synchronized all of the archieve nodes and rpc's with copying the db. However now a new rpc nodes which tries to enter the subnet receives Fatal error and shuts down the bootstrapping process.

Describe the bug
Somehow subnet produces a dirty block and inserts through all of the validators, then newly entered node fails to bootstrap the subnet chain.
I am sharing the both of the archive validator node and a recent node ( which is just indexing node ) logs. After creating a certain block ( 2PGQzuTjzBEcVqTnNmVwhAsk4PsXV88M7DT7wRtgmC973J4TVf ) the validator kills the subnet subprocess.

The last line of the recently joined rpc node gives the following error and shuts down

FATAL <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> handler/handler.go:337 shutting down chain {"reason": "received an unexpected error", "error": "failed to execute job 2PGQzuTjzBEcVqTnNmVwhAsk4PsXV88M7DT7wRtgmC973J4TVf due to failed to verify block in bootstrapping: rpc error: code = Unknown desc = invalid bloom

Logs

1- The subnet archive validator node logs (truncated till that faulty blockID):

[02-15|07:17:19.701] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> proposervm/block.go:289 built block {"blkID": "2PGQzuTjzBEcVqTnNmVwhAsk4PsXV88M7DT7wRtgmC973J4TVf", "innerBlkID": "NiAyL5yt9Uv13jjCoY1RpqTsTY3NHiRPMrQyQ4yVkuUkm6iyr", "height": 4362211, "parentTimestamp": "[02-14|06:14:03.000]", "blockTimestamp": "[02-15|07:17:19.000]"}
INFO [02-15|07:30:53.008] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/miner/worker.go:394: Commit new mining work number=4,362,218 hash=4a2781..681a21 uncles=0 txs=0 gas=0         fees=0            elapsed=1.677ms
[02-19|08:38:52.712] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> subprocess/runtime.go:143 plugin handshake succeeded {"addr": "127.0.0.1:35733"}
[02-19|08:38:52.717] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> rpcchainvm/vm_client.go:165 grpc: serving database {"address": "127.0.0.1:46439"}
[02-19|08:38:52.718] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> rpcchainvm/vm_client.go:184 grpc: serving vm services {"address": "127.0.0.1:32801"}
INFO [02-19|08:38:52.723] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:291: Initializing Subnet EVM VM Version=v0.6.0-fuji Config="{AirdropFile: SnowmanAPIEnabled:false AdminAPIEnabled:false AdminAPIDir: WarpAPIEnabled:false EnabledEthAPIs:[eth eth-filter net web3 internal-eth internal-blockchain internal-transaction] ContinuousProfilerDir: ContinuousProfilerFrequency:15m0s ContinuousProfilerMaxFiles:5 RPCGasCap:50000000 RPCTxFeeCap:100 TrieCleanCache:512 TrieCleanJournal: TrieCleanRejournal:0s TrieDirtyCache:512 TrieDirtyCommitTarget:20 SnapshotCache:256 Preimages:false SnapshotWait:false SnapshotVerify:false Pruning:false AcceptorQueueLimit:64 CommitInterval:4096 AllowMissingTries:false PopulateMissingTries:<nil> PopulateMissingTriesParallelism:1024 PruneWarpDB:false MetricsExpensiveEnabled:true LocalTxsEnabled:false TxPoolJournal:transactions.rlp TxPoolRejournal:1h0m0s TxPoolPriceLimit:1 TxPoolPriceBump:10 TxPoolAccountSlots:16 TxPoolGlobalSlots:5120 TxPoolAccountQueue:64 TxPoolGlobalQueue:1024 APIMaxDuration:0s WSCPURefillRate:0s WSCPUMaxStored:0s MaxBlocksPerRequest:0 AllowUnfinalizedQueries:false AllowUnprotectedTxs:false AllowUnprotectedTxHashes:[0xfefb2da535e927b85fe68eb81cb2e4a5827c905f78381a01ef2322aa9b0aee8e] KeystoreDirectory: KeystoreExternalSigner: KeystoreInsecureUnlockAllowed:false RemoteGossipOnlyEnabled:false RegossipFrequency:1m0s RegossipMaxTxs:16 RegossipTxsPerAddress:1 PriorityRegossipFrequency:1s PriorityRegossipMaxTxs:32 PriorityRegossipTxsPerAddress:16 PriorityRegossipAddresses:[] LogLevel:info LogJSONFormat:false FeeRecipient: OfflinePruning:false OfflinePruningBloomFilterSize:512 OfflinePruningDataDirectory: MaxOutboundActiveRequests:16 MaxOutboundActiveCrossChainRequests:64 StateSyncEnabled:false StateSyncSkipResume:false StateSyncServerTrieCache:64 StateSyncIDs: StateSyncCommitInterval:16384 StateSyncMinBlocks:300000 StateSyncRequestSize:1024 InspectDatabase:false SkipUpgradeCheck:false AcceptedCacheSize:32 TxLookupLimit:0 WarpOffChainMessages:[]}"
INFO [02-19|08:38:52.724] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:443: Config has not specified any coinbase address. Defaulting to the blackhole address. 
INFO [02-19|08:38:52.724] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/state_processor.go:200: Activating new precompile name=contractDeployerAllowListConfig config="&{AllowListConfig:{AdminAddresses:[0x0Ed6431f48560E943cC8c1EdEAe3D7f7eDDE46A7] ManagerAddresses:[] EnabledAddresses:[]} Upgrade:{BlockTimestamp:0xc0008fd700 Disable:false}}"
INFO [02-19|08:38:52.724] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/state_processor.go:200: Activating new precompile name=contractNativeMinterConfig      config="&{AllowListConfig:{AdminAddresses:[0x0Ed6431f48560E943cC8c1EdEAe3D7f7eDDE46A7] ManagerAddresses:[] EnabledAddresses:[]} Upgrade:{BlockTimestamp:0xc0008fd8b8 Disable:false} InitialMint:map[]}"
INFO [02-19|08:38:52.724] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/state_processor.go:200: Activating new precompile name=feeManagerConfig                config="&{AllowListConfig:{AdminAddresses:[0x0Ed6431f48560E943cC8c1EdEAe3D7f7eDDE46A7] ManagerAddresses:[] EnabledAddresses:[]} Upgrade:{BlockTimestamp:0xc0008fda70 Disable:false} InitialFeeConfig:<nil>}"
INFO [02-19|08:38:52.725] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/trie/triedb/hashdb/database.go:521: Persisted trie from memory database nodes=17 size=1.80KiB time="58.951µs" gcnodes=0 gcsize=0.00B gctime=0s livenodes=0 livesize=0.00B
INFO [02-19|08:38:52.726] "<2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:457: lastAccepted = 0xe03d6b84c6086145327e7635a3e216bacf15f6a374445b8ba29281e6d70fb4b7" 
INFO [02-19|08:38:52.727] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/eth/backend.go:134: Allocated memory caches "trie clean"=512.00MiB "trie dirty"=512.00MiB "snapshot clean"=256.00MiB
INFO [02-19|08:38:52.733] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/eth/backend.go:171: Initialising Ethereum protocol network=17772 dbversion=8
INFO [02-19|08:38:52.742] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/state_processor.go:200: Activating new precompile name=contractDeployerAllowListConfig config="&{AllowListConfig:{AdminAddresses:[0x0Ed6431f48560E943cC8c1EdEAe3D7f7eDDE46A7] ManagerAddresses:[] EnabledAddresses:[]} Upgrade:{BlockTimestamp:0xc0008fd700 Disable:false}}"
INFO [02-19|08:38:52.742] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/state_processor.go:200: Activating new precompile name=contractNativeMinterConfig      config="&{AllowListConfig:{AdminAddresses:[0x0Ed6431f48560E943cC8c1EdEAe3D7f7eDDE46A7] ManagerAddresses:[] EnabledAddresses:[]} Upgrade:{BlockTimestamp:0xc0008fd8b8 Disable:false} InitialMint:map[]}"
INFO [02-19|08:38:52.742] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/state_processor.go:200: Activating new precompile name=feeManagerConfig                config="&{AllowListConfig:{AdminAddresses:[0x0Ed6431f48560E943cC8c1EdEAe3D7f7eDDE46A7] ManagerAddresses:[] EnabledAddresses:[]} Upgrade:{BlockTimestamp:0xc0008fda70 Disable:false} InitialFeeConfig:<nil>}"
INFO [02-19|08:38:52.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/trie/triedb/hashdb/database.go:521: Persisted trie from memory database nodes=17 size=1.80KiB time="30.095µs" gcnodes=0 gcsize=0.00B gctime=0s livenodes=0 livesize=0.00B
ERROR[02-19|08:38:52.750] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/genesis.go:246: found mismatch between config on database vs. new config storedConfig="{\"byzantiumBlock\":0,\"chainId\":17772,\"constantinopleBlock\":0,\"contractDeployerAllowListConfig\":{\"adminAddresses\":[\"0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7\"],\"blockTimestamp\":0},\"contractNativeMinterConfig\":{\"adminAddresses\":[\"0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7\"],\"blockTimestamp\":0},\"eip150Block\":0,\"eip155Block\":0,\"eip158Block\":0,\"feeConfig\":{\"gasLimit\":16000000,\"targetBlockRate\":2,\"minBaseFee\":24000000000,\"targetGas\":24000000,\"baseFeeChangeDenominator\":36,\"minBlockGasCost\":0,\"maxBlockGasCost\":10000000,\"blockGasCostStep\":500000},\"feeManagerConfig\":{\"adminAddresses\":[\"0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7\"],\"blockTimestamp\":0},\"homesteadBlock\":0,\"istanbulBlock\":0,\"muirGlacierBlock\":0,\"petersburgBlock\":0,\"subnetEVMTimestamp\":0,\"upgrades\":{\"precompileUpgrades\":[{\"feeManagerConfig\":{\"blockTimestamp\":1671106500,\"disable\":true}},{\"contractNativeMinterConfig\":{\"blockTimestamp\":1671106500,\"disable\":true}},{\"feeManagerConfig\":{\"adminAddresses\":[\"0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7\"],\"blockTimestamp\":1671112800}},{\"contractNativeMinterConfig\":{\"adminAddresses\":[\"0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7\"],\"blockTimestamp\":1671112800}},{\"contractNativeMinterConfig\":{\"blockTimestamp\":1671127200,\"disable\":true}},{\"contractNativeMinterConfig\":{\"adminAddresses\":[\"0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7\",\"0x1a3624ec8355229cc4597c2746c92035cef26241\"],\"blockTimestamp\":1671186630}}]}}" newConfig="{\"byzantiumBlock\":0,\"chainId\":17772,\"constantinopleBlock\":0,\"contractDeployerAllowListConfig\":{\"adminAddresses\":[\"0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7\"],\"blockTimestamp\":0},\"contractNativeMinterConfig\":{\"adminAddresses\":[\"0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7\"],\"blockTimestamp\":0},\"durangoTimestamp\":1707840000,\"eip150Block\":0,\"eip155Block\":0,\"eip158Block\":0,\"feeConfig\":{\"gasLimit\":16000000,\"targetBlockRate\":2,\"minBaseFee\":24000000000,\"targetGas\":24000000,\"baseFeeChangeDenominator\":36,\"minBlockGasCost\":0,\"maxBlockGasCost\":10000000,\"blockGasCostStep\":500000},\"feeManagerConfig\":{\"adminAddresses\":[\"0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7\"],\"blockTimestamp\":0},\"homesteadBlock\":0,\"istanbulBlock\":0,\"muirGlacierBlock\":0,\"petersburgBlock\":0,\"subnetEVMTimestamp\":0,\"upgrades\":{\"precompileUpgrades\":[{\"feeManagerConfig\":{\"blockTimestamp\":1671106500,\"disable\":true}},{\"contractNativeMinterConfig\":{\"blockTimestamp\":1671106500,\"disable\":true}},{\"feeManagerConfig\":{\"adminAddresses\":[\"0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7\"],\"blockTimestamp\":1671112800}},{\"contractNativeMinterConfig\":{\"adminAddresses\":[\"0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7\"],\"blockTimestamp\":1671112800}},{\"contractNativeMinterConfig\":{\"blockTimestamp\":1671127200,\"disable\":true}},{\"contractNativeMinterConfig\":{\"adminAddresses\":[\"0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7\",\"0x1a3624ec8355229cc4597c2746c92035cef26241\"],\"blockTimestamp\":1671186630}}]}}"
runtime engine: ignoring signal: terminated
runtime engine: ignoring signal: terminated
[02-19|12:57:19.072] ERROR <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> subprocess/linux_stopper.go:58 subprocess was killed
[02-19|12:57:19.080] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> subprocess/runtime.go:111 stdout collector shutdown
[02-19|12:57:19.080] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> subprocess/runtime.go:124 stderr collector shutdown
[02-19|14:47:47.711] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> subprocess/runtime.go:143 plugin handshake succeeded {"addr": "127.0.0.1:44835"}
[02-19|14:47:47.714] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> rpcchainvm/vm_client.go:165 grpc: serving database {"address": "127.0.0.1:44867"}
[02-19|14:47:47.715] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> rpcchainvm/vm_client.go:184 grpc: serving vm services {"address": "127.0.0.1:40081"}
INFO [02-19|14:47:47.717] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:291: Initializing Subnet EVM VM Version=v0.6.0-fuji Config="{AirdropFile: SnowmanAPIEnabled:false AdminAPIEnabled:false AdminAPIDir: WarpAPIEnabled:false EnabledEthAPIs:[eth eth-filter net web3 internal-eth internal-blockchain internal-transaction] ContinuousProfilerDir: ContinuousProfilerFrequency:15m0s ContinuousProfilerMaxFiles:5 RPCGasCap:50000000 RPCTxFeeCap:100 TrieCleanCache:512 TrieCleanJournal: TrieCleanRejournal:0s TrieDirtyCache:512 TrieDirtyCommitTarget:20 SnapshotCache:256 Preimages:false SnapshotWait:false SnapshotVerify:false Pruning:false AcceptorQueueLimit:64 CommitInterval:4096 AllowMissingTries:false PopulateMissingTries:<nil> PopulateMissingTriesParallelism:1024 PruneWarpDB:false MetricsExpensiveEnabled:true LocalTxsEnabled:false TxPoolJournal:transactions.rlp TxPoolRejournal:1h0m0s TxPoolPriceLimit:1 TxPoolPriceBump:10 TxPoolAccountSlots:16 TxPoolGlobalSlots:5120 TxPoolAccountQueue:64 TxPoolGlobalQueue:1024 APIMaxDuration:0s WSCPURefillRate:0s WSCPUMaxStored:0s MaxBlocksPerRequest:0 AllowUnfinalizedQueries:false AllowUnprotectedTxs:false AllowUnprotectedTxHashes:[0xfefb2da535e927b85fe68eb81cb2e4a5827c905f78381a01ef2322aa9b0aee8e] KeystoreDirectory: KeystoreExternalSigner: KeystoreInsecureUnlockAllowed:false RemoteGossipOnlyEnabled:false RegossipFrequency:1m0s RegossipMaxTxs:16 RegossipTxsPerAddress:1 PriorityRegossipFrequency:1s PriorityRegossipMaxTxs:32 PriorityRegossipTxsPerAddress:16 PriorityRegossipAddresses:[] LogLevel:info LogJSONFormat:false FeeRecipient: OfflinePruning:false OfflinePruningBloomFilterSize:512 OfflinePruningDataDirectory: MaxOutboundActiveRequests:16 MaxOutboundActiveCrossChainRequests:64 StateSyncEnabled:false StateSyncSkipResume:false StateSyncServerTrieCache:64 StateSyncIDs: StateSyncCommitInterval:16384 StateSyncMinBlocks:300000 StateSyncRequestSize:1024 InspectDatabase:false SkipUpgradeCheck:true AcceptedCacheSize:32 TxLookupLimit:0 WarpOffChainMessages:[]}"
INFO [02-19|14:47:47.719] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:443: Config has not specified any coinbase address. Defaulting to the blackhole address. 
INFO [02-19|14:47:47.719] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/state_processor.go:200: Activating new precompile name=contractDeployerAllowListConfig config="&{AllowListConfig:{AdminAddresses:[0x0Ed6431f48560E943cC8c1EdEAe3D7f7eDDE46A7] ManagerAddresses:[] EnabledAddresses:[]} Upgrade:{BlockTimestamp:0xc000f2c1a0 Disable:false}}"
INFO [02-19|14:47:47.719] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/state_processor.go:200: Activating new precompile name=contractNativeMinterConfig      config="&{AllowListConfig:{AdminAddresses:[0x0Ed6431f48560E943cC8c1EdEAe3D7f7eDDE46A7] ManagerAddresses:[] EnabledAddresses:[]} Upgrade:{BlockTimestamp:0xc000f2c358 Disable:false} InitialMint:map[]}"
INFO [02-19|14:47:47.719] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/state_processor.go:200: Activating new precompile name=feeManagerConfig                config="&{AllowListConfig:{AdminAddresses:[0x0Ed6431f48560E943cC8c1EdEAe3D7f7eDDE46A7] ManagerAddresses:[] EnabledAddresses:[]} Upgrade:{BlockTimestamp:0xc000f2c510 Disable:false} InitialFeeConfig:<nil>}"
INFO [02-19|14:47:47.720] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/trie/triedb/hashdb/database.go:521: Persisted trie from memory database nodes=17 size=1.80KiB time="46.085µs" gcnodes=0 gcsize=0.00B gctime=0s livenodes=0 livesize=0.00B
INFO [02-19|14:47:47.722] "<2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:457: lastAccepted = 0xe03d6b84c6086145327e7635a3e216bacf15f6a374445b8ba29281e6d70fb4b7" 
INFO [02-19|14:47:47.723] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/eth/backend.go:134: Allocated memory caches "trie clean"=512.00MiB "trie dirty"=512.00MiB "snapshot clean"=256.00MiB
INFO [02-19|14:47:47.727] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/eth/backend.go:171: Initialising Ethereum protocol network=17772 dbversion=8
INFO [02-19|14:47:47.735] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/state_processor.go:200: Activating new precompile name=contractDeployerAllowListConfig config="&{AllowListConfig:{AdminAddresses:[0x0Ed6431f48560E943cC8c1EdEAe3D7f7eDDE46A7] ManagerAddresses:[] EnabledAddresses:[]} Upgrade:{BlockTimestamp:0xc000f2c1a0 Disable:false}}"
INFO [02-19|14:47:47.736] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/state_processor.go:200: Activating new precompile name=contractNativeMinterConfig      config="&{AllowListConfig:{AdminAddresses:[0x0Ed6431f48560E943cC8c1EdEAe3D7f7eDDE46A7] ManagerAddresses:[] EnabledAddresses:[]} Upgrade:{BlockTimestamp:0xc000f2c358 Disable:false} InitialMint:map[]}"
INFO [02-19|14:47:47.736] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/state_processor.go:200: Activating new precompile name=feeManagerConfig                config="&{AllowListConfig:{AdminAddresses:[0x0Ed6431f48560E943cC8c1EdEAe3D7f7eDDE46A7] ManagerAddresses:[] EnabledAddresses:[]} Upgrade:{BlockTimestamp:0xc000f2c510 Disable:false} InitialFeeConfig:<nil>}"
INFO [02-19|14:47:47.736] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/trie/triedb/hashdb/database.go:521: Persisted trie from memory database nodes=17 size=1.80KiB time="29.195µs" gcnodes=0 gcsize=0.00B gctime=0s livenodes=0 livesize=0.00B
INFO [02-19|14:47:47.742] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/genesis.go:240: skipping verifying activated network upgrades on chain config 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:329:  
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:330: --------------------------------------------------------------------------------------------------------------------------------------------------------- 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332: Chain ID:  17772 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332: Consensus: Dummy Consensus Engine 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332:  
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332: Hard Forks: 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332:  - Homestead:                   #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md) 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332:  - Tangerine Whistle (EIP 150): #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md) 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332:  - Spurious Dragon/1 (EIP 155): #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md) 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332:  - Spurious Dragon/2 (EIP 158): #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md) 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332:  - Byzantium:                   #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md) 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332:  - Constantinople:              #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md) 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332:  - Petersburg:                  #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md) 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332:  - Istanbul:                    #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md) 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332:  - Muir Glacier:                #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md) 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332: Mandatory Upgrades: 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332:  - SubnetEVM Timestamp:           @0          (https://github.com/ava-labs/avalanchego/releases/tag/v1.10.0) 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332:  - Durango Timestamp:            @1707840000 (https://github.com/ava-labs/avalanchego/releases/tag/v1.11.0) 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332:  - Cancun Timestamp:              @nil        (https://github.com/ava-labs/avalanchego/releases/tag/v1.12.0) 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332:  
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332: Optional Network Upgrades: {} 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332: Precompile Upgrades: {"contractDeployerAllowListConfig":{"adminAddresses":["0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7"],"blockTimestamp":0},"contractNativeMinterConfig":{"adminAddresses":["0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7"],"blockTimestamp":0},"feeManagerConfig":{"adminAddresses":["0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7"],"blockTimestamp":0}} 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332: Upgrade Config: {"precompileUpgrades":[{"feeManagerConfig":{"blockTimestamp":1671106500,"disable":true}},{"contractNativeMinterConfig":{"blockTimestamp":1671106500,"disable":true}},{"feeManagerConfig":{"adminAddresses":["0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7"],"blockTimestamp":1671112800}},{"contractNativeMinterConfig":{"adminAddresses":["0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7"],"blockTimestamp":1671112800}},{"contractNativeMinterConfig":{"blockTimestamp":1671127200,"disable":true}},{"contractNativeMinterConfig":{"adminAddresses":["0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7","0x1a3624ec8355229cc4597c2746c92035cef26241"],"blockTimestamp":1671186630}},{"warpConfig":{"blockTimestamp":1707850000,"quorumNumerator":0}}]} 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332: Fee Config: {"gasLimit":16000000,"targetBlockRate":2,"minBaseFee":24000000000,"targetGas":24000000,"baseFeeChangeDenominator":36,"minBlockGasCost":0,"maxBlockGasCost":10000000,"blockGasCostStep":500000} 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332: Allow Fee Recipients: false 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332:  
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:334: --------------------------------------------------------------------------------------------------------------------------------------------------------- 
INFO [02-19|14:47:47.743] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335:  
INFO [02-19|14:47:47.749] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:732: Loaded most recent local header number=4,362,217 hash=e03d6b..0fb4b7 age=4d7h21m
INFO [02-19|14:47:47.749] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:733: Loaded most recent local full block number=4,362,217 hash=e03d6b..0fb4b7 age=4d7h21m
INFO [02-19|14:47:47.750] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:1812: Loaded Acceptor tip hash=e03d6b..0fb4b7
INFO [02-19|14:47:47.750] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:1820: Skipping state reprocessing root=505ec1..f880f2
INFO [02-19|14:47:47.751] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:1788: Initializing snapshots async=true rebuild=true headHash=e03d6b..0fb4b7 headRoot=505ec1..f880f2
INFO [02-19|14:47:47.789] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:580: Warmed accepted caches start=4,362,186 end=4,362,217 t=37.361112ms
INFO [02-19|14:47:47.789] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:586: Starting Acceptor "queue length"=64
WARN [02-19|14:47:47.807] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/internal/shutdowncheck/shutdown_tracker.go:67: Unclean shutdown detected booted=2024-02-18T04:18:19-0500 age=1d10h29m
INFO [02-19|14:47:47.807] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/txpool/txpool.go:521: Transaction pool price threshold updated price=0
[02-19|14:47:47.812] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> proposervm/vm.go:438 block height index was successfully verified
[02-19|14:47:47.812] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> proposervm/vm.go:230 initialized proposervm {"state": "after fork", "forkHeight": 2, "lastAcceptedHeight": 4362217}
[02-19|14:47:47.813] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> snowman/transitive.go:111 initializing consensus engine
INFO [02-19|14:47:47.815] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:970: Enabled APIs: eth, eth-filter, net, web3, internal-eth, internal-blockchain, internal-transaction 
[02-19|14:47:47.817] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> bootstrap/bootstrapper.go:159 starting bootstrapper
[02-19|14:47:49.920] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> bootstrap/bootstrapper.go:312 bootstrapping started syncing {"numAccepted": 1}
[02-19|14:47:49.921] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> bootstrap/bootstrapper.go:711 executing blocks {"numPendingJobs": 0}
[02-19|14:47:49.921] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> queue/jobs.go:224 executed operations {"numExecuted": 0}
[02-19|14:47:49.924] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> snowman/transitive.go:544 consensus starting {"lastAcceptedBlock": "2gRgasgqEaMczX5N3ov7U5ghpuCfKRa4pXGy1eGPNDBZLHB5jY"}
runtime engine: ignoring signal: terminated

2- The recent rpc node enters the network. The last line gives the FATAL error.

[03-04|12:37:25.567] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> subprocess/runtime.go:143 plugin handshake succeeded {"addr": "127.0.0.1:45239"}
[03-04|12:37:25.570] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> rpcchainvm/vm_client.go:160 grpc: serving database {"address": "127.0.0.1:37129"}
[03-04|12:37:25.570] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> rpcchainvm/vm_client.go:179 grpc: serving vm services {"address": "127.0.0.1:44797"}
INFO [03-04|12:37:25.572] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:289: Initializing Subnet EVM VM Version=v0.6.2 Config="{AirdropFile: SnowmanAPIEnabled:true AdminAPIEnabled:false AdminAPIDir: WarpAPIEnabled:false EnabledEthAPIs:[public-eth public-eth-filter net web3 internal-public-eth internal-public-blockchain internal-public-transaction-pool internal-public-debug internal-private-debug internal-public-tx-pool internal-public-account debug-tracer] ContinuousProfilerDir: ContinuousProfilerFrequency:15m0s ContinuousProfilerMaxFiles:5 RPCGasCap:50000000 RPCTxFeeCap:100 TrieCleanCache:512 TrieCleanJournal: TrieCleanRejournal:0s TrieDirtyCache:512 TrieDirtyCommitTarget:20 TriePrefetcherParallelism:16 SnapshotCache:256 Preimages:false SnapshotWait:false SnapshotVerify:false Pruning:false AcceptorQueueLimit:64 CommitInterval:4096 AllowMissingTries:false PopulateMissingTries:<nil> PopulateMissingTriesParallelism:1024 PruneWarpDB:false MetricsExpensiveEnabled:true LocalTxsEnabled:false TxPoolPriceLimit:1 TxPoolPriceBump:10 TxPoolAccountSlots:16 TxPoolGlobalSlots:5120 TxPoolAccountQueue:64 TxPoolGlobalQueue:1024 TxPoolLifetime:10m0s APIMaxDuration:0s WSCPURefillRate:0s WSCPUMaxStored:0s MaxBlocksPerRequest:0 AllowUnfinalizedQueries:false AllowUnprotectedTxs:false AllowUnprotectedTxHashes:[0xfefb2da535e927b85fe68eb81cb2e4a5827c905f78381a01ef2322aa9b0aee8e] KeystoreDirectory: KeystoreExternalSigner: KeystoreInsecureUnlockAllowed:false PushGossipNumValidators:100 PushGossipNumPeers:0 PushRegossipNumValidators:10 PushRegossipNumPeers:0 PushGossipFrequency:100ms PullGossipFrequency:1s RegossipFrequency:30s PriorityRegossipAddresses:[] LogLevel:info LogJSONFormat:false FeeRecipient: OfflinePruning:false OfflinePruningBloomFilterSize:512 OfflinePruningDataDirectory: MaxOutboundActiveRequests:16 MaxOutboundActiveCrossChainRequests:64 StateSyncEnabled:false StateSyncSkipResume:false StateSyncServerTrieCache:64 StateSyncIDs: StateSyncCommitInterval:16384 StateSyncMinBlocks:300000 StateSyncRequestSize:1024 InspectDatabase:false SkipUpgradeCheck:true AcceptedCacheSize:32 TxLookupLimit:0 SkipTxIndexing:false WarpOffChainMessages:[]}"
INFO [03-04|12:37:25.573] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:442: Config has not specified any coinbase address. Defaulting to the blackhole address.
INFO [03-04|12:37:25.573] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/state_processor.go:200: Activating new precompile name=contractDeployerAllowListConfig config="&{AllowListConfig:{AdminAddresses:[0x0Ed6431f48560E943cC8c1EdEAe3D7f7eDDE46A7] ManagerAddresses:[] EnabledAddresses:[]} Upgrade:{BlockTimestamp:0xc0001a3458 Disable:false}}"
INFO [03-04|12:37:25.573] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/state_processor.go:200: Activating new precompile name=contractNativeMinterConfig      config="&{AllowListConfig:{AdminAddresses:[0x0Ed6431f48560E943cC8c1EdEAe3D7f7eDDE46A7] ManagerAddresses:[] EnabledAddresses:[]} Upgrade:{BlockTimestamp:0xc0001a3678 Disable:false} InitialMint:map[]}"
INFO [03-04|12:37:25.573] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/state_processor.go:200: Activating new precompile name=feeManagerConfig                config="&{AllowListConfig:{AdminAddresses:[0x0Ed6431f48560E943cC8c1EdEAe3D7f7eDDE46A7] ManagerAddresses:[] EnabledAddresses:[]} Upgrade:{BlockTimestamp:0xc0001a3888 Disable:false} InitialFeeConfig:<nil>}"
INFO [03-04|12:37:25.573] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/trie/triedb/hashdb/database.go:521: Persisted trie from memory database nodes=17 size=1.80KiB time="34.797µs" gcnodes=0 gcsize=0.00B gctime=0s livenodes=0 livesize=0.00B
INFO [03-04|12:37:25.580] "<2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:456: lastAccepted = 0x45db17b560a3006acb419e645dea06b3560815cc4655eb1711b2293871907ca3"
INFO [03-04|12:37:25.581] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/eth/backend.go:142: Allocated memory caches "trie clean"=512.00MiB "trie dirty"=512.00MiB "snapshot clean"=256.00MiB
INFO [03-04|12:37:25.588] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/eth/backend.go:180: Initialising Ethereum protocol network=17772 dbversion=8
INFO [03-04|12:37:25.597] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/state_processor.go:200: Activating new precompile name=contractDeployerAllowListConfig config="&{AllowListConfig:{AdminAddresses:[0x0Ed6431f48560E943cC8c1EdEAe3D7f7eDDE46A7] ManagerAddresses:[] EnabledAddresses:[]} Upgrade:{BlockTimestamp:0xc0001a3458 Disable:false}}"
INFO [03-04|12:37:25.597] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/state_processor.go:200: Activating new precompile name=contractNativeMinterConfig      config="&{AllowListConfig:{AdminAddresses:[0x0Ed6431f48560E943cC8c1EdEAe3D7f7eDDE46A7] ManagerAddresses:[] EnabledAddresses:[]} Upgrade:{BlockTimestamp:0xc0001a3678 Disable:false} InitialMint:map[]}"
INFO [03-04|12:37:25.597] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/state_processor.go:200: Activating new precompile name=feeManagerConfig                config="&{AllowListConfig:{AdminAddresses:[0x0Ed6431f48560E943cC8c1EdEAe3D7f7eDDE46A7] ManagerAddresses:[] EnabledAddresses:[]} Upgrade:{BlockTimestamp:0xc0001a3888 Disable:false} InitialFeeConfig:<nil>}"
INFO [03-04|12:37:25.597] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/trie/triedb/hashdb/database.go:521: Persisted trie from memory database nodes=17 size=1.80KiB time="42.37µs"  gcnodes=0 gcsize=0.00B gctime=0s livenodes=0 livesize=0.00B
INFO [03-04|12:37:25.613] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/genesis.go:240: skipping verifying activated network upgrades on chain config
INFO [03-04|12:37:25.614] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:332:
INFO [03-04|12:37:25.614] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:333: ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [03-04|12:37:25.614] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335: Chain ID:  17772
INFO [03-04|12:37:25.614] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335: Consensus: Dummy Consensus Engine
INFO [03-04|12:37:25.614] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335:
INFO [03-04|12:37:25.614] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335: Hard Forks:
INFO [03-04|12:37:25.614] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335:  - Homestead:                   #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md)
INFO [03-04|12:37:25.614] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335:  - Tangerine Whistle (EIP 150): #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md)
INFO [03-04|12:37:25.614] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335:  - Spurious Dragon/1 (EIP 155): #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335:  - Spurious Dragon/2 (EIP 158): #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335:  - Byzantium:                   #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md)
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335:  - Constantinople:              #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md)
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335:  - Petersburg:                  #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md)
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335:  - Istanbul:                    #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md)
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335:  - Muir Glacier:                #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md)
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335: Mandatory Upgrades:
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335:  - SubnetEVM Timestamp:           @0          (https://github.com/ava-labs/avalanchego/releases/tag/v1.10.0)
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335:  - Durango Timestamp:            @1707840000 (https://github.com/ava-labs/avalanchego/releases/tag/v1.11.0)
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335:  - Cancun Timestamp:              @nil        (https://github.com/ava-labs/avalanchego/releases/tag/v1.12.0)
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335:
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335: Optional Network Upgrades: {}
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335: Precompile Upgrades: {"contractDeployerAllowListConfig":{"adminAddresses":["0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7"],"blockTimestamp":0},"contractNativeMinterConfig":{"adminAddresses":["0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7"],"blockTimestamp":0},"feeManagerConfig":{"adminAddresses":["0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7"],"blockTimestamp":0}}
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335: Upgrade Config: {"precompileUpgrades":[{"feeManagerConfig":{"blockTimestamp":1671106500,"disable":true}},{"contractNativeMinterConfig":{"blockTimestamp":1671106500,"disable":true}},{"feeManagerConfig":{"adminAddresses":["0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7"],"blockTimestamp":1671112800}},{"contractNativeMinterConfig":{"adminAddresses":["0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7"],"blockTimestamp":1671112800}},{"contractNativeMinterConfig":{"blockTimestamp":1671127200,"disable":true}},{"contractNativeMinterConfig":{"adminAddresses":["0x0ed6431f48560e943cc8c1edeae3d7f7edde46a7","0x1a3624ec8355229cc4597c2746c92035cef26241"],"blockTimestamp":1671186630}},{"warpConfig":{"blockTimestamp":1707850000,"quorumNumerator":0}},{"warpConfig":{"blockTimestamp":1707850001,"disable":true,"quorumNumerator":0}}]}
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335: Fee Config: {"gasLimit":16000000,"targetBlockRate":2,"minBaseFee":24000000000,"targetGas":24000000,"baseFeeChangeDenominator":36,"minBlockGasCost":0,"maxBlockGasCost":10000000,"blockGasCostStep":500000}
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335: Allow Fee Recipients: false
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:335:
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:337: ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [03-04|12:37:25.615] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:338:
INFO [03-04|12:37:25.624] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:737: Loaded most recent local header number=4,362,210 hash=45db17..907ca3 age=2w5d1h
INFO [03-04|12:37:25.624] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:738: Loaded most recent local full block number=4,362,210 hash=45db17..907ca3 age=2w5d1h
INFO [03-04|12:37:25.628] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:1818: Loaded Acceptor tip hash=45db17..907ca3
INFO [03-04|12:37:25.632] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:1826: Skipping state reprocessing root=b1bf13..5f830f
INFO [03-04|12:37:25.632] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:1794: Initializing snapshots async=true rebuild=true headHash=45db17..907ca3 headRoot=b1bf13..5f830f
INFO [03-04|12:37:25.691] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:585: Warmed accepted caches start=4,362,179 end=4,362,210 t=54.473813ms
INFO [03-04|12:37:25.691] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/blockchain.go:591: Starting Acceptor "queue length"=64
WARN [03-04|12:37:25.714] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/internal/shutdowncheck/shutdown_tracker.go:67: Unclean shutdown detected booted=2023-09-06T22:04:34+0000 age=5mo4w1d
WARN [03-04|12:37:25.714] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/internal/shutdowncheck/shutdown_tracker.go:67: Unclean shutdown detected booted=2023-09-07T07:51:02+0000 age=5mo4w1d
WARN [03-04|12:37:25.714] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/internal/shutdowncheck/shutdown_tracker.go:67: Unclean shutdown detected booted=2023-09-07T14:37:03+0000 age=5mo4w22h
WARN [03-04|12:37:25.714] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/internal/shutdowncheck/shutdown_tracker.go:67: Unclean shutdown detected booted=2023-09-14T13:21:07+0000 age=5mo3w23h
WARN [03-04|12:37:25.714] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/internal/shutdowncheck/shutdown_tracker.go:67: Unclean shutdown detected booted=2023-09-22T09:52:53+0000 age=5mo2w2h
WARN [03-04|12:37:25.714] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/internal/shutdowncheck/shutdown_tracker.go:67: Unclean shutdown detected booted=2023-10-27T14:01:41+0000 age=4mo1w1d
INFO [03-04|12:37:25.714] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/core/txpool/txpool.go:523: Transaction pool price threshold updated price=0
[03-04|12:37:25.721] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> proposervm/vm.go:436 block height index was successfully verified
[03-04|12:37:25.722] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> proposervm/vm.go:228 initialized proposervm {"state": "after fork", "forkHeight": 2, "lastAcceptedHeight": 4362210}
[03-04|12:37:25.723] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> snowman/transitive.go:110 initializing consensus engine
INFO [03-04|12:37:25.724] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:1090: deprecated api name referenced in configuration. deprecated=public-eth new=eth
INFO [03-04|12:37:25.724] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:1090: deprecated api name referenced in configuration. deprecated=public-eth-filter new=eth-filter
INFO [03-04|12:37:25.724] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:1090: deprecated api name referenced in configuration. deprecated=internal-public-eth new=internal-eth
INFO [03-04|12:37:25.724] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:1090: deprecated api name referenced in configuration. deprecated=internal-public-blockchain new=internal-blockchain
INFO [03-04|12:37:25.724] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:1090: deprecated api name referenced in configuration. deprecated=internal-public-transaction-pool new=internal-transaction
INFO [03-04|12:37:25.724] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:1090: deprecated api name referenced in configuration. deprecated=internal-public-debug            new=internal-debug
INFO [03-04|12:37:25.724] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:1090: deprecated api name referenced in configuration. deprecated=internal-private-debug           new=internal-debug
INFO [03-04|12:37:25.724] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:1090: deprecated api name referenced in configuration. deprecated=internal-public-tx-pool          new=internal-tx-pool
INFO [03-04|12:37:25.724] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:1090: deprecated api name referenced in configuration. deprecated=internal-public-account          new=internal-account
INFO [03-04|12:37:25.725] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:989: Enabled APIs: public-eth, public-eth-filter, net, web3, internal-public-eth, internal-public-blockchain, internal-public-transaction-pool, internal-public-debug, internal-private-debug, internal-public-tx-pool, internal-public-account, debug-tracer, snowman
[03-04|12:37:25.729] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> bootstrap/bootstrapper.go:159 starting bootstrapper
[03-04|12:37:25.933] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> bootstrap/bootstrapper.go:312 bootstrapping started syncing {"numAccepted": 1}
[03-04|12:37:27.092] INFO <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> bootstrap/bootstrapper.go:711 executing blocks {"numPendingJobs": 91}
[03-04|12:37:27.129] ERROR <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> bootstrap/block_job.go:90 block failed verification during bootstrapping {"blkID": "2PGQzuTjzBEcVqTnNmVwhAsk4PsXV88M7DT7wRtgmC973J4TVf", "error": "rpc error: code = Unknown desc = invalid bloom (remote: 00000000000000000000000000000000000000000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000020000000000000000000800000000000000000000000000000000400000000082000000000000000000000000000000000000000000000000000004000000020000000000000000000000000000000000000000000000000000000000000000000000000000010000000000000000000010000000000000000020000000000000000000000000000000000000000000000000000000000000000000  local: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000)"}
[03-04|12:37:27.141] FATAL <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> handler/handler.go:337 shutting down chain {"reason": "received an unexpected error", "error": "failed to execute job 2PGQzuTjzBEcVqTnNmVwhAsk4PsXV88M7DT7wRtgmC973J4TVf due to failed to verify block in bootstrapping: rpc error: code = Unknown desc = invalid bloom (remote: 00000000000000000000000000000000000000000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000020000000000000000000800000000000000000000000000000000400000000082000000000000000000000000000000000000000000000000000004000000020000000000000000000000000000000000000000000000000000000000000000000000000000010000000000000000000010000000000000000020000000000000000000000000000000000000000000000000000000000000000000  local: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000) while processing sync message: NodeID-6tWHHU9uVWAf46qTYTsnTnVZJeL65hwgR Op: ancestors Message: chain_id:\"\\xad\\xa9\\xf8K\\\\\\x0bH\\xbe\\x9d\\xaa\\xd8\\\\\\xf8F\\xb8\\xb8]lL<\\x99\\xd0R\\xca\\xf6s\\x9f\\xa2\\xfbb5\\xf5\"  request_id:2  containers:

Operating System
Debian 11 x64

Any help or clarification will be appreciated
@thesoulkiller thesoulkiller added the bug Something isn't working label Mar 4, 2024
@ceyonur
Copy link
Collaborator

ceyonur commented Mar 6, 2024 •
edited

validators that run with the old code (<v0.6.0) after Durango has created a faulty state. Did you copy from a validator DB to new nodes? If so they also have this faulty state.

We're working on a way to mitigate missed upgrades.

@thesoulkiller
Copy link
Author

thesoulkiller commented Mar 12, 2024 •
edited

we have 7 validators, 4 of them were operating normally but the other 3 fall in to this state. I copied db from one of the four validators to the others to mitigate the issue.
I am looking forward for updates !
Thank you

@ceyonur
Copy link
Collaborator

ceyonur commented Mar 26, 2024

Hello there!
We just released Subnet-EVM v0.6.3 focusing on this very issue. You can now coordinate an upgrade for your network to reschedule the Durango activation. I advise to set a timestamp in future, giving enough time so your network can coordinate the upgrade.
See here for more information.
If you have any questions feel free to ask.

@thesoulkiller
Copy link
Author

thesoulkiller commented Apr 22, 2024 •
edited

Hello I have upgraded to avalanchego-v1.11.4 and subnet-evm_0.6.3 but it couldn't detect networkUpgradeOverrides setting. I'm sharing the bootstrap logs.

Apr 22 07:41:28 vmi1540889.contaboserver.net avalanchego[1961877]: INFO [04-22|07:41:28.649] <2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo Chain> github.com/ava-labs/subnet-evm/plugin/evm/vm.go:288: Initializing Subnet EVM VM Version=v0.6.3 Config="{AirdropFile: SnowmanAPIEnabled:false AdminAPIEnabled:false AdminAPIDir: WarpAPIEnabled:false EnabledEthAPIs:[eth eth-filter net web3 internal-eth internal-blockchain internal-transaction] ContinuousProfilerDir: ContinuousProfilerFrequency:15m0s ContinuousProfilerMaxFiles:5 RPCGasCap:50000000 RPCTxFeeCap:100 TrieCleanCache:512 TrieDirtyCache:512 TrieDirtyCommitTarget:20 TriePrefetcherParallelism:16 SnapshotCache:256 Preimages:false SnapshotWait:false SnapshotVerify:false Pruning:false AcceptorQueueLimit:64 CommitInterval:4096 AllowMissingTries:false PopulateMissingTries:<nil> PopulateMissingTriesParallelism:1024 PruneWarpDB:false MetricsExpensiveEnabled:true LocalTxsEnabled:false TxPoolPriceLimit:1 TxPoolPriceBump:10 TxPoolAccountSlots:16 TxPoolGlobalSlots:5120 TxPoolAccountQueue:64 TxPoolGlobalQueue:1024 TxPoolLifetime:10m0s APIMaxDuration:0s WSCPURefillRate:0s WSCPUMaxStored:0s MaxBlocksPerRequest:0 AllowUnfinalizedQueries:false AllowUnprotectedTxs:false AllowUnprotectedTxHashes:[0xfefb2da535e927b85fe68eb81cb2e4a5827c905f78381a01ef2322aa9b0aee8e] KeystoreDirectory: KeystoreExternalSigner: KeystoreInsecureUnlockAllowed:false PushGossipPercentStake:0.9 PushGossipNumValidators:100 PushGossipNumPeers:0 PushRegossipNumValidators:10 PushRegossipNumPeers:0 PushGossipFrequency:100ms PullGossipFrequency:1s RegossipFrequency:30s PriorityRegossipAddresses:[] LogLevel:info LogJSONFormat:false FeeRecipient: OfflinePruning:false OfflinePruningBloomFilterSize:512 OfflinePruningDataDirectory: MaxOutboundActiveRequests:16 MaxOutboundActiveCrossChainRequests:64 StateSyncEnabled:false StateSyncSkipResume:false StateSyncServerTrieCache:64 StateSyncIDs: StateSyncCommitInterval:16384 StateSyncMinBlocks:300000 StateSyncRequestSize:1024 InspectDatabase:false SkipUpgradeCheck:true AcceptedCacheSize:32 TxLookupLimit:0 SkipTxIndexing:false WarpOffChainMessages:[]}"
Apr 22 07:41:28 vmi1540889.contaboserver.net avalanchego[1961877]: [04-22|07:41:28.651] ERROR chains/manager.go:331 error creating chain {"subnetID": "21HEmZx5zVHYcP3JzbmRGVsYdm3HjrM2BMEPoCpoS3fHmZshq9", "chainID": "2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo", "chainAlias": "2KV1ighhTjNpuQq8BVgHeJF3QHdF3KxhY9AqB9M1GfUuBCKjNo", "vmID": "nyfSdZmrxTXbJrxdUoqLegVGQzWF6RVL4jYn7Yr6NsMzpdrFA", "error": "error while creating new snowman vm rpc error: code = Unknown desc = failed to parse upgrade bytes: unknown precompile config: networkUpgradeOverrides"}

EDIT
My bad, I included the configuration in precompileUpgrades, when I just append the upgrade.json with this config, it worked.
Thank you !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
Platform Engineering Group
Archived in project
Milestone
No milestone
Development

No branches or pull requests
2 participants
@thesoulkiller @ceyonur