Stopped mining after - invalid receipt root hash

Mining operation just stopped solving new blocks after:
I0924 12:26:25.288947 23946 chain_manager.go:777] Bad block #282114 (aed92f3e6a65e6e9e5793a37e6f359faf9ed53bcd34d66fa3317fb49fa6dab72)
I0924 12:26:25.288984 23946 chain_manager.go:778] invalid receipt root hash. received=ccdf67e75293cc26b88f287498c5ff410955a0ead8e49dfde931fb46652a1e03 calculated=52c96a14b6923a0f837ae0b98be6ce480717b15ae6810d68eb79ebcc1298c078

I didn't have time to check this deeper, if this occur again, i will post update here.

Seems my node stucked on block 282114 claiming that this block was mined by this node, and couldn't fix itself, im currently rebuilding blockchain form network. Why it didn't reward with uncle? Im not sure what happend.

I0928 21:30:32.788178 12453 chain_manager.go:844] Bad block #303874 (f1fad94b1cac720e7ee5a70af71d1c6116d939a1a6ba9e68edb59e28dc0f1659)
I0928 21:30:32.788210 12453 chain_manager.go:845] invalid receipt root hash. received=a6e0c372dd873a33045feafefbf3334cb22afe14e06e3891b296f26d1168d07c calculated=7867cd230a6f16929644339c64a8861eaf583a2724c39e38d9b71de36f41d88b

Happened again

[obscuren]

What is the version, the cli argument you used and how many miners did you connect to the node?

rpc as usual, and maxpeers 80
version -> Geth/v1.2.0-59a876d8/linux/go1.5 but it happened with previous commit also.
Around 100 workers on node

[d10r]

I'm also having such a case:

I1019 11:09:51.879506    2423 downloader.go:255] Synchronisation failed: block downloading canceled (requested)
E1019 11:09:53.156899    2423 blockchain.go:815] Bad block #399738 (0xa15e2f70137c6fcc8c56eac4c3c22504195aeef8257f5a1c7063a3a6dd013425)
E1019 11:09:53.156950    2423 blockchain.go:816]     invalid receipt root hash. received=2fb4538c5376dccd19099000fb4ac0c15d07fefb7db08f2e2d8d9bf1dc7c9b96 calculated=30e851f90bd7eacdb6ff931942698ca17f2b168b2c9c0523211dbbfd5932d728
...

Seems stuck in an endless loop of retrying this block.

More info about this block:

> eth.getBlock(399738)
{
  difficulty: 6052312126209,
  extraData: "0xd983010202844765746887676f312e342e328777696e646f7773",
  gasLimit: 3141592,
  gasUsed: 653034,
  hash: "0x94c36f9750998d870e4cc000d1efabfee20d96c8392d652664248165a9fc933b",
  logsBloom: "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
  miner: "0x59df0302930fc09bdf428d5467d862a9c73b2830",
  nonce: "0xe778a3b5b6390de3",
  number: 399738,
  parentHash: "0xed2cac557f755a5106933c827639b953fa4b30e4ac54f1148a42923c278a51eb",
  receiptRoot: "0x73a040ebf39f915364d0dc3dee970a47acd18ee5bb986e1dafe14e171eb4563e",
  sha3Uncles: "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
  size: 1174,
  stateRoot: "0xd9ba8607bbc9e0b79ed7bf4cd5d3997d03498434e1013b918ea7df92ca686f3c",
  timestamp: 1445125530,
  totalDifficulty: 2012547993895340928,
  transactions: ["0x01da39e82fdddb7ff8dbd8ef82aa35f9e76d8892f8e16f65404e7fbbbe1aa1b4", "0x31956510e64e428ddc7a7422d66bb4273dcc443365f93510b4430386cb65f7b2", "0x26e0fe4487fb7b08d2b3096e80a188f2e9855b69697e1cf88e8da6c340dbd862", "0x6803443cb1fb77421d98af5337481d78fb551ca17b196ea4ad5b23185341e800", "0x7ad3d2533d632c7e48f1cf06934407461b55107c1e56f8e920843430783dfd23"],
  transactionsRoot: "0xf62c4a2d8204eacc650963c56729a2cccedb2f98f4ee677088049da029ced443",
  uncles: []
}

Running Geth/v1.3.0-dev/linux/go1.5

What can I do about this?

[obscuren]

Could you zip up your database (OS X: ~/Library/Ethereum/chaindata, Linux: ~/.ethereum/chaindata) and upload it somewhere? please make sure you only zip up the chain data folder

[obscuren]

Also what does your eth.getBlock("latest") report you? It could be a bad peer trying to feed you bad blocks. I've seen similar reports on that block with the same hash.

The odd thing is that you do indeed have a block with number 399738 in your database which isn't the block mentioned in the error:

• Error: Bad block #399738 (0xa15e2f70137c6fcc8c56eac4c3c22504195aeef8257f5a1c7063a3a6dd013425)
• OK: hash: "0x94c36f9750998d870e4cc000d1efabfee20d96c8392d652664248165a9fc933b"

@Gustav-Simonsson mentioned a similar issue with the exact same hash. Looking in my own develop database I have accepted that block. This could mean that there's some earlier commit caused a consensus issue. Before you update your repo could you check which commit you are on? git cat-file commit HEAD?

[d10r]

> eth.getBlock("latest")
{
  difficulty: 6049356895681,
  extraData: "0x",
  gasLimit: 3141592,
  gasUsed: 2083277,
  hash: "0xb5babde66b2dc47d75fa0eff7ee6ecee543d0f47987c0af577747efea5a45ea8",
  logsBloom: "0x00000000000000000000000000004000000000000000000000100000000000080000000000000000000000000000000000000000000000000000000080000008000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000010000000004000000004000000000000000000000000000000000000000400000000020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000000000000000",
  miner: "0x070c8963cf4abce1bfaf5973f2e0f7697925b242",
  nonce: "0xe5e8d290a42fb385",
  number: 399739,
  parentHash: "0x94c36f9750998d870e4cc000d1efabfee20d96c8392d652664248165a9fc933b",
  receiptRoot: "0x7f3f79fdaa8e5f272324c8bfda4f5cf836a07596013ec053856936124356b9e5",
  sha3Uncles: "0x4167959c8009c8e312a7ebb8822ca4b7392c003dd4c150107523d6a8365f149c",
  size: 12250,
  stateRoot: "0x76e74be8e79f385e87fbda663861045a736d7992e0db0c944080d768badb9904",
  timestamp: 1445131437,
  totalDifficulty: 2012554043252236609,
  transactions: ["0xb5123d863cb597100525c4c5ba8a975500c41b7d43827c58b8e9c9d569004915", "0x6259686146b96fd7ad810afbda743e9cd39e58668dc5b87f2c1c9c990699e1a1", "0x9c82377a022f44e78980fd3e5dd27668b7507f20f0752818f75a775683eb05d6", "0xb6c32d0f4534d4ce2c426157fc3bf4a12f9c06bc16fbc30cb28aebcc4dc15dbe", "0x166f76b9b9814b060b9ae0bd9e5e5405513c317d17ee89ed1e95a62d1a06cd53", "0x9f6bb5d085ab82e68e0dedebf757cbbaef837a952099ff0b204f5aa0a699de07", "0xb16da62e06a0fea3cc0a14c650744c56a6225f95732bb582538dab1463443b5b", "0x099ed0e5d47a3014bc03264043ca7f7dfda744caa44cf7b384f347797335d9d5", "0x7f5fef12d7468b04760854304d0ee05de520f119f9f08ffde1509828b343aed2", "0x2c6a522da70c402cc7bb9041fa3ec03f6a95067b598f80b5652ce89ced15ec92", "0x8a459135f5650a5dc3abf82076e7e7fa0f03b981021f19460aacfcb81ce69e5c", "0xc163c0aa57cef015c7dd037c5b8022867f6bc4ff2af260f0ce73ac29eaf83a9b", "0x3ff4abedae808e8ef8a868af2552262c1a7e3c3198c53b0b7f4edf529d2fd19c", "0xdc842b4037afa7bb82724dc999493ca4384114af584755107fee65f8a67555ac", "0x44f08e1325bcd807f6e89fa40c869a887cdcc4175920b5234720b96feef2485e", "0x4aecf820bb4e460d0a7393090481547c956c63a5326eb908b88f2c0d9ca98408", "0x71987d18cd4a6cd6f5e13383a0b606c77b722bf54f49ae380289f82b669962ff", "0x407f1b89a236b4f506b312953df666db3a393e3c88e855b4bce3067c82f89d7f", "0xd92f4b166dac02d2ab6aa11ba61fecb6e5d974a8db353119af5c9d85360f73c4", "0x6dc3af1355bb40069ff7af171723f2df10a3c4fb23253cf98ff2c2a22ca137eb", "0x49085501c871fa6ecfbeeba5feceac03f22efd267f1c448f7a9f647b89d22741", "0xa1d03285d0fdc700114b4c4e23b59d69204bd1dcecf1d2325be45e3671031eb1", "0x788a5b46cba525bbec2595674677eb9cdda8c01946703f8cb27689e41b1ae949", "0x330987e17f26c320f992c1f325ba1b31c9728807fc1f92c0d9c97e3f41bcdbe9", "0x641923e46ade862cce8bd257acbdbb115a32c5dffaa1502ef5b006c3b19e839d", "0x8a24618debd31eb43daea36b09d276adcd93ebbd7ed9c24467bc324651671e49", "0xd27c058e6113f50f57f54324b50dc10289b598026901cc74627f2a1292e1f3c6", "0x913b57d80064ac4b77a37d75ee27091077487d8c262edcbfeb202ca6af69f464", "0xccd18642eca949c2ce7ad60ecd83ef108d0913138e08c4287918b0c43df3aa87", "0xe400715c875769b639c140f3f21d2b2060ba5a2c48a76278b230134c641bac3b", "0x822dedb47382ef1dd89d9ca76ced93a284103915d4ad3538231b949671f0fb26", "0xd2e93174f54cb5700eeeacd70e444a70a50f7dd47c10df9143757abe527bbaa6", "0x5058bcb4f97ba90873331a1bb7acba58436d868266854e98c32004652abb0c1e", "0xb3dfac910cc221b2bc370e193d8bb69e0d557f4abe1f5202edbd061d852e3a82", "0xb6d845bf807b412628477d5b96650843f31d53956d2347099336840683cfc185", "0x277bb09e0833fd063951b3144adf0e913bc83ff6068273799c615cdcbd870338", "0xc90282a8b3e58c6172ffe5f787092f65749cce0239bb11dc1f8637473c0c3703", "0x13bc6ffb6fc86af7c0f4ec4cc801e1b0b6648cc31bae194d120319a7a5068826", "0x93734ca66d0e3d9a2a736225242173ba8b9622678360d8eee14794905a0831f4", "0x2d5baf6abb26294690d1a85f4d218198ae2201933b747a693718b5a7e1460d81", "0x94c95e7d485b4cf101809143cbde392724c75db78b810514beadf1fd581e8974", "0x5db1e19ba95726a948fe12f5626682acd687b0679f4903dc107db5a32c67afed", "0xccc56b56f13d8298ca29cc9c76c4572dd59edb410a7a80785638273d280d990e", "0xa0416b8a4883804fe83a2e06ba59af0e643f9959988459128be8d501bc18c613", "0x6127dd734d12bbeec92c44ca191c6eca5166b98b00001e56aa318c8c757bc758", "0x9768b4534a6b051d7d12ea41d316b146b6eec36f3c7b7b6ea8e51393620549f5", "0x1b56fe49dc0da9a1149b2f590db2dd80ad63a6341d9abea0a40c40d38eb6c5d1", "0xc7a00cf5a89165944a05622bb9799c2db9947e1cbd7ce940f7ae828840b24aba", "0x3db65b4d27a08297f7ddc7c268eaa87c3e712da9fe303752158a7fe38938d06b", "0xf0282ff22da68a149f940b16a148ce1b0e902b8a37b13efdd5bfa6972705880b", "0x0ed2b4687897eafef997b7a26a86495b8ed0e49acd90b3c27748d1fba254dabf", "0xc6e4fcfc827c9a18ab603d309d894378f4d55db2fc22865e30eb4936ef380b8c", "0x3a31394dd73e37b5c5fee2cafc6ebd4c109e5960a5111d1e9acb9d5df2218f58", "0x3d7addecf57fbd1db5e652cc257ced9f7f3dd3fc70eb32ecbbb45b887a986111", "0x6850bb0c64add209f0ff99bd2fe3732ec270ea37ba67ccd54eefa0619fc2ee42", "0xfdcfe074b8e11a97e55b1a4f626b4b0f3841f7dced3acd7d7a33e6b3a2c55fd1", "0x2ad03bad02e305d5c37b346f5dd077734d07bd573d7dd84721768a7a84505543", "0xef9d8d37b331ef5898e9cee0dea2368e7d03397b1543f287197acdaf46609fe3", "0xc0049499e41aafdab2ac46c923e79cabb76c1d0f5b19968e5e3ccbbdac69c46d", "0x0e374818cbf20fa5d580b84ff756bc9d9456224f38a9565c795b9438dac04c89", "0xfb1606ff60aaff94ffe532a1d7fffcf00cb38fc8a8da8ea0449eb4bf45d40b50", "0x32ac8f36d3d09103f35bca6863c3efb605fbcd41ce90d9cb5a16fed41baeeefb", "0x96a5fe4226b1897e97b2314bd759bf133a268b49f537f94b8085a68950229aa2", "0x4831248f10c733282cfa229bc2cf5ee7c167665fde99e6137e49971b3610b599", "0xb098d3ba0cb1d91a18cf8f3dec8e3a6d34b3232e6cad3e1e943f08beac36144b", "0xf9e61478e906e18f5b605bd509eca7ff6f5794e603dce0d2d05dba109f06f937", "0x2a718fb59652ea15881dbe456c0cc2aa4734f488fe76be5c89fb213d25193310", "0x59e9029b5b44ac8c24443428ca3c80389fd140d368c27d1c4b82f5da8c868d36", "0xdfce80c3a68b14552de852c80904c0c97036283d5523c774ed8c9c375b3ea8e2", "0x76dbd58ca21a3df7317db49adfa2f9e81c4b4f2ee2df5345c7f4c09fc4ec88a1", "0x7540412e5663153714403e20e93db6969519dc8a572c12eba22c9ad984c5e87f", "0x7360620c32709ecede6bb625993a0dc37667df527e50335c1694b412b3b5fee9", "0xe265bb94e936bfab641cbd22bdd2f2d24e7470ff5208903ff02e2bde16ed1545", "0x5334d824b8ab8b2993ff99d379dbc3f98e6779004e7be110e2cfacfa893ad030", "0x0796e207f494486617d584209ded84d804e0b550413e059451ce1d2adc29d59a", "0xb6c79c89a2d0ee5f65b8b8da68f2b95a087031e62bb716b2c5b60252f1b97884", "0x81b2dee8606ef9a51a9baa58aa6a646f600889a37f9edc6aec37af9253df3b95", "0xec24798c87fd6e4af759b3f6f848f06165f2d1bdb99bfa460e2cc756d6fb52a6", "0xee60b9a4e267f48d9826876d8576f293049f37ea43a7090a1da210d536370480", "0xa627744025097e974426c92bf5c025cace6d8d893e70b9b4763849b60ff8b8be", "0x436b0ded0c23d7f7b04c644ab178987d7115209e2ae9af3568ec1b2853d907cf", "0x5187501f07ccc99ee9dce89869558b7040ff60d07c553b062c344e1a7c0f2a41", "0x67054405c24bcf13ec5018d514c157e5b7cbca59376f85dd103c7a7d5bc045fc", "0xca09ebcccab6899340cdf7d625f904487617a0983056a37211c16b8a895d1deb", "0x1d3ea32ddd89be4c85b149e003dd722d42ac212a22f2f30be5da2e2b6690ac3b", "0xcd84816ecbe35c6d73c2b1e9fb03e76c61fbece4eea916d178c3d9047a5cae0e", "0xf57e002b1421d1fcae4fbeb3ca7f619793c12b45c566a697a78f62bdc07f3d9f", "0xb2ae475af8ccc1585ed76633a95939679309a6bc9518cb55f8329cbebe5d6b0d", "0xf71feaf57be63a30fb80879d80dd3acb4324f1254b86b0a829f18349d63b739f", "0x8eb954a2f7db77af0c79b894ad018e1d666e4d32476c0679b020e22907be5b79", "0x15ad004e32809b79860114e1499944c42c87a15f252fb7af7d194ce92942a9de", "0x455a185fc6f54897d29eb6437338b82ce4570c434e3f299976d5fc496c92f0a4", "0xa311d14d375d2ec2600e2c3130f1e6385888d13abcd03e2173716accea09c2c6", "0xa7935f20323bbd9e7a5aee21c730f50d11d5670558995948d5a656596e5d6513", "0xba908eff1b3cd9e56568978c63bada2be5a99ab3bb188feb9f5a4f15c7a3337c", "0x874274d6e8a96bc58a19b161f6f07017d01e71c5e19acf00870859896b957b15", "0xe7d1375d4b3fac24c02b6feb086dd706c77fc6c3f80197c17cef0d5579015efa", "0xbed3c1df14d335ce0893cccb8aeb754c2edd5c9137d15cbceffb9872e36b9eeb", "0xa2c1dc6401e2ac4ee85ab2255f812f9d2f7905567c6c384ee46975f6f0474080"],
  transactionsRoot: "0xb5f5eeccf079913c49d32380a7a518f39a4a9aef7d3c25731022a171ed40ee3c",
  uncles: ["0xd3c0be3cb434299ac3e242b718944cec70ed8c398ba261356ca9d2bc6719c06d"]
}

I've currently installed ethereum via Ubuntu PPA https://launchpad.net/~ethereum, not sure if/how I can find out the git rev from there.
Just had an update today:

Unpacking geth (1.2.0+804SNAPSHOT20151020103427trusty-0ubuntu1) over (1.2.0+803SNAPSHOT20151016193959trusty-0ubuntu1) ...

The zipped chain folder has ~ 1.7 GB. I'm preparing the upload...

[d10r]

Here's the zipped chain folder: http://tmp.b10s.net/eth/buggy_chain.zip

[obscuren]

Thank you. We are looking in to this

[obscuren]

@bcg-didi where you mining when this occurred?

[d10r]

It was running as

geth console miner minerthreads 1

(or maybe 2 threads, not sure about this)

Update:
hmm, I'm just wondering about that cmdline myself, seems to not make much sense when looking at geth --help. However it seems to do the same as geth --minerthreads 1 console.

[obscuren]

@bcg-didi Thank you. We've identified the issue and prepared a fix. Waiting for tests to be written up and we'll merge and release a new version as soon as it's done