Exception Syncing MainNet Chain from Block file #391

jsolman · 2018-09-21T15:50:06Z

I encountered the following exception while syncing the chain on MainNet using 2.9.0. It could have been hardware failure or something, but wanted to see if others have encountered it and discuss.

The exception was thrown in Blockchain.cs line 495:

                            account.Balances[out_prev.AssetId] -= out_prev.Value;

Doesn't seem like it should ever happen.

Stacktrace:

[ERROR][9/21/18 7:59:10 AM][Thread 0004][akka://NeoSystem/user/$a] The given key '0x602c79718b16e442de58778e148d0b1084e3b2dffd5de6b7b16cee7969282de7' was not present in the dictionary.
Cause: System.Collections.Generic.KeyNotFoundException: The given key '0x602c79718b16e442de58778e148d0b1084e3b2dffd5de6b7b16cee7969282de7' was not present in the dictionary.
   at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
   at Neo.Ledger.Blockchain.Persist(Block block) in /path/to/neo/Ledger/Blockchain.cs:line 495
   at Neo.Ledger.Blockchain.OnImport(IEnumerable`1 blocks) in /path/to/neo/Ledger/Blockchain.cs:line 249
   at Neo.Ledger.Blockchain.OnReceive(Object message) in /path/to/neo/Ledger/Blockchain.cs:line 406
   at Akka.Actor.UntypedActor.Receive(Object message)
   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
   at Akka.Actor.ActorCell.ReceiveMessage(Object message)
   at Akka.Actor.ActorCell.Invoke(Envelope envelope)
[ERROR][9/21/18 7:59:10 AM][Thread 0004][akka://NeoSystem/user/$a] Error while creating actor instance of type Neo.Ledger.Blockchain with 2 args: (Neo.NeoSystem,Neo.Persistence.LevelDB.LevelDBStore)
Cause: [akka://NeoSystem/user/$a#1418755616]: Akka.Actor.PostRestartException: Exception post restart (System.Collections.Generic.KeyNotFoundException) ---> System.TypeLoadException: Error while creating actor instance of type Neo.Ledger.Blockchain with 2 args: (Neo.NeoSystem,Neo.Persistence.LevelDB.LevelDBStore) ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.InvalidOperationException: Operation is not valid due to the current state of the object.
   at Neo.Ledger.Blockchain..ctor(NeoSystem system, Store store) in /path/to/neo/Ledger/Blockchain.cs:line 158
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.RuntimeType.CreateInstanceImpl(BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
   at Akka.Actor.Props.ActivatorProducer.Produce()
   at Akka.Actor.Props.NewActor()
   --- End of inner exception stack trace ---
   at Akka.Actor.Props.NewActor()
   at Akka.Actor.ActorCell.CreateNewActorInstance()
   at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<NewActor>b__0()
   at Akka.Actor.ActorCell.UseThreadContext(Action action)
   at Akka.Actor.ActorCell.NewActor()
   at Akka.Actor.ActorCell.FinishRecreate(Exception cause, ActorBase failedActor)
   --- End of inner exception stack trace ---

The text was updated successfully, but these errors were encountered:

jsolman · 2018-09-21T15:53:58Z

The block file I used was the full chain package from sync.ngd.network
Here are details of the file I obtained from sync.ngd.network:

2018/09/18 | 2742082 | 3.68G | 9h | download MD5:FBC140BAF08939935D7E0A51635F3B7B

jsolman · 2018-09-21T15:55:31Z

~~I suspect something may be wrong with the file~~, but I'm not certain yet. I verified that the MD5 matched though after downloading.

jsolman · 2018-09-21T18:30:33Z

Hmm. it's happened again for me when loading a previous chain file I had used on block 2183574

[ERROR][9/21/18 6:12:05 PM][Thread 0008][akka://NeoSystem/user/$a] The given key was not present in the dictionary.
Cause: System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary.
   at Neo.IO.Caching.DataCache`2.GetAndChange(TKey key, Func`1 factory) in .../neo/IO/Caching/DataCache.cs:line 211
   at Neo.Ledger.Blockchain.Persist(Block block) in .../neo/Ledger/Blockchain.cs:line 474
   at Neo.Ledger.Blockchain.OnImport(IEnumerable`1 blocks) in .../neo/Ledger/Blockchain.cs:line 249
   at Neo.Ledger.Blockchain.OnReceive(Object message) in .../neo/Ledger/Blockchain.cs:line 406
   at Akka.Actor.UntypedActor.Receive(Object message)
   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
   at Akka.Actor.ActorCell.ReceiveMessage(Object message)
   at Akka.Actor.ActorCell.Invoke(Envelope envelope)
[ERROR][9/21/18 6:12:05 PM][Thread 0021][akka://NeoSystem/user/$a] Error while creating actor instance of type Neo.Ledger.Blockchain with 2 args: (Neo.NeoSystem,Neo.Persistence.LevelDB.LevelDBStore)
Cause: [akka://NeoSystem/user/$a#1933411365]: Akka.Actor.PostRestartException: Exception post restart (System.Collections.Generic.KeyNotFoundException) ---> System.TypeLoadException: Error while creating actor instance of type Neo.Ledger.Blockchain with 2 args: (Neo.NeoSystem,Neo.Persistence.LevelDB.LevelDBStore) ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.InvalidOperationException: Operation is not valid due to the current state of the object.
   at Neo.Ledger.Blockchain..ctor(NeoSystem system, Store store) in .../neo/Ledger/Blockchain.cs:line 158
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.RuntimeType.CreateInstanceImpl(BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
   at Akka.Actor.Props.ActivatorProducer.Produce()
   at Akka.Actor.Props.NewActor()
   --- End of inner exception stack trace ---
   at Akka.Actor.Props.NewActor()
   at Akka.Actor.ActorCell.CreateNewActorInstance()
   at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<NewActor>b__0()
   at Akka.Actor.ActorCell.UseThreadContext(Action action)
   at Akka.Actor.ActorCell.NewActor()
   at Akka.Actor.ActorCell.FinishRecreate(Exception cause, ActorBase failedActor)
   --- End of inner exception stack trace ---

Slightly different code path.

jsolman · 2018-09-21T19:01:12Z

This may be due to a change I made not present in current core code. I've backed out my local change and will close the issue if the problem clears up.

jsolman · 2018-09-21T20:58:36Z

Seems it was related to a change I was testing

jsolman · 2018-09-22T08:36:29Z

I spoke too soon; it still happened; just later along this time at block 2450457:

[ERROR][9/21/18 11:31:46 PM][Thread 0023][akka://NeoSystem/user/$a] The given key '0xc56f33fc6ecfcd0c225c4ab356fee59390af8560be0e930faebe74a6daff7c9b' was not present in the dictionary.
Cause: System.Collections.Generic.KeyNotFoundException: The given key '0xc56f33fc6ecfcd0c225c4ab356fee59390af8560be0e930faebe74a6daff7c9b' was not present in the dictionary.
   at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
   at Neo.Ledger.Blockchain.Persist(Block block) in .../neo/Ledger/Blockchain.cs:line 495
   at Neo.Ledger.Blockchain.OnImport(IEnumerable`1 blocks) in .../neo/Ledger/Blockchain.cs:line 249
   at Neo.Ledger.Blockchain.OnReceive(Object message) in .../neo/Ledger/Blockchain.cs:line 406
   at Akka.Actor.UntypedActor.Receive(Object message)
   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
   at Akka.Actor.ActorCell.ReceiveMessage(Object message)
   at Akka.Actor.ActorCell.Invoke(Envelope envelope)
[ERROR][9/21/18 11:31:46 PM][Thread 0024][akka://NeoSystem/user/$a] Error while creating actor instance of type Neo.Ledger.Blockchain with 2 args: (Neo.NeoSystem,Neo.Persistence.LevelDB.LevelDBStore)
Cause: [akka://NeoSystem/user/$a#772508927]: Akka.Actor.PostRestartException: Exception post restart (System.Collections.Generic.KeyNotFoundException) ---> System.TypeLoadException: Error while creating actor instance of type Neo.Ledger.Blockchain with 2 args: (Neo.NeoSystem,Neo.Persistence.LevelDB.LevelDBStore) ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.InvalidOperationException: Operation is not valid due to the current state of the object.
   at Neo.Ledger.Blockchain..ctor(NeoSystem system, Store store) in.../neo/Ledger/Blockchain.cs:line 158
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.RuntimeType.CreateInstanceImpl(BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
   at Akka.Actor.Props.ActivatorProducer.Produce()
   at Akka.Actor.Props.NewActor()
   --- End of inner exception stack trace ---
   at Akka.Actor.Props.NewActor()
   at Akka.Actor.ActorCell.CreateNewActorInstance()
   at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<NewActor>b__0()
   at Akka.Actor.ActorCell.UseThreadContext(Action action)
   at Akka.Actor.ActorCell.NewActor()
   at Akka.Actor.ActorCell.FinishRecreate(Exception cause, ActorBase failedActor)
   --- End of inner exception stack trace ---

Going to have to dive deeper.

jsolman · 2018-09-22T09:49:41Z

Found one place where (outside of neo core code) where I was creating a chain Snapshot and failing to Dispose it; seems like an odd way to cause this if it is related, but trying syncing again with that fixed.

jsolman · 2018-09-22T10:21:39Z

Didn't fix it. Will look deeper at the end of the weekend.

jsolman · 2018-09-22T10:23:32Z

Interesting though; this last time it failed on the same key as previously, but at a much earlier block (1816512)

[ERROR][9/22/18 10:11:06 AM][Thread 0023][akka://NeoSystem/user/$a] The given key '0xc56f33fc6ecfcd0c225c4ab356fee59390af8560be0e930faebe74a6daff7c9b' was not present in the dictionary.
Cause: System.Collections.Generic.KeyNotFoundException: The given key '0xc56f33fc6ecfcd0c225c4ab356fee59390af8560be0e930faebe74a6daff7c9b' was not present in the dictionary.
...

jsolman · 2018-09-23T10:02:07Z

The issue was I wasn't referencing the latest neo-vm and reverting the commits that were merged in this PR (neo-project/neo-vm#52) were critical to proper operation.

Hopefully if anyone else encounters the issue, the final finding here from this thread will be useful for them.

jsolman closed this as completed Sep 21, 2018

jsolman reopened this Sep 22, 2018

jsolman closed this as completed Sep 23, 2018

jsolman mentioned this issue Sep 26, 2018

Suspicion something non-deterministic is happening in the current VM neo-project/neo-vm#53

Closed

Thacryba pushed a commit to simplitech/neo that referenced this issue Feb 17, 2020

Update toc for new added files (neo-project#391)

3d26202

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Exception Syncing MainNet Chain from Block file #391

Exception Syncing MainNet Chain from Block file #391

jsolman commented Sep 21, 2018 •

edited

jsolman commented Sep 21, 2018

jsolman commented Sep 21, 2018 •

edited

jsolman commented Sep 21, 2018 •

edited

jsolman commented Sep 21, 2018

jsolman commented Sep 21, 2018

jsolman commented Sep 22, 2018

jsolman commented Sep 22, 2018 •

edited

jsolman commented Sep 22, 2018

jsolman commented Sep 22, 2018 •

edited

jsolman commented Sep 23, 2018 •

edited

Exception Syncing MainNet Chain from Block file #391

Exception Syncing MainNet Chain from Block file #391

Comments

jsolman commented Sep 21, 2018 • edited

jsolman commented Sep 21, 2018

jsolman commented Sep 21, 2018 • edited

jsolman commented Sep 21, 2018 • edited

jsolman commented Sep 21, 2018

jsolman commented Sep 21, 2018

jsolman commented Sep 22, 2018

jsolman commented Sep 22, 2018 • edited

jsolman commented Sep 22, 2018

jsolman commented Sep 22, 2018 • edited

jsolman commented Sep 23, 2018 • edited

jsolman commented Sep 21, 2018 •

edited

jsolman commented Sep 21, 2018 •

edited

jsolman commented Sep 21, 2018 •

edited

jsolman commented Sep 22, 2018 •

edited

jsolman commented Sep 22, 2018 •

edited

jsolman commented Sep 23, 2018 •

edited