Crash on task pruning

[cburgdorf]

What is wrong?

There's a new crash on master related to pruning that happened fairly quickly after node launch.

Traceback (most recent call last):
  File "/home/ubuntu/trinity/p2p/service.py", line 139, in _run_task_wrapper
    await awaitable
  File "/home/ubuntu/trinity/p2p/service.py", line 158, in _run_daemon_task_wrapper
    await awaitable
  File "/home/ubuntu/trinity/trinity/sync/full/chain.py", line 458, in _launch_prerequisite_tasks
    async for headers in self.wait_iter(get_headers_coro):
  File "/home/ubuntu/trinity/p2p/cancellable.py", line 66, in wait_iter
    timeout=timeout,
  File "/home/ubuntu/trinity/p2p/cancellable.py", line 20, in wait
    return await self.wait_first(awaitable, token=token, timeout=timeout)
  File "/home/ubuntu/trinity/p2p/cancellable.py", line 42, in wait_first
    return await token_chain.cancellable_wait(*awaitables, timeout=timeout)
  File "/home/ubuntu/trinity/venv/lib/python3.6/site-packages/cancel_token/token.py", line 152, in cancellable_wait
    return done.pop().result()
  File "/home/ubuntu/trinity/trinity/sync/common/headers.py", line 756, in new_sync_headers
    next_header_batch = await self.wait(self._stitcher.ready_tasks(max_batch_size))
  File "/home/ubuntu/trinity/p2p/cancellable.py", line 20, in wait
    return await self.wait_first(awaitable, token=token, timeout=timeout)
  File "/home/ubuntu/trinity/p2p/cancellable.py", line 42, in wait_first
    return await token_chain.cancellable_wait(*awaitables, timeout=timeout)
  File "/home/ubuntu/trinity/venv/lib/python3.6/site-packages/cancel_token/token.py", line 152, in cancellable_wait
    return done.pop().result()
  File "/home/ubuntu/trinity/trinity/_utils/datastructures.py", line 590, in ready_tasks
    self._prune_finished(task_id)
  File "/home/ubuntu/trinity/trinity/_utils/datastructures.py", line 665, in _prune_finished
    self._prune(root_task_id)
  File "/home/ubuntu/trinity/trinity/_utils/datastructures.py", line 670, in _prune
    del self._dependencies[prune_task_id]
KeyError: b'\x97\x82c(\xda\xf2\xddo\xec\xe9\x0f\xb4\xca\xa1p\xf7}\x15#\xe6\x8c/?\xc1\xf5\xc8\xdb\x8d\x9e\x05\xb7I'

https://gist.github.com/cburgdorf/57a61c9bb64f294a41b55f7124912bb0

How can it be fixed

🦄 🌈 🔪

[cburgdorf]

cc @carver

[carver]

Huh, interesting. I've been running for a while without seeing that one. Taking a look...

Edit: I did get it to reproduce after 1.5 hrs on master. I just restarted my node on the tip of #347 to see if it happens again.

[carver]

Hah, well I have no idea how you got so "lucky" but apparently if you reset sync twice, after pruning twice, and it happens to start from exactly one header further forward from the last time you reset sync, then it can trigger this case. Just reproduced in a test. Off to fix and open PR...

Edit: I only had a chance to run it for another 2 hours before I had to run, but haven't reproduced it yet!

[cburgdorf]

Haha, I've got the magic hands to screw things up! Joking aside, after I experienced this crash the first time, the second time that I started Trinity it ran for 8 hours or so. I have now restarted it with #347.