HeaderNotFound during sync

[carver]

I can only see two options for how this happened, either one is a bug, and I can't see how either might be possible, yet:

1. Header sync returned a header with a gap: the parent was not yet in the DB
2. Header sync returned an old header that was already sent (which is valid), but the OrderedTaskPreparation object had already pruned it's parent before its parent had been written to the DB. (that part shouldn't happen)

Note that #347 now has a failsafe restart in case the bug reappears so trinity shouldn't crash, but it's still a bug that needs to be hunted down.

---

This is from 7b9e956fedf936c63ed1413ebec7face6962fe5c a recent version of #347.

HeaderNotFound: https://gist.github.com/cburgdorf/3d74f0b9a1c56ee8c9bda108ac7fcf51

Originally posted by @cburgdorf in #347 (comment)

[carver]

Ok, reproduced this one. Perhaps unsurprisingly it's maybe not either of options 1 or 2: there seems to be no gap and it's not an old header. This failed on header #4007799 but #4007798 was received (presumably right before). Continuing to add logging to figure out WTF is going on.

[carver]

Got it: the missing dependency check was assuming that headers were coming in order. It's possible to get a skip back to a really old header in the middle right now. Some options:

1. Return which headers have missing dependencies in the exception
2. Require that all headers sent in a batch by the header syncer be consecutive. Any non-consecutive headers would start a new batch

2 seems easier to implement and reason about. 1 is tricky to do right, because you can't gather all the missing dependencies up front, and you could have several in the same batch. So 2 sounds like the way to go, unless a 3rd option pops up.