Some sync crash fixups

[carver]

What was wrong?

Two crashes during sync, both during peer switchover:

1. The new peer reinserts a header into the queue that the fast-syncer is already trying to download a body for.
2. The full syncer catches up just after a new peer started syncing. It causes havoc to have the peer start skipping some headers midstream. Better to let it just insert the duplicates and let the solution to # 1 handle the situation.

How was it fixed?

1. Fixed with a new DuplicateTasks exception that is caught, so the relevant headers can be skipped.
2. Only check if headers are already in the database on the very first run

Plus some logging improvements.

There's another typing tangent that I wasted a lot of time on. Grumble. Any suggestions for making it work? (where the type: ignore is)

Cute Animal Picture

[pipermerriam]

Can you add a line up above the try:

exc: DuplicateTask[BlockHeader]
try:
    ...
except DuplicateTasks as exc:
   ...

I think that's how you tell mypy ahead of time what type something is.

[pipermerriam]

Alternate idea:

duplicates = cast(Tuple[BlockHeader, ...], exc.duplicates)
self.header_queue.complete(batch_id, duplicates)

Less elegant/type-safe...

[carver]

Pre-defining the exception type still complained... going with the duplicates cast.

[pipermerriam]

I rabbit hole'd this a while ago. There's a solution that works for simple types but not this one but which has a runtime cost.

if isinstance(v, SomeType):
    do_SomeType_specific_thing(v)

mypy correctly understands that within the if context v must be SomeType. However, there's no way to do the equivalent of if isinstance(v, Tuple[SomeType, ...]) that I could find. 😢

[pipermerriam]

No review requested yet, but I did a full pass on this and it looks 👍

[carver]

Yeah, marked WIP because I had no idea what the p2p.discovery failure is. I spent some time digging into it. I can't reproduce locally. I am having a hard time even guessing what might have happened:

• it's not a timeout, the test didn't run that long
• it didn't get nonsense data, that would have shown up in a warning log

So far, I think maybe both alice and bob binded to the same UDP port, and alice responded to her own request for nodes with an empty list. Still not sure why that happened in 0.02s when the timeout is 0.9s. (the wait for response is supposed to hang until it gets the desired number of nodes, no matter the intermediate responses)