Parallel pull from Share

[mitchellwrosen]

• Depends on Use Values literal in elaborateHashes #3155

Overview

This PR implements a concurrent pull procedure involving a few different threads:

• One thread that inserts entities fetched from Share into sqlite. Because writing to sqlite requires a lock, we can't do better than that.
• One thread that elaborates subsets of those hashes and adds the elaborated hashes to the work queue (the set of hashes we need to download).
• One thread to coordinate pulling batches of work off of the work queue and spawning one-shot workers, and deciding when there's no more work left to do.

[ChrisPenner]

This works really nicely 👍🏼 Good work on this lib

[mitchellwrosen]

Thanks!

[ChrisPenner]

Is there a benefit to the (join . atomically) do pattern here over something like:

forever do
  mayNewTempEntities <- atomically $ do ...
  case mayNewTempEntities of
    Nothing -> ...
    Just _ -> ...

?

I find interleaving STM and IO makes it tougher to see what's being accomplished transactionally 😅

[mitchellwrosen]

Sure, we can switch it to not do the join . atomically thing. I think instead of a Maybe (Set Hash) (or w/e) I'll go with a one-off data type, so it's more clear what going down one branch versus another means.