Batch database queries in RdePipeline

[gbrodman]

Currently, we process (repoId, revisionId) pairs for DomainHistory and HostHistory individually -- they may be farmed out to worker nodes in parallel, but each EppResource uses a separate transaction and a separate read, which doesn't scale well when there are lots of domains/hosts. So as a result, we should batch them up so we can load (by default) 500 per transaction at a time.

We don't want to batch-load the domains/hosts at the same time that we retrieve the most recent history entry for each type -- this would mean passing relatively large objects across pipeline steps. Instead, we keep passing the KV<String, Long> and batch retrievals.

Self-scan D.2 number 5

---

This change is 

[CydeWeys]

+a:@weiminyu to have a look as well.

@CydeWeys made 1 comment.
Reviewable status: 0 of 3 files reviewed, all discussions resolved (waiting on weiminyu).

[CydeWeys]

This is going to require some seriously careful testing on sandbox.

@CydeWeys made 1 comment.
Reviewable status: 0 of 3 files reviewed, all discussions resolved (waiting on weiminyu).

[weiminyu]

@weiminyu reviewed 3 files and all commit messages.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on gbrodman).

[gbrodman]

OK so I tested this on alpha a ton.

On alpha, the base repo actually fails to run at all because a toooooon of domains reference the same host -- so we'd try to load the host a million times. The only thing I added against the base repository when testing the baseline was the "Deduplicate referenced hosts for grouping" step, line 463 here.

This doesn't really appear to run any more quickly, at least on alpha. This could be because of the skew in alpha data, but more likely it's because batching means that we need to wait until a batch is ready before sending it off to the next step, versus in the old system we could just send each domain/host off immediately to the next step when it was done. This is still worth doing though, because it'll decrease the load on the database massively.

#3206 reduces the time of the initial query by a large chunk too, from like two minutes down to 30 seconds.

The RDE output for a given day is the same in the old version and this new version

[gbrodman]

going to make a few more changes + add more tests, no need to review yet

[gbrodman]

OK PTAL -- this looks like a lot of code but the main things it actually changes are

1. de-dupes the hosts around line 468 so we don't slam a million copies of the same host through one hot key
2. shards and batches history entry loads in the three places they're done (domains, hosts, domains+hosts for subordinate hosts)
3. updates the initial SQL query to move the TLD check into the inner subquery, meaning we don't need to do the group by / compute modification times for non-real TLDs

[weiminyu]

@weiminyu reviewed 4 files and all commit messages.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on gbrodman).