Optimize backfill receiving to have less missing prev_event thrashing (v2)

[MadLittleMods]

Addressing:

• Optimize synapse/handlers/federation_event.py#L545(backfill receiving) to be more efficient. Calculate the stream_ordering from newest -> oldest (in the correct order) and persist in the oldest -> newest to get the least missing prev_event fetch thrashing
  • Spawned from Fix historical messages backfilling in random order on remote homeservers (MSC2716) #11114 (comment)
  • Why aren't we sorting topologically when receiving backfill events? See Fix historical messages backfilling in random order on remote homeservers (MSC2716) #11114 (comment)

-- #10737

So Synapse is fast enough to merge this MSC2716 Complement test for importing many batches, matrix-org/complement#214 (comment)

---

Complement tests: matrix-org/complement#214

---

Spawned from scratch changes in, #13864

Dev notes

pseudo_events = [
    { "event": 'eventBefore1', "depth": 1 },
    { "event": 'eventBefore2', "depth": 2 },
    { "event": 'Historical message 1', "depth": 3 },
    { "event": 'Historical message 2', "depth": 3 },
    { "event": 'Historical message 3', "depth": 3 },
    { "event": 'eventAfter1', "depth": 3 },
    { "event": 'eventAfter2', "depth": 4 }
]

pseudo_events = [
    { "event": 'eventAfter2', "depth": 4 },
    { "event": 'eventAfter1', "depth": 3 },
    { "event": 'Historical message 3', "depth": 3 },
    { "event": 'Historical message 2', "depth": 3 },
    { "event": 'Historical message 1', "depth": 3 },
    { "event": 'eventBefore2', "depth": 2 },
    { "event": 'eventBefore1', "depth": 1 },
]


sorted_events = [u[1] for u in sorted(
    enumerate(pseudo_events), key=lambda x: (-x[1].get("depth"), x[0])
)]

print(json.dumps(sorted_events, indent=4))

Pull Request Checklist

• Pull request is based on the develop branch
• Pull request includes a changelog file. The entry should:
  • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
  • Use markdown where necessary, mostly for code blocks.
  • End with either a period (.) or an exclamation mark (!).
  • Start with a capital letter.
  • Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
• Pull request includes a sign off
• Code style is correct
  (run the linters)

[MadLittleMods]

Can we rely on the worker instance that backfills events to be a writer instance? I don't think so. The event persister is different than the workers that ingest new events.

How do I do this correctly?