Draft: Optimize MSC2716 /batch_send endpoint v1

[MadLittleMods]

Optimize MSC2716 /batch_send endpoint v1

1. Cache event creation info and context
   - This gets us from ~1.3s -> ~0.8s per 100 event batch.
   - Cache context calculations that will be the same for every event type in the batch.
   - Cache auth_event_ids calculations that will be the same for every event with the same sender and type in the batch.
   - Only store_state_group_id_for_event_id for the events created by create_event (the fresh events not cached) because the other ones will just derive from that and will store their state_group_id later when everything is persisted.

Todo

• Optimize handle_new_client_event to maybe bulk insert
• Maybe cache create_requester_for_user_id_from_app_service although maybe less of a worry now that we've optimized the event creation stuff

Dev notes

1. Register all the users you will be testing with, POST http://localhost:8008/_matrix/client/r0/register with {"type": "m.login.application_service", "username": "maria-01234"}
2. Create a new room with the experimental room version POST http://localhost:8008/_matrix/client/v3/createRoom with { "preset": "public_chat", "name": "profile-msc2716-v2", "room_version": "org.matrix.msc2716v3" }
3. POST http://localhost:8008/_matrix/client/unstable/org.matrix.msc2716/rooms/!NicrUzWIjuYwVfVSbh:my.synapse.server/batch_send?prev_event_id=%24m9fA9pPmnjvAHJRmz-dXol_9J1KrMs3LTiHOBoC2SLQ&batch_id=iythPUbb with the body generated from the script

const insertionTimestamp = new Date('2020-01-01').getTime()

const mxids = [
    "@maria-01234:my.synapse.server",
    "@gabriel-01234:my.synapse.server"
];


function generatePayload() {
    const events = []
    for(let i = 0; i < 10; i++) {
        events[i] = {
          "content": {
            "body": `Historical ${i} (batch=0)`,
            "msgtype": "m.text",
            "org.matrix.msc2716.historical": true
          },
          "origin_server_ts": insertionTimestamp + (i * 1000),
          "sender": mxids[i % mxids.length],
          "type": "m.room.message"
        };
    }

    const result = {
      "state_events_at_start": mxids.map((mxid) => {
          return {
          "content": {
            "displayname": `some-display-name-for-${mxid}`,
            "membership": "join"
          },
          "origin_server_ts": insertionTimestamp,
          "sender": mxid,
          "state_key": mxid,
          "type": "m.room.member"
        };
      }),
      "events": events
    };

    return result;
}

JSON.stringify(generatePayload(), null, 2)

Testing

SYNAPSE_POSTGRES=1 SYNAPSE_TEST_LOG_LEVEL=INFO python -m twisted.trial tests.rest.client.test_room_batch

COMPLEMENT_ALWAYS_PRINT_SERVER_LOGS=1 COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh TestImportHistoricalMessages/parallel/Federation/Backfill_still_works_after_many_batches_are_imported

py-spy profiling and flame graphs to get time spent

1. pip install py-spy
2. Use ps aux | grep synapse.app.homeserver to get the pid of the Synapse server
3. Replace the pid in the following command with what you found above: sudo py-spy record --pid 70177 --rate 300 --format speedscope -o ~/Downloads/msc2716-profile5.speedscope.json

Open in https://www.speedscope.app/

Other profilers

• https://github.com/emeryberger/scalene

Optimization check-ins

On develop (2022-01-25),

34.9s for 3k messages from 2 users
36.5s
34.2s


68.8s for 6k messages from 2 users
76.0s
73.0s
80.8s

After optimizations in #11778 (2022-01-24)

28.4s for 3k messages from 2 users
27.5s
27.0s
50s (with profiler)


56.1s for 6k messages from 2 users
54.3s
55.6s


124.5s for 12k messages from 2 users
113.8s
110.9s
138.0s

---

Matthew 2125 event benchmark between 2 users

22.5s
20.7s
20.7s
20.8s
19.4s
19.5s

with sqlite:

17.3s
18.6s
21.4s
23.1s
22.6s
22.9s

Other

# Normally the state_group stored when persisting the
# event but we have to pre-emptively do it here because we don't want
# the persisting to re-calculate it for us since our goal is to share
# the same state_group across the entire batch. We first created all the
# events above and share contexts as a calculation optimization but they
# also reference the same state_group.
await self.store.store_state_group_id_for_event_ids(
    context._state_group,
    [event.event_id for (event, context) in events_to_persist],
)

async def store_state_group_id_for_event_ids(
    self, state_group_id: int, event_ids: List[str]
) -> None:
    await self.db_pool.simple_upsert_many(
        table="event_to_state_groups",
        key_names=("event_id",),
        key_values=[(event_id,) for event_id in event_ids],
        value_names=("state_group",),
        value_values=[(state_group_id,) for _ in event_ids],
        desc="store_state_group_id_for_event_ids_map",
    )

Pull Request Checklist

• Pull request is based on the develop branch
• Pull request includes a changelog file. The entry should:
  • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
  • Use markdown where necessary, mostly for code blocks.
  • End with either a period (.) or an exclamation mark (!).
  • Start with a capital letter.
  • Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
• Pull request includes a sign off
• Code style is correct
  (run the linters)

[MadLittleMods]

TODO: Revert before merge