Refactor `backfilled` into specific behavior function arguments pt.1 #11396

MadLittleMods · 2021-11-19T00:43:41Z

Refactor backfilled into specific behavior function arguments pt.1. This PR does not replace all backfilled instances. Just persist_events_and_notify and all downstream calls

Using keyword-only arguments (*) to reduce the mistakes from backfilled being left as a positional argument somewhere and being interpreted wrong by our new arguments.

Part of #11300

Dev notes

persist_events_and_notify (added inhibit_push_notifications)
- persist_events and persist_event
  - _event_persist_queue

_event_persist_queue ->

_persist_event_batch (added should_calculate_state_and_forward_extrems)
- _persist_events_and_state_updates (added use_negative_stream_ordering)
  - _persist_events_txn
    - _update_room_depths_txn (added update_room_forward_stream_ordering)
    - _update_metadata_tables_txn
      - _store_room_members_txn (added inhibit_local_membership_updates)

Pull Request Checklist

Pull request is based on the develop branch
Pull request includes a changelog file. The entry should:
- Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
- Use markdown where necessary, mostly for code blocks.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
~~Pull request includes a sign off~~
Code style is correct
(run the linters)

Part of #11300

``` synapse/replication/http/federation.py:72: error: Signature of "_serialize_payload" incompatible with supertype "ReplicationEndpoint" [override] ```

MadLittleMods · 2021-11-19T07:54:05Z

synapse/replication/http/federation.py

+        should_calculate_state_and_forward_extrems,
+        use_negative_stream_ordering,
+        inhibit_local_membership_updates,
+        update_room_forward_stream_ordering,


Is there a way we can define the arguments here like the following? I can't get it to work even without the keyword-only marker.

async def _serialize_payload( store, room_id, event_and_contexts, *, inhibit_push_notifications: bool = False, should_calculate_state_and_forward_extrems: bool = True, use_negative_stream_ordering: bool = False, inhibit_local_membership_updates: bool = False, update_room_forward_stream_ordering: bool = True, ):

When I do this, I get the following error:

synapse/replication/http/federation.py:72: error: Signature of "_serialize_payload" incompatible with supertype "ReplicationEndpoint" [override]

And I can't figure out how to adjust the abstract method to be more flexible,

synapse/synapse/replication/http/_base.py

Lines 131 to 132 in 7ffddd8

@abc.abstractmethod

async def _serialize_payload(**kwargs):

I'm not entirely sure why, but it's the :bool type annotations that are upsetting it rather than the * or the default values. If I remove the :bools, it's fine.

MadLittleMods · 2021-11-19T08:25:07Z

synapse/storage/persist_events.py

@@ -35,6 +35,7 @@
 )

 import attr
+from mypy_extensions import DefaultNamedArg


Is there a better way to tackle this?

The build is failing because I used it builtins.ModuleNotFoundError: No module named 'mypy_extensions', https://github.com/matrix-org/synapse/runs/4261552290?check_suite_focus=true#step:6:739

MadLittleMods · 2021-11-19T08:27:57Z

synapse/handlers/federation_event.py

+                # Backfilled events have negative stream_ordering and happened
+                # in the past so we know that we don't need to update the
+                # stream_ordering tip for the room.
+                update_room_forward_stream_ordering=not backfilled,


In a future PR, I'll extend the behavior arguments further upstream in place of backfilled.

This PR does not replace all backfilled instances. Just persist_events_and_notify and all downstream calls

MadLittleMods · 2021-11-19T08:28:41Z

synapse/handlers/federation_event.py

            )

            if self._ephemeral_messages_enabled:
                for event in events:
                    # If there's an expiry timestamp on the event, schedule its expiry.
                    self._message_handler.maybe_schedule_expiry(event)

-            if not backfilled:  # Never notify for backfilled events
+            if not inhibit_push_notifications:  # Never notify for backfilled events


Behavioral argument replacement: inhibit_push_notifications

MadLittleMods · 2021-11-19T08:29:12Z

synapse/storage/persist_events.py

@@ -448,7 +590,7 @@ async def _persist_event_batch(
            # device lists as stale.
            potentially_left_users: Set[str] = set()

-            if not backfilled:
+            if should_calculate_state_and_forward_extrems:


Behavioral argument replacement: should_calculate_state_and_forward_extrems

MadLittleMods · 2021-11-19T08:29:34Z

synapse/storage/databases/main/events.py

@@ -159,7 +175,7 @@ async def _persist_events_and_state_updates(
        #
        # Note: Multiple instances of this function cannot be in flight at
        # the same time for the same room.
-        if backfilled:
+        if use_negative_stream_ordering:


Behavior argument replacement: use_negative_stream_ordering

Behavior argument replacement: use_negative_stream_ordering

sorry, what does this mean?

It's just the root place that I originally replaced and defines the behavior of the new argument.

MadLittleMods · 2021-11-19T08:30:20Z

synapse/storage/databases/main/events.py

@@ -1682,7 +1740,7 @@ def non_null_str_or_none(val: Any) -> Optional[str]:
            # band membership", like a remote invite or a rejection of a remote invite.
            if (
                self.is_mine_id(event.state_key)
-                and not backfilled
+                and not inhibit_local_membership_updates


Behavioral argument replacement: inhibit_local_membership_updates

MadLittleMods · 2021-11-19T08:31:07Z

synapse/storage/databases/main/events.py

                state_delta_for_room=state_delta_for_room,
                new_forward_extremeties=new_forward_extremeties,
            )
            persist_event_counter.inc(len(events_and_contexts))

-            if not backfilled:
+            # TODO: test that this actuall works
+            if stream < 0:


TODO: Test that this actually works

MadLittleMods · 2021-11-19T08:31:29Z

synapse/storage/databases/main/events.py

        """
        depth_updates: Dict[str, int] = {}
        for event, context in events_and_contexts:
            # Remove the any existing cache entries for the event_ids
            txn.call_after(self.store._invalidate_get_event_cache, event.event_id)
-            if not backfilled:
+            if update_room_forward_stream_ordering:


Behavioral argument replacement: update_room_forward_stream_ordering

MadLittleMods · 2021-11-19T08:31:38Z

synapse/handlers/federation_event.py

+        should_calculate_state_and_forward_extrems: bool = True,
+        use_negative_stream_ordering: bool = False,
+        inhibit_local_membership_updates: bool = False,
+        update_room_forward_stream_ordering: bool = True,


@richvdh How does this look? Is this on the right track?

richvdh

Thanks for putting this up, @MadLittleMods!

I haven't read all the way through the diff, because it's a bit hard to follow. Could I ask you to break it down a bit - let's start somewhere nearer the bottom of the stack: say _persist_events_and_state_updates. I think that will make it much easier to convince ourselves that the changes are correct.

Generally it seems like the right sort of thing, but I fear we might be taking it to extremes. If there are ways of grouping some of these flags together into coherent bits of functionality, I think that would be better. For example - does it ever make sense to set use_negative_stream_ordering=True and update_room_forward_stream_ordering=True together (or both False)? If not, let's get rid of update_room_forward_stream_ordering - otherwise it's just a nightmare to use the function.

richvdh · 2021-11-19T18:44:01Z

synapse/replication/http/federation.py

+        should_calculate_state_and_forward_extrems,
+        use_negative_stream_ordering,
+        inhibit_local_membership_updates,
+        update_room_forward_stream_ordering,


I'm not entirely sure why, but it's the :bool type annotations that are upsetting it rather than the * or the default values. If I remove the :bools, it's fine.

richvdh · 2021-11-19T18:47:04Z

synapse/handlers/federation_event.py

+                # Backfilled events have negative stream_ordering and happened
+                # in the past so we know that we don't need to update the
+                # stream_ordering tip for the room.
+                update_room_forward_stream_ordering=not backfilled,


shouldn't update_room_forward_stream_ordering be implied by use_negative_stream_ordering ?

richvdh · 2021-11-19T18:47:50Z

synapse/handlers/federation_event.py

+                # come down incremental `/sync`
+                use_negative_stream_ordering=backfilled,
+                # Backfilled events do not affect the current local state
+                inhibit_local_membership_updates=backfilled,


would it be sensible to have this implied by should_calculate_state_and_forward_extrems ?

I understand the mental leap where updating the state in the room also encompasses member state events.

My first impression was that it wasn't very obvious but it makes sense ⏩

richvdh · 2021-11-19T18:48:17Z

synapse/handlers/federation_event.py

+                # there is no need to update the forward extrems because we
+                # already know this event happened in the past if it was
+                # backfilled.
+                should_calculate_state_and_forward_extrems=not backfilled,


the should_ prefix seems a bit redundant here.

richvdh · 2021-11-19T18:51:48Z

synapse/handlers/federation_event.py

@@ -1850,7 +1895,7 @@ async def persist_events_and_notify(
                    store=self._store,
                    room_id=room_id,
                    event_and_contexts=batch,
-                    backfilled=backfilled,
+                    inhibit_push_notifications=inhibit_push_notifications,


doesn't this need to pass through all the keyword args?

Good catch 👍 I think so. Should match _serialize_payload.

It's a loose connection and I was having trouble typing it correctly, #11396 (comment), to have the linter yell at me for this.

richvdh · 2021-11-19T18:54:33Z

synapse/storage/databases/main/events.py

@@ -159,7 +175,7 @@ async def _persist_events_and_state_updates(
        #
        # Note: Multiple instances of this function cannot be in flight at
        # the same time for the same room.
-        if backfilled:
+        if use_negative_stream_ordering:


Behavior argument replacement: use_negative_stream_ordering

sorry, what does this mean?

MadLittleMods · 2021-11-24T05:59:29Z

Thanks for the review @richvdh, great points! I've made a smaller PoC in #11417

MadLittleMods requested a review from a team as a code owner November 19, 2021 00:43

MadLittleMods removed the request for review from a team November 19, 2021 00:43

MadLittleMods marked this pull request as draft November 19, 2021 00:43

MadLittleMods force-pushed the madlittlemods/11300-refactor-backfilled-pt1 branch from 35e036a to 96d6b5c Compare November 19, 2021 00:47

MadLittleMods changed the title ~~Draft: Refactor backfilled behavior into specific behavior function parameters~~ Draft: Refactor backfilled behavior into specific behavior function arguments Nov 19, 2021

MadLittleMods force-pushed the madlittlemods/11300-refactor-backfilled-pt1 branch from 96d6b5c to f5e3731 Compare November 19, 2021 01:37

Refactor backfilled behavior into specific function parameters

d203d22

Part of #11300

MadLittleMods force-pushed the madlittlemods/11300-refactor-backfilled-pt1 branch from f5e3731 to d203d22 Compare November 19, 2021 01:37

MadLittleMods added 2 commits November 18, 2021 20:09

Pipe arguments across the function stack

b8c60b9

Fix event queue

d44fdcc

MadLittleMods changed the title ~~Draft: Refactor backfilled behavior into specific behavior function arguments~~ Draft: Refactor backfilled behavior into specific behavior function arguments pt.1 Nov 19, 2021

MadLittleMods changed the title ~~Draft: Refactor backfilled behavior into specific behavior function arguments pt.1~~ Draft: Refactor backfilled arguments into specific behavior function arguments pt.1 Nov 19, 2021

MadLittleMods changed the title ~~Draft: Refactor backfilled arguments into specific behavior function arguments pt.1~~ Draft: Refactor backfilled into specific behavior function arguments pt.1 Nov 19, 2021

MadLittleMods added 2 commits November 19, 2021 00:43

Add changelog

ec82407

Fix incompatible type

56ea69f

``` synapse/replication/http/federation.py:72: error: Signature of "_serialize_payload" incompatible with supertype "ReplicationEndpoint" [override] ```

MadLittleMods commented Nov 19, 2021

View reviewed changes

Update comments docs to explain the why

f01d506

MadLittleMods commented Nov 19, 2021

View reviewed changes

MadLittleMods changed the title ~~Draft: Refactor backfilled into specific behavior function arguments pt.1~~ Refactor backfilled into specific behavior function arguments pt.1 Nov 19, 2021

MadLittleMods commented Nov 19, 2021

View reviewed changes

MadLittleMods requested a review from richvdh November 19, 2021 08:31

richvdh suggested changes Nov 19, 2021

View reviewed changes

MadLittleMods mentioned this pull request Nov 24, 2021

Refactor backfilled into specific behavior function arguments (_persist_events_and_state_updates) #11417

Merged

4 tasks

MadLittleMods added the T-Task Refactoring, removal, replacement, enabling or disabling functionality, other engineering tasks. label Nov 29, 2021

MadLittleMods closed this Jun 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor `backfilled` into specific behavior function arguments pt.1 #11396

Refactor `backfilled` into specific behavior function arguments pt.1 #11396

MadLittleMods commented Nov 19, 2021 •

edited

MadLittleMods Nov 19, 2021

richvdh Nov 19, 2021

MadLittleMods Nov 19, 2021

MadLittleMods Nov 19, 2021

MadLittleMods Nov 19, 2021

MadLittleMods Nov 19, 2021

MadLittleMods Nov 19, 2021

richvdh Nov 19, 2021

MadLittleMods Nov 24, 2021 •

edited

MadLittleMods Nov 19, 2021

MadLittleMods Nov 19, 2021

MadLittleMods Nov 19, 2021 •

edited

MadLittleMods Nov 19, 2021

richvdh left a comment

richvdh Nov 19, 2021

richvdh Nov 19, 2021

richvdh Nov 19, 2021

MadLittleMods Nov 24, 2021 •

edited

richvdh Nov 19, 2021

richvdh Nov 19, 2021

MadLittleMods Nov 24, 2021 •

edited

richvdh Nov 19, 2021

MadLittleMods commented Nov 24, 2021

Refactor backfilled into specific behavior function arguments pt.1 #11396

Refactor backfilled into specific behavior function arguments pt.1 #11396

Conversation

MadLittleMods commented Nov 19, 2021 • edited

Dev notes

Pull Request Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MadLittleMods Nov 24, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MadLittleMods Nov 19, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

richvdh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MadLittleMods Nov 24, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MadLittleMods Nov 24, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MadLittleMods commented Nov 24, 2021

Refactor `backfilled` into specific behavior function arguments pt.1 #11396

Refactor `backfilled` into specific behavior function arguments pt.1 #11396

MadLittleMods commented Nov 19, 2021 •

edited

MadLittleMods Nov 24, 2021 •

edited

MadLittleMods Nov 19, 2021 •

edited

MadLittleMods Nov 24, 2021 •

edited

MadLittleMods Nov 24, 2021 •

edited