Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Use a chain cover index to efficiently calculate auth chain difference #8868

Merged
merged 48 commits into from
Jan 11, 2021
Merged
Show file tree
Hide file tree
Changes from 44 commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
49e888d
Change alg
erikjohnston Dec 1, 2020
8c760ff
Calculate chain ID/seq no on event insertion
erikjohnston Dec 2, 2020
85348e1
Add some docs about the chain cover
erikjohnston Dec 2, 2020
02d1198
Handle old rooms
erikjohnston Dec 3, 2020
61ab47e
Fix schema for sqlite
erikjohnston Dec 3, 2020
6141825
Fix up _get_auth_chain_difference_using_chains_txn
erikjohnston Dec 3, 2020
c7e2ce5
Newsfile
erikjohnston Dec 3, 2020
66e779d
Add type
erikjohnston Dec 3, 2020
cf2243f
Fixup
erikjohnston Dec 3, 2020
bd30c9e
Fix take1
erikjohnston Dec 4, 2020
55f03b9
Fixup
erikjohnston Dec 4, 2020
3e98fb7
More fixups
erikjohnston Dec 4, 2020
9087033
Newsfile
erikjohnston Dec 4, 2020
21b3ef0
Test both new and old methods
erikjohnston Dec 4, 2020
fdaf4da
Note
erikjohnston Dec 4, 2020
7f5ac13
isort
erikjohnston Dec 4, 2020
afb7f80
Don't add links where start and end chain are the same
erikjohnston Dec 7, 2020
dec1f74
Have exists_path_from handle same chain case correctly
erikjohnston Dec 7, 2020
9279940
Add some tests
erikjohnston Dec 7, 2020
6a74e21
Fix unit tests on postgres
erikjohnston Dec 7, 2020
654eff1
Add missing 'auth'
erikjohnston Dec 7, 2020
dbecefd
Fixup typing for execute_values
erikjohnston Dec 8, 2020
123b431
Rename _get_auth_chain_difference_using_chains_txn and add comment
erikjohnston Dec 8, 2020
883e922
Add some definitions
erikjohnston Dec 8, 2020
988f25a
Fixup link confusion
erikjohnston Dec 8, 2020
08ec78b
Make para less dense (hopefully)
erikjohnston Dec 8, 2020
024c802
Add note about auth chain
erikjohnston Dec 8, 2020
4cc769f
Be explicit
erikjohnston Dec 8, 2020
7d75efb
rm variant
erikjohnston Dec 8, 2020
92b5e4b
Add note about current algo
erikjohnston Dec 8, 2020
a9552c2
Update docs/auth_chain_difference_algorithm.md
erikjohnston Dec 8, 2020
7cc6d7e
Fix up _LinkMap
erikjohnston Dec 8, 2020
5fa05f2
Fix up event_chain tests
erikjohnston Dec 8, 2020
8dac80c
Merge remote-tracking branch 'origin/develop' into erikj/auth_chains_…
erikjohnston Dec 9, 2020
e3d0be4
Make sorted_topologically stable and add tests
erikjohnston Dec 9, 2020
cdb88c2
Make _LinkMap use tuples
erikjohnston Dec 9, 2020
0f91c86
Review comments
erikjohnston Dec 9, 2020
888450a
Fix typo
erikjohnston Dec 9, 2020
c9422b6
Handle rooms the server used to be in correctly.
erikjohnston Jan 5, 2021
c8758af
Handle case where we don't have chain info for an event
erikjohnston Jan 6, 2021
b2ac553
Merge remote-tracking branch 'origin/develop' into erikj/auth_chains_…
erikjohnston Jan 6, 2021
d96264d
Merge remote-tracking branch 'origin/develop' into erikj/auth_chains_…
erikjohnston Jan 6, 2021
d64f5f8
Typo
erikjohnston Jan 11, 2021
6071eff
Split out a has_auth_chain_index
erikjohnston Jan 11, 2021
368d3b8
Update docstring
erikjohnston Jan 11, 2021
bea2c47
Move to schema 59
erikjohnston Jan 11, 2021
03dd636
Merge remote-tracking branch 'origin/develop' into erikj/auth_chains_…
erikjohnston Jan 11, 2021
8c1e32c
Fix tests after merge from develop
erikjohnston Jan 6, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/8868.misc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Improve efficiency of large state resolutions for new rooms.
32 changes: 32 additions & 0 deletions docs/auth_chain_diff.dot
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
digraph auth {
nodesep=0.5;
rankdir="RL";

C [label="Create (1,1)"];

BJ [label="Bob's Join (2,1)", color=red];
BJ2 [label="Bob's Join (2,2)", color=red];
BJ2 -> BJ [color=red, dir=none];

subgraph cluster_foo {
A1 [label="Alice's invite (4,1)", color=blue];
A2 [label="Alice's Join (4,2)", color=blue];
A3 [label="Alice's Join (4,3)", color=blue];
A3 -> A2 -> A1 [color=blue, dir=none];
color=none;
}

PL1 [label="Power Level (3,1)", color=darkgreen];
PL2 [label="Power Level (3,2)", color=darkgreen];
PL2 -> PL1 [color=darkgreen, dir=none];

{rank = same; C; BJ; PL1; A1;}

A1 -> C [color=grey];
A1 -> BJ [color=grey];
PL1 -> C [color=grey];
BJ2 -> PL1 [penwidth=2];

A3 -> PL2 [penwidth=2];
A1 -> PL1 -> BJ -> C [penwidth=2];
}
Binary file added docs/auth_chain_diff.dot.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
108 changes: 108 additions & 0 deletions docs/auth_chain_difference_algorithm.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
# Auth Chain Difference Algorithm

The auth chain difference algorithm is used by V2 state resolution, where a
naive implementation can be a significant source of CPU and DB usage.
erikjohnston marked this conversation as resolved.
Show resolved Hide resolved

### Definitions

A *state set* is a set of state events; e.g. the input of a state resolution
algorithm is a collection of state sets.

The *auth chain* of a set of events are all the events' auth events and *their*
auth events, recursively (i.e. the events reachable by walking the graph induced
by an event's auth events links).

The *auth chain difference* of a collection of state sets is the union minus the
intersection of the sets of auth chains corresponding to the state sets, i.e an
event is in the auth chain difference if it is reachable by walking the auth
event graph from at least one of the state sets but not from *all* of the state
sets.

## Breadth First Walk Algorithm

A way of calculating the auth chain difference without calculating the full auth
chains for each state set is to do a parallel breadth first walk (ordered by
depth) of each state set's auth chain. By tracking which events are reachable
from each state set we can finish early if every pending event is reachable from
every state set.

This can work well for state sets that have a small auth chain difference, but
can be very inefficient for larger differences. However, this algorithm is still
used if we don't have a chain cover index for the room (e.g. because we're in
the process of indexing it).

## Chain Cover Index

Synapse computes auth chain differences by pre-computing a "chain cover" index
for the auth chain in a room, allowing efficient reachability queries like "is
event A in the auth chain of event B". This is done by assigning every event a
*chain ID* and *sequence number* (e.g. `(5,3)`), and having a map of *links*
between chains (e.g. `(5,3) -> (2,4)`) such that A is reachable by B (i.e. `A`
is in the auth chain of `B`) if and only if either:

1. A and B have the same chain ID and `A`'s sequence number is less than `B`'s
sequence number; or
2. there is a link `L` between `B`'s chain ID and `A`'s chain ID such that
`L.start_seq_no` <= `B.seq_no` and `A.seq_no` <= `L.end_seq_no`.

There are actually two potential implementations, one where we store links from
each chain to every other reachable chain (the transitive closure of the links
graph), and one where we remove redundant links (the transitive reduction of the
links graph) e.g. if we have chains `C3 -> C2 -> C1` then the link `C3 -> C1`
would not be stored. Synapse uses the former implementations so that it doesn't
need to recurse to test reachability between chains.

### Example

An example auth graph would look like the following, where chains have been
formed based on type/state_key and are denoted by colour and are labelled with
`(chain ID, sequence number)`. Links are denoted by the arrows (links in grey
are those that would be remove in the second implementation described above).

![Example](auth_chain_diff.dot.png)

Note that we don't include all links between events and their auth events, as
most of those links would be redundant. For example, all events point to the
create event, but each chain only needs the one link from it's base to the
create event.

## Using the Index

This index can be used to calculate the auth chain difference of the state sets
by looking at the chain ID and sequence numbers reachable from each state set:

1. For every state set lookup the chain ID/sequence numbers of each state event
2. Use the index to find all chains and the maximum sequence number reachable
from each state set.
erikjohnston marked this conversation as resolved.
Show resolved Hide resolved
3. The auth chain difference is then all events in each chain that have sequence
numbers between the maximum sequence number reachable from *any* state set and
the minimum reachable by *all* state sets (if any).

Note that steps 2 is effectively calculating the auth chain for each state set
(in terms of chain IDs and sequence numbers), and step 3 is calculating the
difference between the union and intersection of the auth chains.

### Worked Example

For example, given the above graph, we can calculate the difference between
state sets consisting of:

1. `S1`: Alice's invite `(4,1)` and Bob's second join `(2,2)`; and
2. `S2`: Alice's second join `(4,3)` and Bob's first join `(2,1)`.

Using the index we see that the following auth chains are reachable from each
state set:

1. `S1`: `(1,1)`, `(2,2)`, `(3,1)` & `(4,1)`
2. `S2`: `(1,1)`, `(2,1)`, `(3,2)` & `(4,3)`

And so, for each the ranges that are in the auth chain difference:
1. Chain 1: None, (since everything can reach the create event).
2. Chain 2: The range `(1, 2]` (i.e. just `2`), as `1` is reachable by all state
sets and the maximum reachable is `2` (corresponding to Bob's second join).
3. Chain 3: Similarly the range `(1, 2]` (corresponding to the second power
level).
4. Chain 4: The range `(1, 3]` (corresponding to both of Alice's joins).

So the final result is: Bob's second join `(2,2)`, the second power level
`(3,2)` and both of Alice's joins `(4,2)` & `(4,3)`.
22 changes: 18 additions & 4 deletions synapse/storage/database.py
Original file line number Diff line number Diff line change
Expand Up @@ -179,6 +179,9 @@ def __getattr__(self, name):
_CallbackListEntry = Tuple["Callable[..., None]", Iterable[Any], Dict[str, Any]]


R = TypeVar("R")


class LoggingTransaction:
"""An object that almost-transparently proxies for the 'txn' object
passed to the constructor. Adds logging and metrics to the .execute()
Expand Down Expand Up @@ -266,6 +269,20 @@ def execute_batch(self, sql: str, args: Iterable[Iterable[Any]]) -> None:
for val in args:
self.execute(sql, val)

def execute_values(self, sql: str, *args: Any) -> List[Tuple]:
"""Corresponds to psycopg2.extras.execute_values. Only available when
using postgres.

Always sets fetch=True when caling `execute_values`, so will return the
results.
"""
assert isinstance(self.database_engine, PostgresEngine)
from psycopg2.extras import execute_values # type: ignore

return self._do_execute(
erikjohnston marked this conversation as resolved.
Show resolved Hide resolved
lambda *x: execute_values(self.txn, *x, fetch=True), sql, *args
)

def execute(self, sql: str, *args: Any) -> None:
self._do_execute(self.txn.execute, sql, *args)

Expand All @@ -276,7 +293,7 @@ def _make_sql_one_line(self, sql: str) -> str:
"Strip newlines out of SQL so that the loggers in the DB are on one line"
return " ".join(line.strip() for line in sql.splitlines() if line.strip())

def _do_execute(self, func, sql: str, *args: Any) -> None:
def _do_execute(self, func: Callable[..., R], sql: str, *args: Any) -> R:
sql = self._make_sql_one_line(sql)

# TODO(paul): Maybe use 'info' and 'debug' for values?
Expand Down Expand Up @@ -347,9 +364,6 @@ def interval(self, interval_duration_secs: float, limit: int = 3) -> str:
return top_n_counters


R = TypeVar("R")


class DatabasePool:
"""Wraps a single physical database and connection pool.

Expand Down
Loading