WIP: Implement archived rooms (include_leave filter on /sync)

[MadLittleMods]

Implement archived rooms (include_leave filter on /sync)

http://localhost:8008/_matrix/client/r0/sync?filter={ "room": { "include_leave": true, "timeline": { "limit": 5 } } }

Fix #1323

Dev notes

• GET /_matrix/client/r0/sync (see filter query parameter), https://matrix.org/docs/spec/client_server/r0.6.1#get-matrix-client-r0-sync
• POST /_matrix/client/r0/user/{userId}/filter (RoomFilter has include_leave), https://matrix.org/docs/spec/client_server/r0.6.1#post-matrix-client-r0-user-userid-filter
• syncapi/routing/routing.go#L45-L47
• syncapi/sync/requestpool.go#L127-L225
• Where the filter query parameter is handled: syncapi/sync/request.go#L67-L89
• Where we need to pass the filter from the request: syncapi/storage/shared/syncserver.go#L803
• "LeaveResponse represents a /sync response for a room which is under the 'leave' key.": syncapi/types/types.go#L459-L469 -> LeaveResponse
• https://github.com/matrix-org/sytest/blob/develop/docs/dendrite-setup.md#how-to-set-up-sytest-for-dendrite
• https://github.com/matrix-org/dendrite/blob/master/docs/sytest.md
• filter.go#L120-L135 -> gomatrixserverlib.DefaultStateFilter()
• eventauth.go#L33-L34

Internal doc about getting started developing on dendrite: https://docs.google.com/document/d/14USwq1faRk7I2m_LacBUipFGF0-hdrglmiFpiNeGM5o/edit

Internal doc syncing on this PR: https://docs.google.com/document/d/1F3EKyzZL5xOP2GQuqmC6H3BhYESXCLjUEPNEW_dMO94/edit

---

• cmd/client-api-proxy/main.go#L147

---

less +G /usr/local/var/log/postgres.log

---

$ ./build.sh && ./bin/dendrite-monolith-server --tls-cert server.crt --tls-key server.key --config dendrite.yaml

$ curl -XPOST -d '{"auth":{"type":"m.login.dummy"},"username":"root","password":"foobarbaz123"}' http://localhost:8008/_matrix/client/r0/register

http://localhost:8008/_matrix/client/r0/sync
Authorization: Bearer <access_token>

http://localhost:8008/_matrix/client/r0/sync?filter={ "room": { "include_leave": true, "timeline": { "limit": 5 } } }

Sytest

• Sytest: tests/31sync/09archived.pl#L306-L391
• Sytest docs/dendrite-setup.md

POSTGRES=1 ./run-tests.pl tests/31sync/09archived.pl -I Dendrite::Monolith -d ../dendrite/bin -W ../dendrite/sytest-whitelist --stop-on-fail

docker run --rm --name sytest-dendrite -v "/Users/eric/Documents/github/sytest:/sytest" \
  -v "/Users/eric/Documents/github/dendrite:/src" \
  -v "/Users/eric/logs:/logs" \
  -v "/Users/eric/go:/gopath" -e "POSTGRES=1" -e "DENDRITE_TRACE_HTTP=1" \
  matrixdotorg/sytest-dendrite:latest tests/31sync/09archived.pl



docker run --rm --name sytest -v "/Users/eric/Documents/github/sytest:/sytest" \
  -v "/Users/eric/Documents/github/synapse:/src" \
  -v "/Users/eric/logs:/logs" \
  -v "/Users/eric/go:/gopath" -e "POSTGRES=1" \
  matrixdotorg/sytest-synapse:py37 tests/31sync/09archived.pl

---

util.GetLogger(req.Context()).WithFields(log.Fields{
	"filter":        f,
	"IncludeLeave":  *f.Room.IncludeLeave,
	"timelineLimit": timelineLimit,
	"filterQuery":   filterQuery,
	"err":           err,
}).Info("Input filter")


log.WithFields(logrus.Fields{
	"filter":     *filter,
	"filterAsdf": fmt.Sprintf("%+v", *filter),
	"limit1":     filter.Room.Timeline.Limit,
}).Info("waeweafewfafewa Adding rooms to sync response")

dumping the events

type myadsfevent struct {
		event_id, sender, eventType, origin_server_ts, content string
	}

	events := make([]myadsfevent, len(recentStreamEvents))
	for i, v := range recentStreamEvents {
		events[i] = myadsfevent{
			event_id:         v.HeaderedEvent.Event.EventID(),
			sender:           v.HeaderedEvent.Event.Sender(),
			eventType:        v.HeaderedEvent.Event.Type(),
			origin_server_ts: strconv.FormatUint(uint64(v.HeaderedEvent.Event.OriginServerTS()), 10),
			content:          string(v.HeaderedEvent.Event.Content()),
		}
	}

	logrus.Info("sheehrhsegrehrehr")

	spew.Dump(events)
	logrus.WithFields(logrus.Fields{
		"joinEventIndex":     joinEventIndex,
		"leaveEventIndex":    leaveEventIndex,
		"recentStreamEvents": fmt.Sprintf("%+v", events),
	}).Info("cutting down the events")

Todo

• Handle CompleteSync
• Handle IncrementalSync
• Make sure it works with inline filter and filter ID
• Do we need to worry about This will fail on join -> leave -> sensitive msg -> join -> leave?

Pull Request Checklist

• I have added any new tests that need to pass to sytest-whitelist as specified in docs/sytest.md
• Pull request includes a sign off

Signed-off-by: Your Name <your@email.example.org>

[MadLittleMods]

@kegsay Can you do a once over review on this to sanity check, make sure I am not doing something the hard way or should go at a different angle?

I just have the CompleteSync part sorted out and need to look into IncrementalSync. The Sytest, Archived rooms only contain history from before the user left, gets a bit further now.

[MadLittleMods]

@neilalexander Do you know of a utility function to accomplish this?

Maybe I'm going about this wrong? Should I be doing the filterStreamEventsAccordingToHistoryVisibility filtering in p.DB.RecentEvents(...) call and that will setup the state to come out correct with p.DB.CurrentState(...) as well?

[MadLittleMods]

Filtering when we select the events is probably the way to go. Related to #1671 (comment)

[MadLittleMods]

As an update here, @neilalexander is going to take a look at adding timeline types filtering to d.OutputEvents.SelectRecentEvents(...) to and then we can come back to this one. Adding this is tricky because we also have to support SQLite which has problems with the dynamic array queries.

[MadLittleMods]

Type filtering now available thanks to @neilalexander via #1721 🚀

Will merge that in here and see how far I get!

[kegsay]

From what I can see, this code will continue to serve live events for rooms the user has left. We need to be limiting the max stream position (types.Range) to the leave event of the user at the very very least. That still isn't ideal since it'll leak events between a join->leave->join->leave in a single request (which if it's a complete sync will be the entire room.

[kegsay]

Given we're := on :90 this line is spurious.

[kegsay]

It's unclear which slice will have dupes removed from. Please document this.

[kegsay]

What's the status on this PR @MadLittleMods ?

[MadLittleMods]

@kegsay It's definitely stale but it's still in my browser tabs.

From #1323 (comment), "Might require better/full history visibility implementation" if we want to handle the join -> leave -> join.

Otherwise, may just need to give this some time again to get it back in shape.

[kegsay]

Please re-open this PR against latest master. Until then, closing it so it isn't constantly outstanding on our open PR list.