Add partial implementation for the sync endpoint. #106

Merged
merged 1 commit into from Dec 28, 2016

Projects

None yet

3 participants

@farodin91
Member
farodin91 commented Oct 15, 2016 edited

Spec

https://matrix.org/docs/spec/client_server/r0.2.0.html#get-matrix-client-r0-sync

Parameters

  • since
  • full_state
  • filter
  • timeout
  • set_presence

State

  • next_batch
  • presence
    • events
  • rooms
    • join
      • unread_notifications
      • timeline
        • limited
        • prev_batch
        • events (basic)
      • state
        • events
      • account_data
        • events
      • ephemeral
        • events
    • leave
      • timeline
        • limited
        • prev_batch
        • events
      • state
        • events
    • invite
      • invite_state
        • events

Details

  • Support partial timeline.events, timeline.limited
  • No support for timeline.prev_batch
  • Support following in timeline.events (m.room.member, m.room.message, m.room.history_visibility)
  • Support timeline limiting by filter
  • Support partial since, full_state
  • Remove empty rooms rooms.join, rooms.leave, rooms.invite
  • Parse all parameters since, filter, full_state, set_presence, timeout
  • No support for parameter: set_presence, timeout
  • Basic invalidation for parameters
  • Basic macro for TryInto e.g. impl MessageEvent for Event
@farodin91 farodin91 referenced this pull request Oct 15, 2016
Open

API: GET /sync #8

9 of 31 tasks complete
@jimmycuadra
Member

However you'd like to do it is fine. I'm OK with having only some parts of it implemented in the first PR.

@mujx
Member
mujx commented Oct 16, 2016

Wouldn't be better to move the todo list into the sync issue #8 and break it into smaller ones instead of a giant PR?

@farodin91
Member

At this moment thought about only implement the batch system and rooms/join/account_data for the start.

@farodin91
Member

@mujx Could you look into?

src/api/r0/sync.rs
+ set_presence = PresenceState::Unavailable;
+ }
+ ("timeout", value) => {
+ timeout = u64::from_str_radix(value, 10).map_err(|_| ApiError::unknown(None))?;
@mujx
mujx Oct 20, 2016 Member

You can use a more detailed explanation about the error.

timeout = u64::from_str_radix(value, 10)
    .map_err(|err| ApiError::invalid_param("timeout", err.description()))?;
src/api/r0/sync.rs
+ ("timeout", value) => {
+ timeout = u64::from_str_radix(value, 10).map_err(|_| ApiError::unknown(None))?;
+ }
+ _ => (),
@mujx
mujx Oct 20, 2016 Member

Each parameter matching should be extensive resulting to an invalid_param error if no match was found. Here you drop back to the defaults silently. Also add tests for invalid options.

src/filter.rs
@@ -0,0 +1,148 @@
+//! Matrix filter.
@mujx
mujx Oct 20, 2016 Member

Not sure if ruma is the best place for these structs. @jimmycuadra ? Also the structs from src/sync.rs.

@mujx
mujx Oct 20, 2016 Member

Maybe they should be moved to ruma-identifiers like UserId.

@farodin91
farodin91 Oct 20, 2016 Member

I think we have to save filters.

@farodin91
farodin91 Oct 20, 2016 Member

but the FilterId

src/sync.rs
+ }
+
+ /// Parse a string to a `Batch`.
+ pub fn parse(str: &str) -> Result<Batch, ApiError> {
@mujx
mujx Oct 20, 2016 edited Member

You can simplify like so

pub fn parse(str: &str) -> Result<Batch, String> {
    let values: Vec<&str> = str.split("_").collect();

    if values.len() != 2 {
        return Err(String::from("Wrong number of tokens"));
    }

    let room_key = u64::from_str_radix(values[0], 10)
        .map_err(|err| err.to_string())?;

    let presence_key = u64::from_str_radix(values[1], 10)
        .map_err(|err| err.to_string())?;

    Ok(Batch::new(room_key, presence_key))
}

and then call

let batch = Batch::parse(value)
    .map_err(|err| ApiError::invalid_param("since", &err))?;

to return any errors.

Generally, use a descriptive message instead of None. It will be useful later for debugging.

src/sync.rs
+ let v: Vec<&str> = str.split("_").collect();
+ let mut room_key = 0;
+ let mut presence_key = 0;
+ if v.len() != 2 {
@mujx
mujx Oct 20, 2016 Member

The since parameter does not follow the form X_X where X is an integer

src/sync.rs
+}
+
+#[test]
+fn batch_parse_non_number_should_fail() {
@mujx
mujx Oct 20, 2016 Member

You can remove the should_fail suffix. Add an assert with the error message to explain the test.

@farodin91
Member

I'll split filter API in extra PR? To reduce complexity.

@farodin91 farodin91 requested a review from jimmycuadra Dec 9, 2016
src/sync.rs
+ let value = match event.event_type.as_ref() {
+ "m.room.member" => {
+ let event = event.clone();
+ let member_event: MemberEvent = (*event).try_into()?;
@farodin91
farodin91 Dec 9, 2016 Member

Any tipp to get this working cannot move out of borrowed content?

@jimmycuadra
Member

This needs the conflicts resolved once more. Sorry I've taken so long to review this! I'll get to it right away once it's ready to go.

@jimmycuadra

This is really, really fantastic work. Thank you so much for doing this! I left a few comments. All minor stuff.

src/api/r0/filter.rs
@@ -13,7 +13,7 @@ use models::filter::{Filter as DataFilter};
use models::user::User;
use modifier::SerializableResponse;
-/// Filter
+/// `Filter`
@jimmycuadra
jimmycuadra Dec 23, 2016 Member

I should've noticed this on the other filter PR, but this needs a real docstring that explains what the type is for.

src/api/r0/filter.rs
@@ -36,7 +36,7 @@ pub struct Filter {
pub not_senders: Vec<UserId>,
}
-/// RoomEventFilter
+/// `RoomEventFilter`
@jimmycuadra
jimmycuadra Dec 23, 2016 Member

Needs a real docstring.

src/api/r0/filter.rs
@@ -84,7 +84,7 @@ fn is_false(test: &bool) -> bool {
}
-/// RoomFilter
+/// `RoomFilter`
@jimmycuadra
jimmycuadra Dec 23, 2016 Member

Needs a real docstring.

src/api/r0/filter.rs
@@ -113,7 +113,7 @@ pub struct RoomFilter {
pub rooms: Vec<RoomId>,
}
-/// EventFormat
+/// `EventFormat`
@jimmycuadra
jimmycuadra Dec 23, 2016 Member

Needs a real docstring.

src/api/r0/filter.rs
@@ -160,7 +160,7 @@ impl ::serde::Deserialize for EventFormat {
}
}
-/// FilterResponse
+/// `FilterResponse`
@jimmycuadra
jimmycuadra Dec 23, 2016 Member

Needs a real docstring.

src/api/r0/mod.rs
pub use self::versions::Versions;
pub use self::filter::{GetFilter, PostFilter};
mod account;
mod directory;
mod event_creation;
-mod filter;
+pub mod filter;
@jimmycuadra
jimmycuadra Dec 23, 2016 Member

It feels odd to have this one sub-module be public. We could move the shared types out to a top-level module, but thinking farther ahead, these types are just gonna get pulled out into ruma-client-api, so maybe this is fine for now.

src/api/r0/sync.rs
+ use iron::status::Status;
+ use ruma_identifiers::EventId;
+
+ /// [https://github.com/matrix-org/sytest/blob/develop/tests/31sync/03joined.pl#L3]
@jimmycuadra
jimmycuadra Dec 23, 2016 Member

Let's use a specific commit hash rather than just the "develop" branch so that this URL will continue to point to the right thing even when that file is modified in the future.

src/api/r0/sync.rs
+ assert_eq!(room, None);
+ }
+
+ /// [https://github.com/matrix-org/sytest/blob/develop/tests/31sync/03joined.pl#L43]
@jimmycuadra
jimmycuadra Dec 23, 2016 Member

develop --> commit hash

src/api/r0/sync.rs
+ Test::assert_json_keys(room.find("ephemeral").unwrap(), vec!["events"]);
+ }
+
+ /// [https://github.com/matrix-org/sytest/blob/develop/tests/31sync/03joined.pl#L81]
@jimmycuadra
jimmycuadra Dec 23, 2016 Member

develop --> commit hash

src/api/r0/sync.rs
+ assert_eq!(room, None);
+ }
+
+ /// [https://github.com/matrix-org/sytest/blob/develop/tests/31sync/04timeline.pl#L1]
@jimmycuadra
jimmycuadra Dec 23, 2016 Member

develop --> commit hash

src/api/r0/sync.rs
+ assert_eq!(EventId::try_from(event.find("event_id").unwrap().as_str().unwrap()).unwrap().opaque_id(), event_id_2);
+ }
+
+ /// [https://github.com/matrix-org/sytest/blob/develop/tests/31sync/04timeline.pl#L223]
@jimmycuadra
jimmycuadra Dec 23, 2016 Member

develop --> commit hash

src/batch.rs
+
+/// A State Ordering.
+#[derive(Debug, Clone)]
+pub struct Batch {
@jimmycuadra
jimmycuadra Dec 23, 2016 Member

Is Batch going to be used in other places besides the sync endpoint? If not, this type could go in the sync module.

src/batch.rs
+ }
+
+ /// Make a String from a `Batch`.
+ pub fn to_str(&self) -> String {
@jimmycuadra
jimmycuadra Dec 23, 2016 Member

Let's impl Display for Batch and then use to_string instead. https://doc.rust-lang.org/stable/std/string/trait.ToString.html

src/batch.rs
+ }
+
+ /// Parse a string to a `Batch`.
+ pub fn parse(str: &str) -> Result<Batch, String> {
@jimmycuadra
jimmycuadra Dec 23, 2016 Member

What about implementing std::str::FromStr to get str's parse method?

src/middleware/mod.rs
@@ -37,7 +37,7 @@ macro_rules! middleware_chain {
};
}
-/// MiddlewareChain
+/// `MiddlewareChain`
@jimmycuadra
jimmycuadra Dec 23, 2016 Member

Needs a real docstring.

src/test.rs
+ (access_token, room_id)
+ }
+
+ pub fn get_next_batch(response: &Response) -> String {
@jimmycuadra
jimmycuadra Dec 23, 2016 Member

Needs a docstring.

src/test.rs
+ .to_string()
+ }
+
+ /// Sync.
@jimmycuadra
jimmycuadra Dec 23, 2016 Member

Needs a better docstring.

src/test.rs
+ response
+ }
+
+ /// Sync.
@jimmycuadra
jimmycuadra Dec 23, 2016 Member

Needs a better docstring.

src/test.rs
+
+ }
+
+ /// Sync.
@jimmycuadra
jimmycuadra Dec 23, 2016 Member

Needs a better docstring.

src/test.rs
+ response
+ }
+
+ pub fn assert_json_keys(json: &Value, keys: Vec<&str>) {
@jimmycuadra
jimmycuadra Dec 23, 2016 Member

Needs a docstring.

@jimmycuadra

I requested a few more small changes.

I'll push a new commit to master soon with an update to the Docker image. You can wait until that and then rebase and squash commits.

If you wanna discuss what to rename sync::Sync, let's chat on Matrix.

src/middleware/mod.rs
@@ -37,7 +37,7 @@ macro_rules! middleware_chain {
};
}
-/// `MiddlewareChain`
+/// `MiddlewareChain` ensures the all endpoints has a chain function.
@jimmycuadra
jimmycuadra Dec 23, 2016 Member

"ensures that all endpoints have"

src/sync.rs
+
+ Ok(Batch::new(room_key, presence_key))
+ }
+
@jimmycuadra
jimmycuadra Dec 23, 2016 Member

Extra newline.

src/sync.rs
@@ -257,3 +305,29 @@ impl Sync {
}))
}
}
+
+
@jimmycuadra
jimmycuadra Dec 23, 2016 Member

Extra newline.

src/sync.rs
+ }))
+ }
+}
+
@jimmycuadra
jimmycuadra Dec 23, 2016 Member

Extra newline.

src/test.rs
@@ -314,7 +315,7 @@ impl Test {
}
- /// Sync.
+ /// Query sync with multiple parameters.
@jimmycuadra
jimmycuadra Dec 23, 2016 Member

This is the same docstring as the sync function above. Did you mean "query sync since a specific point, with multiple parameters"?

@farodin91
farodin91 Dec 23, 2016 Member

The query parameter is called since.

src/test.rs
+ let response = self.get(&response_path);
+ assert_eq!(response.status, Status::Ok);
+ response
+
@jimmycuadra
jimmycuadra Dec 23, 2016 Member

Extra newline.

src/api/r0/sync.rs
+/// The `/sync` endpoint.
+pub struct GetSync;
+
+middleware_chain!(GetSync, [AccessTokenAuth]);
@jimmycuadra
jimmycuadra Dec 23, 2016 Member

I'm not crazy about the name GetSync for the handler. I think it should stay as Sync. I'm not sure what a better name for sync::Sync should be, though. The API endpoint is for a client to sync, but what sync::Sync does is actually just query a bunch of data with various filters, so maybe we could change the sync module into a more generic query module? Again, I'm not sure what that module's Sync type should be named. What do you think?

@mujx
mujx Dec 23, 2016 Member

My personal preference would be to keep the Sync endpoint as it is and move the sync structs and functions into models/events.rs and r0/sync.rs. The ideal solution would be to import these structs from ruma-client-api.

src/models/filter.rs
+ !test
+}
+
+
@jimmycuadra
jimmycuadra Dec 23, 2016 Member

Extra newline.

src/middleware/mod.rs
@@ -37,7 +37,7 @@ macro_rules! middleware_chain {
};
}
-/// MiddlewareChain
+/// `MiddlewareChain` ensures that all endpoints have.
@mujx
mujx Dec 23, 2016 Member

I think jimmy meant ensures that all endpoints have a chain function.

+ /// Return `RoomEvent`'s by `RoomId` and since.
+ pub fn find_room_events(connection: &PgConnection, room_id: &RoomId, since: i64) -> Result<Vec<Event>, ApiError> {
+ let events: Vec<Event> = events::table
+ .filter(events::event_type.like("m.room.%"))
@mujx
mujx Dec 23, 2016 Member

There are some m.call.* events defined as room events also.

@farodin91
farodin91 Dec 23, 2016 Member

This could be extend in the future. My implementation only support member and message.

src/test.rs
@@ -252,6 +253,88 @@ impl Test {
.unwrap()
.to_string()
}
+
+ /// Send a message to room.
+ pub fn message(&self, access_token: &str, room_id: &str, message: &str) -> Response {
@mujx
mujx Dec 23, 2016 Member

send_message would be a better name for this function.

src/test.rs
+ }
+
+ /// Create a User and Room.
+ pub fn initial_fixtures(&self) -> (String, String) {
@mujx
mujx Dec 23, 2016 Member

These initial_fixtures hide a lot of info when reading the tests. The gain is only one line and they are not configurable. If you keep them add a parameter for the name of the user and the room options.

src/test.rs
+ }
+
+ /// Query sync with multiple parameters included since.
+ pub fn sync_again(&self, access_token: &str, filter: Option<&str>, full_state: bool, since: &str) -> Response {
@mujx
mujx Dec 23, 2016 Member

You can probably merge sync_again and sync into one.

+ ("timeout", value) => {
+ timeout = u64::from_str_radix(value, 10).map_err(|err| ApiError::invalid_param("timeout", err.description()))?;
+ }
+ _ => (),
@mujx
mujx Dec 23, 2016 Member

This should probably fail with a bad request error.

@farodin91
farodin91 Dec 23, 2016 Member

access_token is also a query param

@mujx
mujx Dec 23, 2016 Member

Skip the access_token.

src/api/r0/sync.rs
+ let room = response.json().find_path(&["rooms", "join", room_id.as_ref()]).unwrap();
+ assert!(room.is_object());
+
+ let response = test.sync_again(&access_token, None, false, &next_batch);
@mujx
mujx Dec 23, 2016 Member

It would more readable if those functions take a struct as a parameter so we don't have to read the signature to know which value goes where. You can use SyncOptions.

src/query.rs
+ Ok(state)
+ }
+
+ fn convert_events_to_timeline(events: Vec<Event>, timeline_filter: &Option<RoomEventFilter>) -> Result<(i64, Timeline), ApiError> {
@mujx
mujx Dec 23, 2016 Member

Add some doc strings.

src/query.rs
+ }))
+ }
+
+ /// Handle sync rooms
@mujx
mujx Dec 23, 2016 Member

Also here its better to provide more info about the function.

src/test.rs
+ }
+
+ /// Create a User and Room.
+ pub fn initial_fixtures(&self, username: &str, public: bool) -> (String, String) {
@mujx
mujx Dec 23, 2016 Member

I meant to pass the whole room options like "{"visibility": "public"}" etc. It's more general and this way it's clear what you mean.

src/api/r0/sync.rs
+ let test = Test::new();
+ let (access_token, room_id) = test.initial_fixtures("carl", true);
+
+ let options = SyncOptions::new(
@mujx
mujx Dec 23, 2016 Member

A constructor still hides the fields. Just create a struct so the fields and their values are visible.

let options = SyncOptions {
   since: ...,
   full_state: ...,
   ...
}
@farodin91 farodin91 changed the title from Add basic implementation for the sync endpoint. to Add partial implementation for the sync endpoint. Dec 26, 2016
@farodin91 farodin91 Add partial implementation for the sync endpoint.
* Support partial timeline.events, timeline.limited
* No support for timeline.prev_batch
* Support following in timeline.events (m.room.member, m.room.message, m.room.history_visibility)
* Support timeline limiting by filter
* Support partial since, full_state
* Remove empty rooms rooms.join, rooms.leave, rooms.invite
* Parse all parameters since, filter, full_state, set_presence, timeout
* No support for parameter: set_presence, timeout
* Basic invalidation for parameters
* Basic macro for TryInto e.g. impl MessageEvent for Event
c5ff84c
@jimmycuadra jimmycuadra merged commit bf4fd61 into ruma:master Dec 28, 2016

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
@jimmycuadra
Member

🎉 🎉 🎉 Amazing work, @farodin91! 🎉 🎉 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment