RFC: Sync manager

[mhammond]

As promised, I wrote up an initial RFC for the sync-manager a few of us have been discussing. It's not complete, but there should be enough there to get initial feedback.

Rendered

[rfk]

Thanks for kicking this off @mhammond!

[thomcc]

First review pass. Looks like it's moving in the right direction but I have some concerns.

• Callbacks are going to be tricky to make work due to ownership -- Even completely ignoring FFI issues.
• Which token is the storage_token?
• I think we probably need to flesh out how clients collection will be synced a little more.
• Several other vague concerns...

P.S. I edited your initial PR comment to include the link to the rendered markdown doc, hope you don't mind.

[linacambridge]

This is fantastic, @mhammond (and I ❤️ this format), thanks for starting the discussion!

[mhammond]

I've pushed a new revision which I hope addresses most of the comments above - see #5d6e33b for the diff, but the main changes are:

Main changes:

• Clarified that the tokenserver token may fail due to either auth issues or due to normal expiry (but for now I'm still sticking with the manager handling both those cases.

• Clarified some stuff around commands.

• Removed all references to callbacks, and made the (brave?) assertion that a rust implemented Sync manager will only ever deal with rust implemented engines, and further, that the engines and sync manager must all be in the same library.

• Removed all references to JSON

• Tried to clarify comments around the "not necessarily a 1:1 relationship between engines and collections"

• Added a few more things the manager will not do, and added some notes on telemetry.

• Clarified that the "clients" collection will be managed by the manager rather than as a new engine.

• Added the concept of a "prepare" function.

I'm going to resolve the conversations above which I hope I've suitably addressed - please open new conversations for what I've missed or made worse.

[grigoryk]

I left some inline comments.

Could you expand on how the proposed SyncManager will tie in with the individual storage layers? E.g. history has a "sync" method, and SyncManager has a sync(engines...) method. Will the rust syncmanager implementation invoke rust engines itself, passing in the state that it maintains?
It also wasn't entirely clear to me who's responsible for persisting that state. You mention that the state is FFI friendly to serve non-Rust implementations. I imagine in that case history/logins/etc storage library APIs will expose a "prepare" function over FFI, to set up the sync state, and the swift/js/whatever sync manager will be expected to call that as needed? And in the case of applications that just use the Rust implementation of the manager, they will not need to interact with prepare/sync functions exposed by the storage libs. Is that correct?

I'm assuming that Rust SyncManager will persist the state itself, since its APIs do not expose it.

[mhammond]

I've pushed a new version which I hope addresses most of the previous comments. I'll re-flag everyone for review and setup a meeting to discuss this.

[rfk]

I think we may be able to avoid or simplify this complication in an always-megazord world?

[mhammond]

We'll usually need/want to ensure we have the right set of "instances" with references to each other - the existing singletons are all outside rust. This is certainly doable and not major though, so I'll tone that down in the doc.

[rfk]

Hrm...we will need to think carefully about how this will interact with device registration over in the FxA component. Will the client record need to contain the FxA device id like it does on desktop?

[mhammond]

The client engine will want the FxA device ID, simply so it can populate fxaDeviceId. Desktop uses this for device names etc in various places.

It's probably ideal if we could use the fxa device ID as the client ID, which I'm leaning towards - in my mind, the only issue with that is that the FxA IDs are more ephemeral (ie, don't survive some password/account related events) - but is that actually a problem? We'd track this and clean up after ourselves (ie, we'd delete our old client record when we noticed the ID change, etc)

[rfk]

Does this "service state" get persisted, both in-memory and across restarts?

[mhammond]

I saw this as being completely transient from the component's POV - the API never needs it back. The embedding app might want to have it stick around in memory, but I don't think it ever needs to be persisted.

[linacambridge]

I went back and forth on this after our meeting yesterday, but I like this approach. Keeping the service state coarse still lets the high-level sync manager recover, but avoids pushing complex error handling logic into the app. And, considering that the app will probably want to remember this state, anyway, it makes sense that we don't try to store it, too.

[rfk]

For posterity: why not?

[mhammond]

"Because the ID to too ephemeral"? :) As above, I'm rethinking that - it's a real PITA for client code to deal with them being different, so we should suck it up and see what breaks :)

[rfk]

nit: is it technically "places" and "forms" stores since we implement a single syncable-store abstraction for bookmarks and history?

[linacambridge]

I went back and forth on this after our meeting yesterday, but I like this approach. Keeping the service state coarse still lets the high-level sync manager recover, but avoids pushing complex error handling logic into the app. And, considering that the app will probably want to remember this state, anyway, it makes sense that we don't try to store it, too.

[linacambridge]

whispers Three-way merging. I'm joking a bit, but also...what if the sync manager persisted the list of declined engines, recorded local changes since the last sync, and merged them with the server's view once the network is back up? That does push some complexity for noting declined state changes up into the high-level manager, which could be an argument for having the sync manager manage its own storage.

As you say, though, it's an extreme edge case, and having the manager manage its own database is probably not worth the benefit of handling conflicting declined changes.

[mhammond]

whispers Three-way merging.

We could do that - but should we?

Storing the transition is simple because it is a bool state - so just storing (name, new_state) seems enough - old_state can be implied. And we already store declined state, so adding this probably doesn't blow out "state budget" and force us into our own state storage.

It's more about user intent. It's like logins, where we could reconcile "device-a changed the username but not the password" and "device-b changed the password but not the username". We could reconcile and end up with both "new" values, but it seems unlikely to reflect user intent.

Similarly here - it's not clear to me "device-a declined bookmarks but left history enabled" but "device-b declined history and left bookmarks enabled" is best served by merging these rather than just taking the latest.

But yeah - "not clear" is the keyword here, so I'm erring on the side of simplicity - both in implementation and explanation - but I'm totally open to the fact I'm underthinking this :)

[linacambridge]

Merge away!