Conversation
|
|
||
| Replication is the ability of a cozy-stack to copy / move all or a subset of its data to another support. It should cover 4 use cases | ||
|
|
||
| - Moving : The one-time act of moving a cozy from one hosting provider to another. |
There was a problem hiding this comment.
I don't think moving should be a use case for replication. I prefer if we do moving by exporting data to a tarball, and then importing the tarball. The files inside the tarball should be organized in a documented way, and with standard formats, to open interoperability with cloud solutions.
| - Moving : The one-time act of moving a cozy from one hosting provider to another. | ||
| - Devices : The continuous act of syncing change to a subset of the cozy documents and files to and from an user cozy to the same user's Devices through cozy-desktop and cozy-mobile | ||
| - Sharing : The continuous act of syncing change to a subset of the cozy documents and files to and from another user's cozy. | ||
| - Backup : The regular but not continuous act of syncing the cozy data to |
There was a problem hiding this comment.
I don't think replication is a good way to backup data. For backups, you want to ignore views to gain disk space. You want to have partial and full backups. You may want to encrypt the backups and upload them somewhere with cheap space (S3, glacier, things like that).
| Replication is the ability of a cozy-stack to copy / move all or a subset of its data to another support. It should cover 4 use cases | ||
|
|
||
| - Moving : The one-time act of moving a cozy from one hosting provider to another. | ||
| - Devices : The continuous act of syncing change to a subset of the cozy documents and files to and from an user cozy to the same user's Devices through cozy-desktop and cozy-mobile |
There was a problem hiding this comment.
I think this use case is more general than what is said here. It's possibility for an app to sync data and files: the app can be on a device, but we will want also to support the client-side apps with an offline mode later (with a pouchdb in a service worker for example).
| CouchDB replication is a well-understood, safe and stable algorithm ensuring replication between two couchdb databases. | ||
| Rsync is a well-understood, safe and stable algorithm to synchronize files hierarchy between two hosts. | ||
|
|
||
| **Moving** the cozy is similar to a one time **Backup** |
There was a problem hiding this comment.
It's a bit a simplification, you also have DNS and TLS certificates to change
|
|
||
|
|
||
| CouchDB replication is a well-understood, safe and stable algorithm ensuring replication between two couchdb databases. | ||
| Rsync is a well-understood, safe and stable algorithm to synchronize files hierarchy between two hosts. |
There was a problem hiding this comment.
You use a very similar sentence for CouchDB replication and rsync that make them look more similar that they are. CouchDB replication can work in a single shot mode, but also in a continuous way, rsync is only in single shot. CouchDB can detect conflicts (it can happen easily when you have offline), rsync can't. Rsync is probably not a good match when data can change on both hosts.
| Some useful packages: | ||
| - Pure go rsync implementation : https://godoc.org/github.com/smtc/rsync | ||
| - Go bindings for librsync : https://github.com/silvasur/golibrsync | ||
| - A package to sync folder in FS or Swift: http://rclone.org/ (not rsync based, always whole file transfer) |
There was a problem hiding this comment.
I'd be really happy to make a pull request to rclone to support cozy cloud VFS
|
|
||
| ## Details | ||
|
|
||
| - In step 5, the replicator can also attempt to `PUT :target/:docid` if doc are too heavy, but this should not happens in cozy-stack considering there wont be attachment in couchdb. |
There was a problem hiding this comment.
On the things I'd do differently with emails if I were to start a new, is to consider the content as a text/plain or text/html attachment. It is really inefficient to pass the whole content just to see the messages list.
There was a problem hiding this comment.
OK, but maybe we will have to keep the body in couchdb, with a new doctype (maybe io.cozy.email.contents), for indexation/search.
| In both case, we need to support | ||
|
|
||
| - `PUT :both/_local/:revdocid` to store the current sequence number. | ||
|
|
There was a problem hiding this comment.
I think it should be nice to also support _all_docs. It is currently used by cozy-mobile to speed up the synchronization. I think pouchdb try to use it for replication, and has some fallbacks when it's not available.
|
|
||
| ## Continuous replication | ||
|
|
||
| It is impossible to implement it by simply proxying to couchdb (see unlimited inactive users). But neither the Device nor the Backup option need open continuous replication. |
There was a problem hiding this comment.
Cozy-desktop uses continuous replication (well, only a continuous changes-feed), so I disagree here to say that Device doesn't need open continuous replication.
|
|
||
| Depending on benchmarking, we can do some optimization on the feed : | ||
| - close feeds when the user is not on screen | ||
| - multiplex different applications' feed, so each open cozy will only use one socket to the server. And then use a [SharedWorker](https://developer.mozilla.org/en/docs/Web/API/SharedWorker) or postMessage to pass the events to each application javascript context (either iframe or browser tabs). |
There was a problem hiding this comment.
The apps won't be on the same domains and we will make them not embeddable in iframes for security reasons. I don't think this let us some options to mutualize connections between apps. But we can see that later, it's only optimizations.
|
|
||
| It is impossible to implement it by simply proxying to couchdb (see unlimited inactive users). | ||
|
|
||
| The current version of cozy-desktop uses it. **TODISCUSS** It could be replaced by 3-minutes polling without big losses in functionality, eventually with some more triggers based on user activity. |
There was a problem hiding this comment.
Don't you think it's good place/time to document the trick exposed by Jan? Listening to the CouchDB API that gives us the database which have changed recently (was it /_db_updates?)
|
LGTM. @jinroh are you okay too to merge this PR? |
3 participants
This is a big big wip.
Can we figure out an unified sharing/replication/permissions model that we can explain in 5min or less ?
@nono @jinroh @Gara64 @Ljinod
❤️