Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compatibility with new Firefox Sync #33

Open
ogasser opened this issue Nov 22, 2013 · 199 comments
Open

Compatibility with new Firefox Sync #33

ogasser opened this issue Nov 22, 2013 · 199 comments

Comments

@ogasser
Copy link
Contributor

ogasser commented Nov 22, 2013

The Sync Storage API v2.0 is in final draft status. According to the roadmap we could see them incorporated into the Mozilla clients in early to mid 2014. To prepare for this change we need to start implementing the new API before it is actually deployed to all Firefox clients.

Also, Firefox needs to allow defining a custom sync server. We opened this issue for that in the Mozilla bugtracker: https://bugzilla.mozilla.org/show_bug.cgi?id=989756 – please follow that.

@RoboMod
Copy link
Contributor

RoboMod commented Dec 1, 2013

Is there any client working with the new API yet? Without that testing will be really complex and effortful.

@ogasser
Copy link
Contributor Author

ogasser commented Dec 1, 2013

I don't think that there is any client using the new API yet. I found another document listing the most significant changes between versions.

I agree that it will be difficult to test the new API without any working client, I hope Mozilla will release a developer preview soon.

In the mean time it is better to write unit tests for the current API, see issue #30.

@posativ
Copy link

posativ commented Jan 8, 2014

Hi,

I got some feedback (on my server implementation in Python) that the Firefox Nightly on Windows 8 supports the new sync API. Furthermore, it seems Mozilla will drop support for self-hosted sync server, see posativ/weave-minimal#26.

@ogasser
Copy link
Contributor Author

ogasser commented Jan 8, 2014

Thx for the heads up @posativ!

I tried Firefox 29 Nightly and there is a new Sync option called Sync with Windows 8 style Nightly. I supposed that this would activate the Sync.next protocol. I could, however, not get past the account creation: The Firefox client does not do anything after sending the PUT user command.

At least in Firefox 29 Nightly the option for self hosted Sync server is still present. I don't interpret Mozilla's document as if they completely want to remove the self hosted option:

Self-hosted. Offer to migrate to Mozilla's system if there's no locked pref to indicate that that's not desirable, provide docs to support continued self-hosting, inform the user of decommissioning plans so they can make an informed choice.

To me this sounds like they want to inform self hosted users about Sync.next and offer them the migration. Why would they provide docs to support continued self hosting if it was removed from Firefox entirely? The decommissioning refers to the Sync v1.1 service hosted by Mozilla themselves imho.

@posativ
Copy link

posativ commented Jan 8, 2014

I could, however, not get past the account creation: The Firefox client does not do anything after sending the PUT user command.

I couldn't get past through it neither.

To me this sounds like they want to inform self hosted users about Sync.next and offer them the migration.

Yes, you are right. I asked them on IRC and they continue to support self-hosted sync servers (although you need a bit more than just the storage API, as they are moving to Firefox Account + new sync API).

@ogasser
Copy link
Contributor Author

ogasser commented Jan 9, 2014

Did they say what else is needed in addition to the storage API? I saw the Storage API v2.0 document but could not find anything else new (such as a new version of the User API/Firefox Account API).

@posativ
Copy link

posativ commented Jan 9, 2014

Yes, you need at least the new sync API and a tokenserver to authenticate against Firefox Account (you can self-host this, too, but it's a quite complex Node.js app).

@ogasser
Copy link
Contributor Author

ogasser commented Jan 9, 2014

So we need to implement the following Cloud services:

And obviously also:

@ogasser
Copy link
Contributor Author

ogasser commented Feb 5, 2014

Mozilla included their new Sync implementation into Firefox 27. The old version is not removed, though.

Additionally, the Storage API version was changed: It is now called SyncStorage API v1.5.

@renaudallard
Copy link

EOL for Legacy Sync is proposed to be Fx31.
Current nightly is Fx29. On android Fx29 you can already see in the accounts: 'Firefox Sync (deprecated)'. Given that current nightly 29 is the only one to accept FIPS compliant servers (no RC4), this is annoying.

@ogasser
Copy link
Contributor Author

ogasser commented Feb 5, 2014

@renaudallard EOL for Legacy Sync means that it won't be possible to add new devices and Mozilla could disable their Sync servers. Self-hosted Sync servers which are already set up will continue to work.

Nonetheless, we should look into implementing the new Sync protocol.

@ogasser
Copy link
Contributor Author

ogasser commented Feb 10, 2014

The onepw protocol has to be implemented on the Fx Accounts server.

@craigarno
Copy link

Blast... I just installed Firefox 29 beta on my OpenSUSE system and I can't find any way to use my OwnCloud Sync. FF29 wants to use the new Sync/Authentication approach.

Sync is too useful to be breaking like this.

I have only two other choices; don't sync, or block upgrades.

Almost forgot
https://support.mozilla.org/en-US/kb/how-to-update-to-the-new-firefox-sync?redirectlocale=en-US&as=u&redirectslug=how-sync-works-old-version-firefox&utm_source=inproduct

https://docs.services.mozilla.com/howtos/run-sync-1.5.html

I'm glad OGasser is looking into this issue.

@craigarno
Copy link

Firefox 28 Linux still works with OwnCloud6/Mozilla-Sync. So my Sync setup can last through about one more release update cycle. Then I'll have to look into ESR Releases as a possible solution:
http://www.mozilla.org/en-US/firefox/organizations/faq/
http://www.mozilla.org/en-US/firefox/organizations/all/
http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/

I'm posting so others who don't know about ESR releases can follow as needed to stay in-sync. This just buys time, it isn't a permanent solution.

@posativ
Copy link

posativ commented Mar 22, 2014

IIRC the next Firefox ERS (Firefox 31) still supports the old sync (legacy now), but I am unable to find the corresponding message from the sync-dev mailing list.

Firefox 29 is still in beta, just wait for a RC. The last time I tried Aurora (Firefox 29 back then I think), had an option to use the old mechanism.

@1of16
Copy link

1of16 commented Mar 24, 2014

are there any "good" news? as far as I can see it in the firefox 29 beta 1 there is no possibility to use our sync? or will it be possible to still use it if you update firefox?

@craigarno
Copy link

As @posativ said above, Firefox 29 is still in Beta, meaning it isn't a finished product. I think he is correct in saying wait for a RC release. Beyond this I did leave a message about "autonomy" with the Mozilla team (means nothing more than I made noise). This isn't a democracy, they'll decide what direction they need/want to take after reading input. The only way to make change is devote your time as a developer to the Mozilla effort. I think they've done a great job so far and may be trying to move a little too fast. I just did an upgrade on OpenSUSE 12.3 and Firefox is still at 27.0.1. I use Waterfox (x64) in Windows and it is still at 27.0.2. I went to http://www.mozilla.org/firefox and v28 is offered for download for both Windows and Linux. FF29 isn't "ready" yet. Go to Mozilla's site and make yourself heard if you have a concern. Even better if you can find a way to help address your concerns in new builds!

@craigarno
Copy link

I just installed FF29beta Windows and it appears to sync with OC 6.0.2. FF29beta inherited an already existing profile from an earlier version of Firefox. So I can't answer the question, "What if you're starting from scratch?"

@ogasser does OC/mozilla_sync also perform "pairing" for two FF seats? Or is "pairing" handled by Mozilla servers and only sync data is stored in OC?

@jancborchardt jancborchardt changed the title Sync Storage API v2.0 Compatibility with new Firefox Sync Mar 30, 2014
@jancborchardt
Copy link
Contributor

We raised an issue about the need for a »Custom server« option in the Mozilla bugtracker: https://bugzilla.mozilla.org/show_bug.cgi?id=989756

Please vote for it and add you to the cc list there to let them know we care. :)

@craigarno
Copy link

Done.

I think this issue is greater than just being able to use the OwnCloud/mozilla_sync Server for data. The issue is one of autonomy. Will OwnCloud/mozilla_sync in conjunction with Mozilla Firefox allow the Sync operation to perform it's function without an internet connection? (i.e. without Mozilla Servers) Pull your internet connection and find out... or does doing so "break" everything?

I made a comment along these lines in response to another question here
https://bugzilla.mozilla.org/show_bug.cgi?id=512969#c13

My question is, is this the same issue you raise in
https://bugzilla.mozilla.org/show_bug.cgi?id=989756 ? Or is this a different enough issue to require it's own entry in Mozilla's Bug Tracking system?

@tanghus
Copy link

tanghus commented Mar 31, 2014

@craigarno
Copy link

To answer @1of16 's query, I think we have Mozilla's attention:
https://bugzilla.mozilla.org/show_bug.cgi?id=989756#c14 (a thread started by @ogasser )

Dan Callahan is a self admitted "paid contributor" who says "The ability to host your own is not going away."

What isn't clear is who is paying Mr. Callahan to contribute. The implicit assumption is Mozilla.org, or an entity with closely aligned objectives. I guess time will tell. Our comments and their responses are now public.

It is suggested before posting at Mozilla that https://bugzilla.mozilla.org/page.cgi?id=etiquette.html be read.

@jancborchardt
Copy link
Contributor

Yes, if you agree with the issue on the Mozilla Bugzilla, please refrain from posting additional comments. There is a vote functionality at the top, and you can subscribe to the issue.
(@craigarno the issue on Bugzilla was started not by me but by @ogasser who develops the mozilla_sync app.)

As you see from Richard Newman’s comment: https://bugzilla.mozilla.org/show_bug.cgi?id=989756#c12 – they are working on it, but due to the several components of the new system it’s more difficult than just offering a »custom server« option.

@ogasser coming from Richard Newman’s comment – would it be possible/feasible for us to implement the whole range of Firefox Accounts functionality?

@ogasser
Copy link
Contributor Author

ogasser commented Apr 1, 2014

Yes, the plan is to provide users with the possibility to create their on Fx Account with mozilla_sync. See also the onepw link referenced above.

@craigarno
Copy link

It seemed for a while we were hitting a wall and dead-end which looked like it meant the end of Firefox - OwnCloud/Mozilla_sync capability. I rely on this capability for the reasons I've mentioned. If you guys have a direction your believe will converge on success, I'll pull my horns back in and stand back. Unfortunately I'm not in a position to offer coding help at this time. That's why I've been performing test, which takes less of my time. If you need more "help", let me know, otherwise I'll just watch.

Sorry about the mixup on who started the thread. I'll go back and edit my posts to set them straight.

Do you believe strongly enough that implementing Firefox Accounts in Mozilla_sync is the right way, architecturally, to address the need for Firefox Accounts for Sync? i.e. should I close out owncloud/core#7980? The question I see about doing this is how will accounts be managed if they are a part of Mozilla_sync? Or is your vision to build bridges to existing "account" systems with their own account management features? It seemed reasonable to me from a UI point of view to manage Firefox Accounts from the existing OwnCloud "Users" area. Or does the OwnCloud Framework allow you to register Mozilla_sync hooks to the "Users" area for data and any additional features, similar to how Mozilla_sync shows up in the existing Admin menu? I'm asking for a little (small) insight into how OwnCloud works internally.

@MichaWen
Copy link

It definitely happened on my Android Phone.

After the update I recognized a Firefox-Logo in the notification tray and touched it without reading the additional text.

I thought it was on of these notifications for receiving a link, shared through "send to another device" from my tablet.

Firefox has been opened and showed me a form to create a Sync account. Then I checked the Android Settings / Accounts and "Firefox Sync (outdated)" was gone.

Meanwhile I was able to recover it using the "I'm not near another device" link and everything is ok for now.

@rnewman
Copy link

rnewman commented Jan 29, 2016

Yes, that's the old Sync -> FxA migration flow. That you were able to recover an old Sync account on the device proves that. Note that your old Sync account will indeed disappear in two releases, about 12 weeks.

@elraro
Copy link

elraro commented Feb 17, 2016

@Larx how u make work mozilla_sync owncloud with android? i tried ALL, a lot of versions of firefox android and nothing...

@rnewman
Copy link

rnewman commented Feb 17, 2016

@elraro: you'll need a hosting setup that doesn't use SNI.

@Ra72xx
Copy link

Ra72xx commented Feb 18, 2016

@elraro: I didn't do anything special, I just added a Firefox Sync (legacy) adapter on Android. My OC is self-hosted, and I retain some old Firefox version on my desktop devices in order to initially set up a profile with legacy sync on them.

@Ra72xx
Copy link

Ra72xx commented Mar 13, 2016

With the update to FF45 on Android (other devices still on 44), the legacy sync started behaving weired (doubling bookmarks all the time etc.). On one Android device, the legacy sync adapter completely disappeared already, on another it was still present.
As there is no solution for self-hosting (one that a normal human being can set up) is available, I switched to Firefox's own servers :-( .

@franzhartwig
Copy link

@Larx: Giving Mozilla all your data is not nessesary. You can install your own syncserver, only authenticating with the Mozilla server:

Install dependencies:
sudo apt-get install python-dev git-core python-virtualenv

Compile and install syncserver:

cd /opt
sudo git clone https://github.com/mozilla-services/syncserver
cd syncserver
sudo make build

Generate random secret for syncserver:
head -c 20 /dev/urandom | sha1sum

Edit syncserver config:

sudo nano syncserver.ini
public_url = http://localhost:5000/ -> replace to real (DynDNS) address
sqluri = sqlite:////opt/syncserver/file.db
#or, if you use mysql:
sqluri = pymysql://root:yourmysqlpassword@localhost:3306/syncserver
secret = yourrandomnumber
allow_new_users = true # (this should be changed to false later)

Install pymysql module, if you decide to use mysql:

cd /opt/syncserver/syncserver/
./local/bin/pip install PyMySQL

Create database, if you decide to use mysql:

mysql -u root -p
CREATE DATABASE syncserver;
GRANT ALL PRIVILEGES ON syncserver.* TO sync IDENTIFIED BY „password“;
exit;

Set ownerchip and link to pserve:

sudo chown -R pi:pi /opt/syncserver
sudo ln -s /opt/syncserver/local/bin/gunicorn /usr/bin/gunicorn

Test syncserver:
gunicorn --paste /opt/syncserver/syncserver.ini

Stop syncserver:
Ctrl + C

Create init-Script:

sudo nano /etc/init.d/syncserver
Paste the following:

#!/bin/bash

## # BEGIN INIT INFO
# Provides: syncserver
# Required-Start: $all
# Required-Stop:
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: starts the Firefox Sync Server
# Description: Starts the Firefox Sync Server
## # END INIT INFO

PROJECT=/opt/syncserver
VIRTUALENV=/opt/syncserver/local
PID_DIR=/opt/syncserver
PID_FILE=/opt/syncserver/syncserver.pid
LOG_FILE=/opt/syncserver/logs/syncserver.log
USER=pi
GROUP=pi
PROD_FILE=/opt/syncserver/syncserver.ini

source $VIRTUALENV/bin/activate

cd $PROJECT

case „$1“ in
start)
gunicorn --paste $PROD_FILE --daemon --log-file=$LOG_FILE \
--pid $PID_FILE --user=$USER --group=$GROUP
;;
stop)
kill cat $PID_FILE
;;
restart)
kill cat $PID_FILE
gunicorn --paste $PROD_FILE --daemon --log-file=$LOG_FILE \
--pid $PID_FILE --user=$USER --group=$GROUP

;;
*)
echo $“Usage: $0 {start|stop|restart}“
exit 1
esac

exit $RET_VAL

Install init files and start syncserver:

sudo update-rc.d syncserver defaults
sudo service syncserver start

Configuring logrotate for rotating the syncserver logs may be a good idea.
The log is written to /opt/syncserver/logs/syncserver.log

In Firefox, go to about:config and change the syncserver URI:
services.sync.tokenServerURI

@elraro
Copy link

elraro commented Mar 13, 2016

there is a Firefox alternative with sync 1.1, Pale Moon

@Ra72xx
Copy link

Ra72xx commented Mar 13, 2016

Thanks for the instructions, I'll look into that (can I delete my data from Mozilla's servers once I've done the switch)?
I've looked into Palemoon, too, but it lacks proper Android support.

@franzhartwig
Copy link

I did not sync my data to Mozilla - don't know, if it can be deleted. Maybe you should backup your Firefox profile, delete all synced data, sync to Mozilla. Disable your internet connection, restore your old profile, change services.sync.tokenServerURI and enable your internet connection.

@jknockaert
Copy link

@franzhartwig Thank you for posting instructions for setting up a syncserver. But what is actually the point in hosting the data yourself as long as an external party is controlling the access to that data?

@ckruetze
Copy link

@jknockaert I'm following this discussion too as I would love this to work with owncloud in the future.
As for your question while not perfect it would help us a lot if we host the data.
Our server is on a private network and clients connect to it through VPN. While I wouldn't want Mozilla to control access to my data, they can't access it them self without the VPN.

@jknockaert
Copy link

@ckruetze I have been thinking along similar lines where the syncserver would only be available on the local network (possibly with vpn access to that network). But what will you do when Mozilla pulls the plug on the authentication service? They've been pulling the plug on mostly everything they've started up over the last decade, including Firefox OS, the old Sync, XUL, Thunderbird development, etc. And I'm afraid that Firefox for Android may well be the next thing to be dumped, leaving not much to be synced anyway.

@franzhartwig
Copy link

@jknockaert Of course, it would be better to run the account server on my own. But what are the choices? You can stop all syncing. Or you can sync your data to Mozilla.
Using external account servers is a common way. On many websites you see something like "Login with Facebook", "Login with Google", "Login with Yahoo". It's OpenID, not an exotic thing. So authenticating with my Mozilla account and syncing to my own syncserver is not so bad. If there is an easy way to setup the account server, I'll do it. But as long you need nodejs and install with npm, it's a pain. I tried it, but I failed.
I don't think that Mozilla will pull the plug on the authentication service. If they'll do it, they will announce it long time before doing it. I don't think of it. I am running Ubuntu on my computer. What if Canonical will stop developing Ubuntu? So what, I will have to change my OS. If Mozilla pulls the plug to the authentication server, I will have to look for another solution. Actually, my setup is my solution because Mozilla killed Sync 1.1. Why should I think of something, I don't know to happen?

@Ra72xx
Copy link

Ra72xx commented Mar 14, 2016

If it would be easy, somebody would probably have thought of it, but cannot the sync server be patched to accept owncloud authorization instead of Mozilla authentication?

@franzhartwig
Copy link

@Larx Maybe the syncserver can be patched to accept the owncloud authentication. But I am not able to read the code, not to mention to patch it.

@allo-
Copy link

allo- commented Mar 14, 2016

I am looking a bit in the WebExtensions API, it looks like it would be not too complicated to write a sync addon with it, which has the simplicity of sync 1.1. The problem: Important stuff like the history api are not yet supported, which means there is a gap between the first possible own sync and sync 1.1. Disregarding the development time as well.
The example addons seem really easy, where much can be achived with little code and chrome like api (so the addon could be chrome/opera/vivaldi/firefox all in one).
I am not sure about the sync 1.1 data model, i think creating an own simpler protocol would be good. key-value store on server side, some objects storing which keys are bookmarks, etc. with timestamps, as list of deleted objects for syncing deletions and objects stored as client-side encrypted data (needs some encryption library which does not look insecure).

@marcelklehr
Copy link

marcelklehr commented Jun 5, 2016

@allo- I think your approach could work. I've started work on an Addon using the WebExtensions API that taps into this hole: Enter floccus!

I invite anyone to join me as this seems something the browser vendors just won't start themselves.

The addon in its current state can only pull bookmarks from the Bookmarks app, but I intend to either push the people behind that app to pull the pull request that adds CRUD methods to the REST API, or adopt WebDAV as a syncing protocol.

As @allo- pointed out, Firefox does not yet support the History API, but it shouldn't be too long until that happens.

I'd also like to cc @marklindhout who has done some promising work on utilizing WebDAV for syncing Bookmarks.

@allo-
Copy link

allo- commented Jun 5, 2016

Looks like a nice start.

I think must have is for many people some kind of client-side encryption, that's the important point of sync 1.1.
And if you're not implementing the sync 1.1 protocol, a nice (uncomplicated) API with good documentation would be good, so other implementations (i.e. not requireing a whole owncloud) will be possible.

I had here a few lines of code iterating bookmarks and windows/tabs, but are waiting on the history api. Personally i would prefer a very dumb server as key-value store:

  • Full Data in key-value pairs (encrypted)
  • Collections (just information which keys represent which objects)
  • Computers: coupled computers, associated with collections
  • Log: timestamped actions, especially "delete", for incremental sync and removing stuff which once was in a collection. add/delete/change, type (bookmark, history), item (key)
  • Session: Data which is not stored long term, i.e. tabs open on other computers. A computer refreshes it regularily and it is possibly deleted if it's too old. Other options would be "send tab to pc" or some kind of shared clipboard. No log here, always full loading.

@craigarno
Copy link

I would also like to see independence from external servers for all phases, like authentication, initial setup, and operation. Or from past experience; I'd like independence from Mozilla servers with compatibility where desired. Where this is coming from is my older MozillaSync 1.1/OwnCloud setup still works well for me as an already established user. Setting up a "new user" is quite difficult because there are dependencies on Mozilla servers which no longer support Sync 1.1 services. And Mozilla recently broke, then restored Sync 1.5 login because of recent attack activity. I prefer not to find myself in this Mozilla server dependency pit again. What Sync 1.1 does, it does very well once it's setup. I'd like the Add-on Sync to work on a local network (without internet access), or securely across the internet with only my OwnCloud resources and local computer. I use MozillaSync 1.1 with my OwnCloud setup when I travel, or work off site. At least one of my clients blocks all external traffic except HTTP and HTTPS, so the solution needs to be able to use these interfaces, and I see this proposed one [Floccus] does.

If I had secure MozillaSync 1.1 "like" services in an add-on for Firefox/Waterfox (adding Chrome is a plus as long as Google servers aren't involved) which works with my OwnCloud setup with client side encryption... then Mozilla's discontinuing Sync 1.1 support in Firefox should be a non-issue for me.

From what little I can see, Floccus looks good guys! (yes, I went and looked, don't know what to do with it, but I know how to read)

@marklindhout
Copy link

Hi all, just responding to the mention I got via @marcelklehr: what I have right now is a grunt-based workflow that allows one to build Firefox and Chrome plugins easily, and a set of platform-agnostic central JavaScript files that can talk to wrappers for each browser's http request, bookmarking, and settings -API's.

However, that is all based on the add-on frameworks the browser vendors offered about half a year ago, and at that time the WebExtensions API was not ready for production yet. (I have no idea as to it's current state though, it might well be ready)

I've mostly done the groundwork for very basic bookmark management: Settings page and saving, making WebDav requests, creating bookmarks and folders all work. However, there's no sync logic yet in place, no encryption in settings storage, no decided format for actually saving the bookmark data.

So, it's rather rough, but if you need to get up and running fast, my workflow can definitely help you out with that.

@allo-
Copy link

allo- commented Jun 6, 2016

Chrome seems to have quite a bit of the API, firefox has this page as documentation for their progress:
http://arewewebextensionsyet.com/

One big step will be addUrl for history, with the possiblity to change the creation time, the type (typed url, visited link, etc.) and so on, see https://bugzilla.mozilla.org/show_bug.cgi?id=1265836

The extension discussed there will be firefox only at the moment, as the chrome/opera api just has the url parameter. Without the possibilty to set the creation time, a useful sync will be hard. I guess if this should be a chrome extension as well, you need to open a bug in the chromium project for the same feature.

@rnewman
Copy link

rnewman commented Jun 6, 2016

At the risk of sounding like a party pooper: the WebExtensions bookmarks API isn't transactional, so even if you safely use WebDAV transactions, you cannot safely and correctly synchronize structural bookmark changes using that API.

You can get part of the way there by implementing your own two-phase reads and writes, but I doubt you'd ever get to a point where you had no lingering bugs and/or some amount of potential data loss.

Beware also of undefined behavior around restoring bookmark backups, Refresh Firefox, etc. The import events don't seem fine-grained enough to me. Neither do the change events seem to be documented to fire on all kinds of changes.

Amusingly, Firefox Sync (née Weave) itself started many years ago as an add-on using Places bookmark APIs to synchronize bookmarks via WebDAV. WebDAV was ditched very early on for its truly atrocious scalability, and over time we've recognized that building a synchronization system that isn't intimately tied to the data layer is an exercise in frustration.

I simply would not attempt to build an offline-capable synchronization system without (a) transactions, (b) very detailed change/lifecycle hooks, and (c) thorough documentation of the API guarantees.

@allo-
Copy link

allo- commented Jun 6, 2016

That what i would address with the concept, i lied out roughly:

  • Do a full constructive sync every now and then, using the whole collection/key-value-store. Do not delete anything, but restore (later: Setting how careful this should work, rather losing one item or rather restoring one incorrectly)
  • sync on action: use the "bookmark added", "bookmark removed", "page visisted" events to push it in real time (maybe bundles every 30 seconds or something like this).
  • incremental sync: use the log to get actions from other devices, compare to an own log, which pushed items really were stored.
  • on inconstencies fall back to full sync again.

With very conservative deletion policy (the action log should be a good indicator, a missing data-item not), the system will be mostly self healing, even when something got lost in between.

@allo-
Copy link

allo- commented Jun 6, 2016

By the way, you're all php folks? I am still using https://github.com/allo-/django-sync-server and could sketch some of the database and API design in django very fast, trying to provide a useful api, which can be implemented with other frameworks as well. But of course such a backend would not really fit into owncloud.

@marcelklehr
Copy link

marcelklehr commented Jun 10, 2016

First up, sorry for the delay on my part.

On the status of the addon:
I'm in the process of getting floccus approved for AMO, so that it can be tested. @rnewman I'm already employing a layer of indirection for syncing the bookmarks that holds only the intersection of client and server. The syncing algorithm covers everything other than nested bookmarks and client-side update (i.e. you can't change the URL/title, atm) This is a first test though, and I believe it is best to transition to a different backend than the ownCloud bookmarks app.

Something that I think would be a perfect fit to the client-side encryption requirement is CRDTs: The server doesn't need to know anything, it just needs to store the encrypted changes to the data and the client would have no transformation overhead. For adding a device using PAKE to negotiate a session key for sharing the master key would be nice (thanks to @callahad for the suggestion).

Regarding a server API: @rnewman Thank you for your insight! What was so atrocious about WebDAV? I'm guessing every item (i.e. bookmarks, etc.) was treated as a WebDAV 'file'/resource? When using CRDTs, only a log is necessary, so I think the Atom Publishing Protocol would be a nice thing to utilize.

@allo-
Copy link

allo- commented Jun 10, 2016

I think with different Clients and Servers in Mind, Encryption should not be too complicated. More like base64 encoded AES or something like this. Client-only is useful, coupling can be done by shared secret (random UUID?) or when both devices are live and logged in to the same account there could be some Key exchange like in Diffie-Hellman scheme.
Such stuff should be not too complicated, as crypto schemes made up by yourself usually fail. So some library for key exchange and encryption using most basic standard algorithms would be useful.

@callahad
Copy link

PAKE might be worth looking into for key exchange. For an applied example, check out Brian Warner's talk on his project, Magic Wormhole, at PyCon last month.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests