New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Compatibility with new Firefox Sync #33
Comments
Is there any client working with the new API yet? Without that testing will be really complex and effortful. |
I don't think that there is any client using the new API yet. I found another document listing the most significant changes between versions. I agree that it will be difficult to test the new API without any working client, I hope Mozilla will release a developer preview soon. In the mean time it is better to write unit tests for the current API, see issue #30. |
Hi, I got some feedback (on my server implementation in Python) that the Firefox Nightly on Windows 8 supports the new sync API. Furthermore, it seems Mozilla will drop support for self-hosted sync server, see posativ/weave-minimal#26. |
Thx for the heads up @posativ! I tried Firefox 29 Nightly and there is a new Sync option called Sync with Windows 8 style Nightly. I supposed that this would activate the Sync.next protocol. I could, however, not get past the account creation: The Firefox client does not do anything after sending the At least in Firefox 29 Nightly the option for self hosted Sync server is still present. I don't interpret Mozilla's document as if they completely want to remove the self hosted option:
To me this sounds like they want to inform self hosted users about Sync.next and offer them the migration. Why would they provide docs to support continued self hosting if it was removed from Firefox entirely? The decommissioning refers to the Sync v1.1 service hosted by Mozilla themselves imho. |
I couldn't get past through it neither.
Yes, you are right. I asked them on IRC and they continue to support self-hosted sync servers (although you need a bit more than just the storage API, as they are moving to Firefox Account + new sync API). |
Did they say what else is needed in addition to the storage API? I saw the Storage API v2.0 document but could not find anything else new (such as a new version of the User API/Firefox Account API). |
Yes, you need at least the new sync API and a tokenserver to authenticate against Firefox Account (you can self-host this, too, but it's a quite complex Node.js app). |
So we need to implement the following Cloud services:
And obviously also: |
Mozilla included their new Sync implementation into Firefox 27. The old version is not removed, though. Additionally, the Storage API version was changed: It is now called SyncStorage API v1.5. |
EOL for Legacy Sync is proposed to be Fx31. |
@renaudallard EOL for Legacy Sync means that it won't be possible to add new devices and Mozilla could disable their Sync servers. Self-hosted Sync servers which are already set up will continue to work. Nonetheless, we should look into implementing the new Sync protocol. |
The onepw protocol has to be implemented on the Fx Accounts server. |
Blast... I just installed Firefox 29 beta on my OpenSUSE system and I can't find any way to use my OwnCloud Sync. FF29 wants to use the new Sync/Authentication approach. Sync is too useful to be breaking like this. I have only two other choices; don't sync, or block upgrades. https://docs.services.mozilla.com/howtos/run-sync-1.5.html I'm glad OGasser is looking into this issue. |
Firefox 28 Linux still works with OwnCloud6/Mozilla-Sync. So my Sync setup can last through about one more release update cycle. Then I'll have to look into ESR Releases as a possible solution: I'm posting so others who don't know about ESR releases can follow as needed to stay in-sync. This just buys time, it isn't a permanent solution. |
IIRC the next Firefox ERS (Firefox 31) still supports the old sync (legacy now), but I am unable to find the corresponding message from the sync-dev mailing list. Firefox 29 is still in beta, just wait for a RC. The last time I tried Aurora (Firefox 29 back then I think), had an option to use the old mechanism. |
are there any "good" news? as far as I can see it in the firefox 29 beta 1 there is no possibility to use our sync? or will it be possible to still use it if you update firefox? |
As @posativ said above, Firefox 29 is still in Beta, meaning it isn't a finished product. I think he is correct in saying wait for a RC release. Beyond this I did leave a message about "autonomy" with the Mozilla team (means nothing more than I made noise). This isn't a democracy, they'll decide what direction they need/want to take after reading input. The only way to make change is devote your time as a developer to the Mozilla effort. I think they've done a great job so far and may be trying to move a little too fast. I just did an upgrade on OpenSUSE 12.3 and Firefox is still at 27.0.1. I use Waterfox (x64) in Windows and it is still at 27.0.2. I went to http://www.mozilla.org/firefox and v28 is offered for download for both Windows and Linux. FF29 isn't "ready" yet. Go to Mozilla's site and make yourself heard if you have a concern. Even better if you can find a way to help address your concerns in new builds! |
I just installed FF29beta Windows and it appears to sync with OC 6.0.2. FF29beta inherited an already existing profile from an earlier version of Firefox. So I can't answer the question, "What if you're starting from scratch?" @ogasser does OC/mozilla_sync also perform "pairing" for two FF seats? Or is "pairing" handled by Mozilla servers and only sync data is stored in OC? |
We raised an issue about the need for a »Custom server« option in the Mozilla bugtracker: https://bugzilla.mozilla.org/show_bug.cgi?id=989756 Please vote for it and add you to the cc list there to let them know we care. :) |
Done. I think this issue is greater than just being able to use the OwnCloud/mozilla_sync Server for data. The issue is one of autonomy. Will OwnCloud/mozilla_sync in conjunction with Mozilla Firefox allow the Sync operation to perform it's function without an internet connection? (i.e. without Mozilla Servers) Pull your internet connection and find out... or does doing so "break" everything? I made a comment along these lines in response to another question here My question is, is this the same issue you raise in |
Voted and shared https://twitter.com/tanghus/status/450582369830199296 |
To answer @1of16 's query, I think we have Mozilla's attention: Dan Callahan is a self admitted "paid contributor" who says "The ability to host your own is not going away." What isn't clear is who is paying Mr. Callahan to contribute. The implicit assumption is Mozilla.org, or an entity with closely aligned objectives. I guess time will tell. Our comments and their responses are now public. It is suggested before posting at Mozilla that https://bugzilla.mozilla.org/page.cgi?id=etiquette.html be read. |
Yes, if you agree with the issue on the Mozilla Bugzilla, please refrain from posting additional comments. There is a vote functionality at the top, and you can subscribe to the issue. As you see from Richard Newman’s comment: https://bugzilla.mozilla.org/show_bug.cgi?id=989756#c12 – they are working on it, but due to the several components of the new system it’s more difficult than just offering a »custom server« option. @ogasser coming from Richard Newman’s comment – would it be possible/feasible for us to implement the whole range of Firefox Accounts functionality? |
Yes, the plan is to provide users with the possibility to create their on Fx Account with mozilla_sync. See also the onepw link referenced above. |
It seemed for a while we were hitting a wall and dead-end which looked like it meant the end of Firefox - OwnCloud/Mozilla_sync capability. I rely on this capability for the reasons I've mentioned. If you guys have a direction your believe will converge on success, I'll pull my horns back in and stand back. Unfortunately I'm not in a position to offer coding help at this time. That's why I've been performing test, which takes less of my time. If you need more "help", let me know, otherwise I'll just watch. Sorry about the mixup on who started the thread. I'll go back and edit my posts to set them straight. Do you believe strongly enough that implementing Firefox Accounts in Mozilla_sync is the right way, architecturally, to address the need for Firefox Accounts for Sync? i.e. should I close out owncloud/core#7980? The question I see about doing this is how will accounts be managed if they are a part of Mozilla_sync? Or is your vision to build bridges to existing "account" systems with their own account management features? It seemed reasonable to me from a UI point of view to manage Firefox Accounts from the existing OwnCloud "Users" area. Or does the OwnCloud Framework allow you to register Mozilla_sync hooks to the "Users" area for data and any additional features, similar to how Mozilla_sync shows up in the existing Admin menu? I'm asking for a little (small) insight into how OwnCloud works internally. |
It definitely happened on my Android Phone. After the update I recognized a Firefox-Logo in the notification tray and touched it without reading the additional text. I thought it was on of these notifications for receiving a link, shared through "send to another device" from my tablet. Firefox has been opened and showed me a form to create a Sync account. Then I checked the Android Settings / Accounts and "Firefox Sync (outdated)" was gone. Meanwhile I was able to recover it using the "I'm not near another device" link and everything is ok for now. |
Yes, that's the old Sync -> FxA migration flow. That you were able to recover an old Sync account on the device proves that. Note that your old Sync account will indeed disappear in two releases, about 12 weeks. |
@Larx how u make work mozilla_sync owncloud with android? i tried ALL, a lot of versions of firefox android and nothing... |
@elraro: you'll need a hosting setup that doesn't use SNI. |
@elraro: I didn't do anything special, I just added a Firefox Sync (legacy) adapter on Android. My OC is self-hosted, and I retain some old Firefox version on my desktop devices in order to initially set up a profile with legacy sync on them. |
With the update to FF45 on Android (other devices still on 44), the legacy sync started behaving weired (doubling bookmarks all the time etc.). On one Android device, the legacy sync adapter completely disappeared already, on another it was still present. |
@Larx: Giving Mozilla all your data is not nessesary. You can install your own syncserver, only authenticating with the Mozilla server: Install dependencies: Compile and install syncserver:
Generate random secret for syncserver: Edit syncserver config:
Install pymysql module, if you decide to use mysql:
Create database, if you decide to use mysql:
Set ownerchip and link to pserve:
Test syncserver: Stop syncserver: Create init-Script:
Install init files and start syncserver:
Configuring logrotate for rotating the syncserver logs may be a good idea. In Firefox, go to about:config and change the syncserver URI: |
there is a Firefox alternative with sync 1.1, Pale Moon |
Thanks for the instructions, I'll look into that (can I delete my data from Mozilla's servers once I've done the switch)? |
I did not sync my data to Mozilla - don't know, if it can be deleted. Maybe you should backup your Firefox profile, delete all synced data, sync to Mozilla. Disable your internet connection, restore your old profile, change services.sync.tokenServerURI and enable your internet connection. |
@franzhartwig Thank you for posting instructions for setting up a syncserver. But what is actually the point in hosting the data yourself as long as an external party is controlling the access to that data? |
@jknockaert I'm following this discussion too as I would love this to work with owncloud in the future. |
@ckruetze I have been thinking along similar lines where the syncserver would only be available on the local network (possibly with vpn access to that network). But what will you do when Mozilla pulls the plug on the authentication service? They've been pulling the plug on mostly everything they've started up over the last decade, including Firefox OS, the old Sync, XUL, Thunderbird development, etc. And I'm afraid that Firefox for Android may well be the next thing to be dumped, leaving not much to be synced anyway. |
@jknockaert Of course, it would be better to run the account server on my own. But what are the choices? You can stop all syncing. Or you can sync your data to Mozilla. |
If it would be easy, somebody would probably have thought of it, but cannot the sync server be patched to accept owncloud authorization instead of Mozilla authentication? |
@Larx Maybe the syncserver can be patched to accept the owncloud authentication. But I am not able to read the code, not to mention to patch it. |
I am looking a bit in the WebExtensions API, it looks like it would be not too complicated to write a sync addon with it, which has the simplicity of sync 1.1. The problem: Important stuff like the history api are not yet supported, which means there is a gap between the first possible own sync and sync 1.1. Disregarding the development time as well. |
@allo- I think your approach could work. I've started work on an Addon using the WebExtensions API that taps into this hole: Enter floccus! I invite anyone to join me as this seems something the browser vendors just won't start themselves. The addon in its current state can only pull bookmarks from the Bookmarks app, but I intend to either push the people behind that app to pull the pull request that adds CRUD methods to the REST API, or adopt WebDAV as a syncing protocol. As @allo- pointed out, Firefox does not yet support the History API, but it shouldn't be too long until that happens. I'd also like to cc @marklindhout who has done some promising work on utilizing WebDAV for syncing Bookmarks. |
Looks like a nice start. I think must have is for many people some kind of client-side encryption, that's the important point of sync 1.1. I had here a few lines of code iterating bookmarks and windows/tabs, but are waiting on the history api. Personally i would prefer a very dumb server as key-value store:
|
I would also like to see independence from external servers for all phases, like authentication, initial setup, and operation. Or from past experience; I'd like independence from Mozilla servers with compatibility where desired. Where this is coming from is my older MozillaSync 1.1/OwnCloud setup still works well for me as an already established user. Setting up a "new user" is quite difficult because there are dependencies on Mozilla servers which no longer support Sync 1.1 services. And Mozilla recently broke, then restored Sync 1.5 login because of recent attack activity. I prefer not to find myself in this Mozilla server dependency pit again. What Sync 1.1 does, it does very well once it's setup. I'd like the Add-on Sync to work on a local network (without internet access), or securely across the internet with only my OwnCloud resources and local computer. I use MozillaSync 1.1 with my OwnCloud setup when I travel, or work off site. At least one of my clients blocks all external traffic except HTTP and HTTPS, so the solution needs to be able to use these interfaces, and I see this proposed one [Floccus] does. If I had secure MozillaSync 1.1 "like" services in an add-on for Firefox/Waterfox (adding Chrome is a plus as long as Google servers aren't involved) which works with my OwnCloud setup with client side encryption... then Mozilla's discontinuing Sync 1.1 support in Firefox should be a non-issue for me. From what little I can see, Floccus looks good guys! (yes, I went and looked, don't know what to do with it, but I know how to read) |
Hi all, just responding to the mention I got via @marcelklehr: what I have right now is a grunt-based workflow that allows one to build Firefox and Chrome plugins easily, and a set of platform-agnostic central JavaScript files that can talk to wrappers for each browser's http request, bookmarking, and settings -API's. However, that is all based on the add-on frameworks the browser vendors offered about half a year ago, and at that time the WebExtensions API was not ready for production yet. (I have no idea as to it's current state though, it might well be ready) I've mostly done the groundwork for very basic bookmark management: Settings page and saving, making WebDav requests, creating bookmarks and folders all work. However, there's no sync logic yet in place, no encryption in settings storage, no decided format for actually saving the bookmark data. So, it's rather rough, but if you need to get up and running fast, my workflow can definitely help you out with that. |
Chrome seems to have quite a bit of the API, firefox has this page as documentation for their progress: One big step will be addUrl for history, with the possiblity to change the creation time, the type (typed url, visited link, etc.) and so on, see https://bugzilla.mozilla.org/show_bug.cgi?id=1265836 The extension discussed there will be firefox only at the moment, as the chrome/opera api just has the |
At the risk of sounding like a party pooper: the WebExtensions bookmarks API isn't transactional, so even if you safely use WebDAV transactions, you cannot safely and correctly synchronize structural bookmark changes using that API. You can get part of the way there by implementing your own two-phase reads and writes, but I doubt you'd ever get to a point where you had no lingering bugs and/or some amount of potential data loss. Beware also of undefined behavior around restoring bookmark backups, Refresh Firefox, etc. The import events don't seem fine-grained enough to me. Neither do the change events seem to be documented to fire on all kinds of changes. Amusingly, Firefox Sync (née Weave) itself started many years ago as an add-on using Places bookmark APIs to synchronize bookmarks via WebDAV. WebDAV was ditched very early on for its truly atrocious scalability, and over time we've recognized that building a synchronization system that isn't intimately tied to the data layer is an exercise in frustration. I simply would not attempt to build an offline-capable synchronization system without (a) transactions, (b) very detailed change/lifecycle hooks, and (c) thorough documentation of the API guarantees. |
That what i would address with the concept, i lied out roughly:
With very conservative deletion policy (the action log should be a good indicator, a missing data-item not), the system will be mostly self healing, even when something got lost in between. |
By the way, you're all php folks? I am still using https://github.com/allo-/django-sync-server and could sketch some of the database and API design in django very fast, trying to provide a useful api, which can be implemented with other frameworks as well. But of course such a backend would not really fit into owncloud. |
First up, sorry for the delay on my part. On the status of the addon: Something that I think would be a perfect fit to the client-side encryption requirement is CRDTs: The server doesn't need to know anything, it just needs to store the encrypted changes to the data and the client would have no transformation overhead. For adding a device using PAKE to negotiate a session key for sharing the master key would be nice (thanks to @callahad for the suggestion). Regarding a server API: @rnewman Thank you for your insight! What was so atrocious about WebDAV? I'm guessing every item (i.e. bookmarks, etc.) was treated as a WebDAV 'file'/resource? When using CRDTs, only a log is necessary, so I think the Atom Publishing Protocol would be a nice thing to utilize. |
I think with different Clients and Servers in Mind, Encryption should not be too complicated. More like base64 encoded AES or something like this. Client-only is useful, coupling can be done by shared secret (random UUID?) or when both devices are live and logged in to the same account there could be some Key exchange like in Diffie-Hellman scheme. |
PAKE might be worth looking into for key exchange. For an applied example, check out Brian Warner's talk on his project, Magic Wormhole, at PyCon last month. |
The Sync Storage API v2.0 is in final draft status. According to the roadmap we could see them incorporated into the Mozilla clients in early to mid 2014. To prepare for this change we need to start implementing the new API before it is actually deployed to all Firefox clients.
Also, Firefox needs to allow defining a custom sync server. We opened this issue for that in the Mozilla bugtracker: https://bugzilla.mozilla.org/show_bug.cgi?id=989756 – please follow that.
The text was updated successfully, but these errors were encountered: