Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to set custom identity.sync.tokenserver.uri for self-hosted Sync #5006

Closed
SimonBasca opened this issue May 20, 2019 · 36 comments

Comments

@SimonBasca
Copy link
Contributor

commented May 20, 2019

User Agent: Mozilla/5.0 (X11; CrOS x86_64 11316.165.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.122 Safari/537.36

Related issue: #3150

Steps to reproduce:

There is currently no way to specify a custom URL for a Firefox Sync Server (token url) while using the publicly available, Mozilla-hosted Firefox Accounts service.

Actual results:

There is no option to do this.

Expected results:

There should be an option like on Android to specify to custom token URL for self-hosted sync server. This option is available on desktop Firefox and on Firefox for Android.

@SimonBasca SimonBasca added the Bug 🐞 label May 20, 2019

@SimonBasca SimonBasca added this to Needs Triage in Firefox iOS Development via automation May 20, 2019

@SimonBasca

This comment has been minimized.

Copy link
Contributor Author

commented May 20, 2019

Jerry Heiselman:
I have attempted to create a fxa-client-configuration file located at https://thor.heiselman.com/.well-known/fxa-client-configuration that mirrors the one served by accounts.firefox.com with the one change being the URL for the sync tokenserver url to point to my own.

When configuring this URL in Firefox on iOS using the method of revealing the hidden Advance Account Settings menu by tapping repeatedly on the Version string in the Settings. I filled in the base URL for my server (https://thor.heiselman.com) and enable the "Use Custom Account Service" option. Once I enable the option, I can see the client request the fxa-client-configuration file from my server successfully, however, it then attempts to load the login page on my server despite the configuration pointing all other services back to the public Firefox service.

m.m.naseri@gmail.com:
Can we bump the priority of this bug from a P3? Many people seem to be holding off moving to the FF ecosystem simply because they can’t carry their privately stored data without the additional cost of setting up an account server.

This clearly breaks the user experience and I’d have imagined that after a couple years of back and forth on Github this would’ve received more attention.

@farhanpatel farhanpatel added the P3 label May 21, 2019

@farhanpatel farhanpatel moved this from Needs Triage to Backlog in Firefox iOS Development May 21, 2019

@mmnaseri

This comment has been minimized.

Copy link

commented May 24, 2019

Also mentioned in #3150 (which was closed in favor of the bug in the bugzilla system).

@jheiselman

This comment has been minimized.

Copy link

commented May 24, 2019

I was the submitter of the Bugzilla report. I am still willing to work with someone here to help resolve this.

@jellium

This comment has been minimized.

Copy link

commented May 30, 2019

This option (choose what Firefox Sync server to use) should definitely be available on iOS (as it appears to be available on Android). Looking forward to being able to set it up.

@mwegner

This comment has been minimized.

Copy link

commented May 30, 2019

I recently made the switch to Firefox as my default browser (I think a lot of developer types have it installed, but drift back to their old setups for various comfort/muscle memory reasons). I would absolutely love to have the iOS version support a custom sync server.

There are quite a few "homelab" types that run significant infrastructure at home (i.e. VM hosts), and running something like a custom sync server doesn't add any extra overhead to their tech setup. I would absolutely run it, but it doesn't make any sense unless all of my Firefox devices can also use it.

@garvankeeley garvankeeley added this to the v18 milestone May 31, 2019

@garvankeeley

This comment has been minimized.

Copy link
Contributor

commented Jun 5, 2019

When implemented, ensure URLs are https or localhost. This is a sec requirement from app services.

@jheiselman

This comment has been minimized.

Copy link

commented Jun 5, 2019

@garvankeeley if you look at my comment (copied from the original bug), all URLs are HTTPS with valid LE certs.

@farhanpatel farhanpatel removed this from the v18 milestone Jun 11, 2019

@garvankeeley

This comment has been minimized.

Copy link
Contributor

commented Jun 13, 2019

On desktop this is set in about:config using identity.fxaccounts.autoconfig.uri
Docs: https://moz-services-docs.readthedocs.io/en/latest/howtos/run-fxa.html

@jheiselman

This comment has been minimized.

Copy link

commented Jun 13, 2019

@garvankeeley this issue is specifically for Firefox on iOS which has no about:config

@nook24

This comment has been minimized.

Copy link

commented Jun 13, 2019

Is there a schedule available for this feature?

Many people seem to be holding off moving to the FF ecosystem simply because they can’t carry their privately stored data without the additional cost of setting up an account server.

This is exactly the reason why I don't use FF sync on any of my devices. Just because a self-hosted sync server it's not implemented on iOS :(

@jheiselman

This comment has been minimized.

Copy link

commented Jun 13, 2019

Same here @nook24. I already have the sync server setup for my Linux desktop. But I don’t use FF on my iPhone because of this. I would in a heartbeat though.

@garvankeeley

This comment has been minimized.

Copy link
Contributor

commented Jun 13, 2019

I have confirmed this works as-intended (by the Sync team), but the intention is that the entire stack is being hosted on the custom URL where the config file is.

@garvankeeley

This comment has been minimized.

Copy link
Contributor

commented Jun 13, 2019

Closing this bug as works as-intended, the docs here still apply as to how to use your own fxa and sync stack: https://moz-services-docs.readthedocs.io/en/latest/howtos/run-fxa.html

Firefox iOS Development automation moved this from Backlog to Done Jun 13, 2019

@jheiselman

This comment has been minimized.

Copy link

commented Jun 13, 2019

I’m very disappointed that we (the users) simply aren’t being heard. We know it’s working as intended. We want how it works to change. I feel like plenty of people have laid out perfectly valid reasons for the behavior to change. And the lack of willingness to even acknowledge that this leaves iOS at a distinct disadvantage shows a poor attitude towards the Firefox community as a whole.

I stand by my offer to help test any changes if anyone is willing to actually listen and attempt to implement this feature request.

@justindarc

This comment has been minimized.

Copy link
Contributor

commented Jun 13, 2019

You are being heard. In fact, we were willing to spend a few hours today investigating this before concluding that this is not solely an iOS issue, but an FxA issue for something that is currently not supported. With limited resources, we have to prioritize issues and feature requests and the overall number of users who want this feature is almost immeasurably small.

@justindarc

This comment has been minimized.

Copy link
Contributor

commented Jun 13, 2019

You are also welcome to run your own full FxA/Sync stack and you can follow the steps outlined here to configure iOS to work with it: https://moz-services-docs.readthedocs.io/en/latest/howtos/run-fxa.html

@jheiselman

This comment has been minimized.

Copy link

commented Jun 13, 2019

I will be available to have a discussion around 2:00 pm US/Central and for most of the day after.

As for running a full stack server, it has been stated by several users that there are specific reasons for some of us to not want to do that.

@garvankeeley

This comment has been minimized.

Copy link
Contributor

commented Jun 13, 2019

This is a discussion to have with the FxA/Sync team, they implemented this feature in the product with the intention that it be consistent with Android behaviour. If more users make this request across the various products then I could see that changing their opinion.

@nook24

This comment has been minimized.

Copy link

commented Jun 13, 2019

Thanks for your time and investigation effort. I really appreciate this.

From the docs :

Note By default, a server set up using this guide will defer authentication to the Mozilla-hosted accounts server at https://accounts.firefox.com.
You can safely use the Mozilla-hosted Firefox Accounts server in combination with a self-hosted sync storage server. The authentication and encryption protocols are designed so that the account server does not know the user’s plaintext password, and therefore cannot access their stored sync data.

Alternatively, you can also Run your own Firefox Accounts Server to control all aspects of the system. The process for doing so is currently very experimental and not well documented.

I would say there should be a big fat warning, that the docs will not work for iOS devices.

I had given the /.well-known/fxa-client-configuration trick a shot but this didn't work. (As already expected).

This is a discussion to have with the FxA/Sync team, they implemented this feature in the product with the intention that it be consistent with Android behaviour.

I would really like to know why the option is not available on iOS. Is this an restriction from Apple or so?
I mean, it's available on Desktop and Android.
Again from the docs:

Since Firefox 33, Firefox for Android has supported custom sync servers. To configure Android Firefox 44 and later to talk to your new Sync server, just set the “identity.sync.tokenserver.uri” exactly as above before signing in to Firefox Accounts and Sync on your Android device.

Are there just not enough iOS based Firefox users?

Please don't get me wrong. I don't want to blame anyone why this isn't implemented already. I'm just wondering why there is a different behavior.

@garvankeeley

This comment has been minimized.

Copy link
Contributor

commented Jun 13, 2019

I would say there should be a big fat warning, that the docs will not work for iOS devices.
I had given the /.well-known/fxa-client-configuration trick a shot but this didn't work. (As already expected).

Can you indicate what part doesn't work specifically on iOS? If so, I can report this to the FxA/Sync team to investigate further.
The bug report here indicates that the content server is not on the same host as the /.well-known/fxa-client-configuration, which according to the server team, is not supported (which is correct behaviour for all platforms).

I would really like to know why the option is not available on iOS. Is this an restriction from Apple or so?

Firefox iOS should behave like Desktop and Android, if I can show that it isn't behaving consistently, I can get traction on getting something fixed.

@jheiselman

This comment has been minimized.

Copy link

commented Jun 13, 2019

Firefox on iOS doesn't have the ability to use about:config to configure the syncserver (token server) URL. Therefore, it already doesn't have feature parity with Android and desktop versions. This means that iOS users cannot follow the same setup procedure as Android and desktop users.

iOS users only have the option to "Use Custom Account Service". This asks for a single URL at which it will retrieve the /.well-known/fxa-client-configuration. The structure of this file is a listing of the different components with URLs for each piece. The documentation indicates that one can set each value independently.

While Firefox on iOS does query and retrieve the fxa-client-configuration file, it doesn't obey it as far as the different URLs are listed. As stated earlier in this issue, I downloaded the file from the public Firefox Accounts service and changed only the URL for the token server and hosted that changed file on my own server (same one hosting the syncserver). The file contents are as follows:

{"auth_server_base_url":"https://api.accounts.firefox.com","oauth_server_base_url":"https://oauth.accounts.firefox.com","pairing_server_base_uri":"wss://channelserver.services.mozilla.com","profile_server_base_url":"https://profile.accounts.firefox.com","sync_tokenserver_base_url":"https://thor.heiselman.com/sync/token/1.0/sync/1.5"}

Firefox on iOS doesn't seem to use any of these other values and instead attend to authorize against the hosting server instead of any of the others listed here.

@justindarc

This comment has been minimized.

Copy link
Contributor

commented Jun 13, 2019

Ok, after talking with some more FxA folks, it seems that you are correct and this should be supportable as long as we add a new pref on iOS explicitly for identity.sync.tokenserver.uri. I'll update the title to add clarity here so that this work can get scheduled.

@justindarc justindarc reopened this Jun 13, 2019

Firefox iOS Development automation moved this from Done to Needs Triage Jun 13, 2019

@justindarc justindarc changed the title Allow for custom Firefox Sync Server while using Mozilla hosted FxA server Add ability to set custom identity.sync.tokenserver.uri for self-hosted Sync Jun 13, 2019

@justindarc justindarc added Feature 🎁 and removed P3 Bug 🐞 labels Jun 13, 2019

@nook24

This comment has been minimized.

Copy link

commented Jun 13, 2019

Sounds good! Looking forward to get this.
Many thanks to all of you.

@jheiselman

This comment has been minimized.

Copy link

commented Jun 13, 2019

Thank you justindarc. This looks like it will work great for us!

@justindarc

This comment has been minimized.

Copy link
Contributor

commented Jun 13, 2019

No problem. Sorry for the confusion.

@justindarc justindarc self-assigned this Jun 17, 2019

@project-bot project-bot bot moved this from Needs Triage to In Progress in Firefox iOS Development Jun 17, 2019

justindarc added a commit to justindarc/firefox-ios that referenced this issue Jun 17, 2019

justindarc added a commit to justindarc/firefox-ios that referenced this issue Jun 17, 2019

justindarc added a commit to justindarc/firefox-ios that referenced this issue Jun 17, 2019

@justindarc

This comment has been minimized.

Copy link
Contributor

commented Jun 17, 2019

WIP pull request: #5158

justindarc added a commit to justindarc/firefox-ios that referenced this issue Jun 18, 2019

justindarc added a commit to justindarc/firefox-ios that referenced this issue Jun 18, 2019

justindarc added a commit to justindarc/firefox-ios that referenced this issue Jun 18, 2019

justindarc added a commit to justindarc/firefox-ios that referenced this issue Jun 18, 2019

Firefox iOS Development automation moved this from In Progress to Done Jun 20, 2019

@fireglow

This comment has been minimized.

Copy link

commented Jul 7, 2019

I got the update on iOS today, thanks to all involved for making this happen!
Is it working for everybody as expected?
All I'm seeing is a GET /mozilla/token/1.0/sync/1.5 HTTP/2.0" 200 893.
Nothing for /storage/, no actual syncing.

Edit: it's 100% working on lighttpd! Thanks again to everybody!

@s-fiebig

This comment has been minimized.

Copy link

commented Jul 7, 2019

Installed the iOS update an hour ago and everything is working as expected. Set the token server, restarted Firefox and logged in to my Firefox account. Syncing is also working in both directions.
@fireglow My sync server is running under http (at least for now), maybe this makes a difference.

Thanks to all involved for making this happen!

@nook24

This comment has been minimized.

Copy link

commented Jul 7, 2019

Works like expected for me, many thanks to all.

This is my setup: https://daniel-ziegler.com/computer/netzwerk/linux/2019/07/07/Firefox-Sync-iOS/

@Balooforever

This comment has been minimized.

Copy link

commented Jul 9, 2019

Doesn't work for me :(
I use the docker ffsync image and a reverse-proxy.
I add my server : https://mydomain/token/1.0/sync/1.5

@jheiselman

This comment has been minimized.

Copy link

commented Jul 10, 2019

I finally got the chance to test this out and am running into a problem.

I'm running the syncserver from Docker Hub (same as I always have). Firefox on my iPhone is able to set the token server, I'm able to login, and I see the request make it to the sync server. The device is added to my (Mozilla hosted) Firefox Account and my desktop browser pops up a notification saying that the iPhone was added to my account. I am able to send pages back and forth between my two browsers. However, my bookmarks, history, open tabs, and logins/passwords are not syncing.

All of the requests to my sync server are resulting in HTTP 200 status codes and I was using this sync server with my Android phone, so I think everything is setup properly there, but I'd be happy to do some testing/logging if someone points me in a direction.

@Balooforever

This comment has been minimized.

Copy link

commented Jul 10, 2019

Same for me @jheiselman , thank for your post, i'm not english : it's hard for me to explain !
I use Docker Hub too and it's doesn't work.

@captn3m0

This comment has been minimized.

Copy link

commented Jul 14, 2019

Same issue here. Logs give 200:

10.8.0.1 - - [14/Jul/2019:20:59:32 +0000] "GET /token/1.0/sync/1.5 HTTP/2.0" 200 513 "-" "Firefox-iOS-FxA/18.0b15690 (iPhone; iPhone OS 12.3.1) (Firefox)" 22306 "Host-firesync-bb8-fun-0" "http://172.22.0.22:5000" 4ms
10.8.0.1 - - [14/Jul/2019:20:59:45 +0000] "GET /token/1.0/sync/1.5 HTTP/2.0" 200 513 "-" "Firefox-iOS-FxA/18.0b15690 (iPhone; iPhone OS 12.3.1) (Firefox)" 22325 "Host-firesync-bb8-fun-0" "http://172.22.0.22:5000" 2ms
10.8.0.1 - - [14/Jul/2019:21:00:00 +0000] "GET /token/1.0/sync/1.5 HTTP/2.0" 200 513 "-" "Firefox-iOS-FxA/18.0b15690 (iPhone; iPhone OS 12.3.1) (Firefox)" 22339 "Host-firesync-bb8-fun-0" "http://172.22.0.22:5000" 2ms
10.8.0.1 - - [14/Jul/2019:21:00:03 +0000] "GET /token/1.0/sync/1.5 HTTP/2.0" 200 513 "-" "Firefox-iOS-FxA/18.0b15690 (iPhone; iPhone OS 12.3.1) (Firefox)" 22340 "Host-firesync-bb8-fun-0" "http://172.22.0.22:5000" 2ms
10.8.0.1 - - [14/Jul/2019:21:00:37 +0000] "GET /token/1.0/sync/1.5 HTTP/2.0" 200 513 "-" "Firefox-iOS-FxA/18.0b15690 (iPhone; iPhone OS 12.3.1) (Firefox)" 22360 "Host-firesync-bb8-fun-0" "http://172.22.0.22:5000" 2ms
10.8.0.1 - - [14/Jul/2019:21:00:40 +0000] "GET /token/1.0/sync/1.5 HTTP/2.0" 200 513 "-" "Firefox-iOS-FxA/18.0b15690 (iPhone; iPhone OS 12.3.1) (Firefox)" 22364 "Host-firesync-bb8-fun-0" "http://172.22.0.22:5000" 2ms

"Send tab to device" feature works sometimes (with a lot of lag and Firefox restarts). Bookmarks and other content sync doesn't work at all.

Running the latest mozilla/syncserver image (3f9bc839727f)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.