Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to set custom identity.sync.tokenserver.uri for self-hosted Sync #5006

Closed
SimonBasca opened this issue May 20, 2019 · 81 comments
Closed

Comments

@SimonBasca
Copy link
Contributor

@SimonBasca SimonBasca commented May 20, 2019

User Agent: Mozilla/5.0 (X11; CrOS x86_64 11316.165.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.122 Safari/537.36

Related issue: #3150

Steps to reproduce:

There is currently no way to specify a custom URL for a Firefox Sync Server (token url) while using the publicly available, Mozilla-hosted Firefox Accounts service.

Actual results:

There is no option to do this.

Expected results:

There should be an option like on Android to specify to custom token URL for self-hosted sync server. This option is available on desktop Firefox and on Firefox for Android.

@SimonBasca SimonBasca added this to Needs Triage in Firefox iOS Development via automation May 20, 2019
@SimonBasca
Copy link
Contributor Author

@SimonBasca SimonBasca commented May 20, 2019

Jerry Heiselman:
I have attempted to create a fxa-client-configuration file located at https://thor.heiselman.com/.well-known/fxa-client-configuration that mirrors the one served by accounts.firefox.com with the one change being the URL for the sync tokenserver url to point to my own.

When configuring this URL in Firefox on iOS using the method of revealing the hidden Advance Account Settings menu by tapping repeatedly on the Version string in the Settings. I filled in the base URL for my server (https://thor.heiselman.com) and enable the "Use Custom Account Service" option. Once I enable the option, I can see the client request the fxa-client-configuration file from my server successfully, however, it then attempts to load the login page on my server despite the configuration pointing all other services back to the public Firefox service.

m.m.naseri@gmail.com:
Can we bump the priority of this bug from a P3? Many people seem to be holding off moving to the FF ecosystem simply because they can’t carry their privately stored data without the additional cost of setting up an account server.

This clearly breaks the user experience and I’d have imagined that after a couple years of back and forth on Github this would’ve received more attention.

Loading

@farhanpatel farhanpatel added the P3 label May 21, 2019
@farhanpatel farhanpatel moved this from Needs Triage to Backlog in Firefox iOS Development May 21, 2019
@mmnaseri
Copy link

@mmnaseri mmnaseri commented May 24, 2019

Also mentioned in #3150 (which was closed in favor of the bug in the bugzilla system).

Loading

@jheiselman
Copy link

@jheiselman jheiselman commented May 24, 2019

I was the submitter of the Bugzilla report. I am still willing to work with someone here to help resolve this.

Loading

@jellium
Copy link

@jellium jellium commented May 30, 2019

This option (choose what Firefox Sync server to use) should definitely be available on iOS (as it appears to be available on Android). Looking forward to being able to set it up.

Loading

@mwegner
Copy link

@mwegner mwegner commented May 30, 2019

I recently made the switch to Firefox as my default browser (I think a lot of developer types have it installed, but drift back to their old setups for various comfort/muscle memory reasons). I would absolutely love to have the iOS version support a custom sync server.

There are quite a few "homelab" types that run significant infrastructure at home (i.e. VM hosts), and running something like a custom sync server doesn't add any extra overhead to their tech setup. I would absolutely run it, but it doesn't make any sense unless all of my Firefox devices can also use it.

Loading

@garvankeeley garvankeeley added this to the v18 milestone May 31, 2019
@garvankeeley
Copy link
Contributor

@garvankeeley garvankeeley commented Jun 5, 2019

When implemented, ensure URLs are https or localhost. This is a sec requirement from app services.

Loading

@jheiselman
Copy link

@jheiselman jheiselman commented Jun 5, 2019

@garvankeeley if you look at my comment (copied from the original bug), all URLs are HTTPS with valid LE certs.

Loading

@farhanpatel farhanpatel removed this from the v18 milestone Jun 11, 2019
@garvankeeley
Copy link
Contributor

@garvankeeley garvankeeley commented Jun 13, 2019

On desktop this is set in about:config using identity.fxaccounts.autoconfig.uri
Docs: https://moz-services-docs.readthedocs.io/en/latest/howtos/run-fxa.html

Loading

@jheiselman
Copy link

@jheiselman jheiselman commented Jun 13, 2019

@garvankeeley this issue is specifically for Firefox on iOS which has no about:config

Loading

@nook24
Copy link

@nook24 nook24 commented Jun 13, 2019

Is there a schedule available for this feature?

Many people seem to be holding off moving to the FF ecosystem simply because they can’t carry their privately stored data without the additional cost of setting up an account server.

This is exactly the reason why I don't use FF sync on any of my devices. Just because a self-hosted sync server it's not implemented on iOS :(

Loading

@jheiselman
Copy link

@jheiselman jheiselman commented Jun 13, 2019

Same here @nook24. I already have the sync server setup for my Linux desktop. But I don’t use FF on my iPhone because of this. I would in a heartbeat though.

Loading

@garvankeeley
Copy link
Contributor

@garvankeeley garvankeeley commented Jun 13, 2019

I have confirmed this works as-intended (by the Sync team), but the intention is that the entire stack is being hosted on the custom URL where the config file is.

Loading

@garvankeeley
Copy link
Contributor

@garvankeeley garvankeeley commented Jun 13, 2019

Closing this bug as works as-intended, the docs here still apply as to how to use your own fxa and sync stack: https://moz-services-docs.readthedocs.io/en/latest/howtos/run-fxa.html

Loading

Firefox iOS Development automation moved this from Backlog to Done Jun 13, 2019
@jheiselman
Copy link

@jheiselman jheiselman commented Jun 13, 2019

I’m very disappointed that we (the users) simply aren’t being heard. We know it’s working as intended. We want how it works to change. I feel like plenty of people have laid out perfectly valid reasons for the behavior to change. And the lack of willingness to even acknowledge that this leaves iOS at a distinct disadvantage shows a poor attitude towards the Firefox community as a whole.

I stand by my offer to help test any changes if anyone is willing to actually listen and attempt to implement this feature request.

Loading

@justindarc
Copy link
Contributor

@justindarc justindarc commented Jun 13, 2019

You are being heard. In fact, we were willing to spend a few hours today investigating this before concluding that this is not solely an iOS issue, but an FxA issue for something that is currently not supported. With limited resources, we have to prioritize issues and feature requests and the overall number of users who want this feature is almost immeasurably small.

Loading

@justindarc
Copy link
Contributor

@justindarc justindarc commented Jun 13, 2019

You are also welcome to run your own full FxA/Sync stack and you can follow the steps outlined here to configure iOS to work with it: https://moz-services-docs.readthedocs.io/en/latest/howtos/run-fxa.html

Loading

@jheiselman
Copy link

@jheiselman jheiselman commented Jun 13, 2019

I will be available to have a discussion around 2:00 pm US/Central and for most of the day after.

As for running a full stack server, it has been stated by several users that there are specific reasons for some of us to not want to do that.

Loading

@garvankeeley
Copy link
Contributor

@garvankeeley garvankeeley commented Jun 13, 2019

This is a discussion to have with the FxA/Sync team, they implemented this feature in the product with the intention that it be consistent with Android behaviour. If more users make this request across the various products then I could see that changing their opinion.

Loading

@nook24
Copy link

@nook24 nook24 commented Jun 13, 2019

Thanks for your time and investigation effort. I really appreciate this.

From the docs :

Note By default, a server set up using this guide will defer authentication to the Mozilla-hosted accounts server at https://accounts.firefox.com.
You can safely use the Mozilla-hosted Firefox Accounts server in combination with a self-hosted sync storage server. The authentication and encryption protocols are designed so that the account server does not know the user’s plaintext password, and therefore cannot access their stored sync data.

Alternatively, you can also Run your own Firefox Accounts Server to control all aspects of the system. The process for doing so is currently very experimental and not well documented.

I would say there should be a big fat warning, that the docs will not work for iOS devices.

I had given the /.well-known/fxa-client-configuration trick a shot but this didn't work. (As already expected).

This is a discussion to have with the FxA/Sync team, they implemented this feature in the product with the intention that it be consistent with Android behaviour.

I would really like to know why the option is not available on iOS. Is this an restriction from Apple or so?
I mean, it's available on Desktop and Android.
Again from the docs:

Since Firefox 33, Firefox for Android has supported custom sync servers. To configure Android Firefox 44 and later to talk to your new Sync server, just set the “identity.sync.tokenserver.uri” exactly as above before signing in to Firefox Accounts and Sync on your Android device.

Are there just not enough iOS based Firefox users?

Please don't get me wrong. I don't want to blame anyone why this isn't implemented already. I'm just wondering why there is a different behavior.

Loading

@garvankeeley
Copy link
Contributor

@garvankeeley garvankeeley commented Jun 13, 2019

I would say there should be a big fat warning, that the docs will not work for iOS devices.
I had given the /.well-known/fxa-client-configuration trick a shot but this didn't work. (As already expected).

Can you indicate what part doesn't work specifically on iOS? If so, I can report this to the FxA/Sync team to investigate further.
The bug report here indicates that the content server is not on the same host as the /.well-known/fxa-client-configuration, which according to the server team, is not supported (which is correct behaviour for all platforms).

I would really like to know why the option is not available on iOS. Is this an restriction from Apple or so?

Firefox iOS should behave like Desktop and Android, if I can show that it isn't behaving consistently, I can get traction on getting something fixed.

Loading

@jheiselman
Copy link

@jheiselman jheiselman commented Jun 13, 2019

Firefox on iOS doesn't have the ability to use about:config to configure the syncserver (token server) URL. Therefore, it already doesn't have feature parity with Android and desktop versions. This means that iOS users cannot follow the same setup procedure as Android and desktop users.

iOS users only have the option to "Use Custom Account Service". This asks for a single URL at which it will retrieve the /.well-known/fxa-client-configuration. The structure of this file is a listing of the different components with URLs for each piece. The documentation indicates that one can set each value independently.

While Firefox on iOS does query and retrieve the fxa-client-configuration file, it doesn't obey it as far as the different URLs are listed. As stated earlier in this issue, I downloaded the file from the public Firefox Accounts service and changed only the URL for the token server and hosted that changed file on my own server (same one hosting the syncserver). The file contents are as follows:

{"auth_server_base_url":"https://api.accounts.firefox.com","oauth_server_base_url":"https://oauth.accounts.firefox.com","pairing_server_base_uri":"wss://channelserver.services.mozilla.com","profile_server_base_url":"https://profile.accounts.firefox.com","sync_tokenserver_base_url":"https://thor.heiselman.com/sync/token/1.0/sync/1.5"}

Firefox on iOS doesn't seem to use any of these other values and instead attend to authorize against the hosting server instead of any of the others listed here.

Loading

@jheiselman
Copy link

@jheiselman jheiselman commented Dec 7, 2019

Yes, this solved my issue as well. Forcing the connection to HTTP 1.1 fixed it.

Loading

@Mardiie
Copy link

@Mardiie Mardiie commented Dec 7, 2019

Loading

@jheiselman
Copy link

@jheiselman jheiselman commented Dec 8, 2019

I’m not sure I’d agree that you’re compromising security. HTTP/2 is more about efficiency. If you secure your server with a certificate, then it’s still encrypted communication. Besides, you can still accept HTTP/2 over the internet and terminate and use HTTP/1.1 just between the front end proxy and the syncserver.

That said, I agree that this is a workaround at best, not a solution.

Loading

@fireglow
Copy link

@fireglow fireglow commented Dec 8, 2019

Just to chime in a little:
Python 2.7, the flavor of Python the sync software is written in, will go End-of-Life at the end of this month, year, and decade.
Mozilla already has indicated that there will be no rewrite for Python 3.
I gather there's a rewrite of these services in Rust in the works, at https://github.com/mozilla-services/syncstorage-rs
It's as of now unclear to me how all these parts will fit together in a way so us self-hosters will be able migrate over.

Loading

@jheiselman
Copy link

@jheiselman jheiselman commented Dec 8, 2019

Fire glow, good point! I hadn’t considered if there was a rewrite in the works. Hopefully the rust version is released soonish.

Loading

@mavidser
Copy link

@mavidser mavidser commented Dec 10, 2019

Besides, you can still accept HTTP/2 over the internet and terminate and use HTTP/1.1 just between the front end proxy and the syncserver.

@jheiselman I was under the impression that since it's only the iOS client which has this issue, HTTP 1.1 is needed on the internet, and doesn't matter on the syncserver. If that's the case, the client problems might still persist with the rust server too.

Loading

@cheywood
Copy link

@cheywood cheywood commented Dec 12, 2019

Seeing the same issue here; login works and I can see the server being hit but nothing comes across. The same sync server works without issues on Android, desktop Linux and macOS.

Firefox iOS v20.2, iOS v12.4.4, sync server v1.8.0, nginx v1.10.3.

Sync log from iPad upon attempt to sync:

iPad-sync-failure-sync.20191212T102735+0100.log

Maybe this warrants a separate issue?

Loading

@mmnaseri
Copy link

@mmnaseri mmnaseri commented Jun 29, 2020

There is now a regression after FF 25 (as reported in #6535) that has broken this workflow, and as such, this feature is not usable on iOS anymore. Not being able to access history/bookmarks on my primary mobile device is a deal breaker for most users.

Loading

@Blitzbirnep
Copy link

@Blitzbirnep Blitzbirnep commented Jul 2, 2020

Any chances to get this fixed?

Setup
I have a similar setup as described before. I am self-hosting the sync server Mozilla docker image from here on a synology nas. The reverse proxy is configured without http2 support and uses https protocol with hsts.

Issue
Invalid OAuth parameter: redirect_url with Firfox IOS App 27.0 (18428) when trying to connect to the custom sync server

Loading

@sprangen
Copy link

@sprangen sprangen commented Jul 8, 2020

Any chances to get this fixed?

Setup
I have a similar setup as described before. I am self-hosting the sync server Mozilla docker image from here on a synology nas. The reverse proxy is configured without http2 support and uses https protocol with hsts.

Issue
Invalid OAuth parameter: redirect_url with Firfox IOS App 27.0 (18428) when trying to connect to the custom sync server

i can second that. same behaviour here.

Loading

@delacroix0815
Copy link

@delacroix0815 delacroix0815 commented Jul 9, 2020

Same here. When trying to activate sync after i set a custom sync server, i get two choices:

  1. Scan QR-Code provided by stable.dev.lcip.org/pair
    or
  2. Use Email Address instead

1> when i go to stable.dev.lcip.org/pair, i always get a message like 'are you using your system camera? Then you have to connect out of a Firefox App', and no QR Code is showing up. Also the URL bar changes to 'https://stable.dev.lcip.org/pair/unsupported'.

2> Using the Email Button gives Invalid OAuth parameter: redirect_url

Loading

@parsifallo
Copy link

@parsifallo parsifallo commented Jul 9, 2020

same here

Loading

@HackintoshHD
Copy link

@HackintoshHD HackintoshHD commented Jul 25, 2020

Same problem as described by @delacroix0815 above with Firefox/iOS 27.0 (18428) running on iPadOS 13.6.

Loading

@BobWs
Copy link

@BobWs BobWs commented Jul 26, 2020

Still the same problem on Synology Nas with this image https://github.com/crazy-max/docker-firefox-syncserver

Loading

@Thlb
Copy link

@Thlb Thlb commented Aug 12, 2020

Same here ...

Loading

@jellium
Copy link

@jellium jellium commented Sep 5, 2020

This issue should be reopened since the feature appears to have been removed on latest versions (not sure when it was actually removed).

Loading

@jheiselman
Copy link

@jheiselman jheiselman commented Sep 5, 2020

It’s not removed, it just doesn’t work. You can still enable the developer settings by tapping repeatedly on the version. There’s still a spot for a Custom Sync Token Server under Advanced Sync Settings. And it seems to actually connect. It just doesn’t sync bookmarks or tabs anymore. The “Send to device” works. That’s pretty much it.

I feel like this feature is just abandoned which is too bad. I may as well switch back to Safari.

Loading

@jellium
Copy link

@jellium jellium commented Sep 5, 2020

It’s not removed, it just doesn’t work. You can still enable the developer settings by tapping repeatedly on the version. There’s still a spot for a Custom Sync Token Server under Advanced Sync Settings. And it seems to actually connect. It just doesn’t sync bookmarks or tabs anymore. The “Send to device” works. That’s pretty much it.

I feel like this feature is just abandoned which is too bad. I may as well switch back to Safari.

I cannot seem to find the Advanced Sync Settings within the debug mode anymore. That is why I believed the feature was actually removed. Where is this settings button hidden?

Loading

@jheiselman
Copy link

@jheiselman jheiselman commented Sep 5, 2020

Settings -> Tap Version row under About heading near the bottom. After some number of taps (10?) the debug menus show up at the bottom. Back near the top will be the Advanced Sync Server settings. Note that it will only show up if you are not currently signed in to Sync.

Loading

@Thlb
Copy link

@Thlb Thlb commented Sep 5, 2020

@jheiselman This feature is completly broken, given we cannot link a device to a custom sync server anymore (see #6535). What's the point of being able to configure a custom server if you cannot connect to it ?

Loading

@jheiselman
Copy link

@jheiselman jheiselman commented Sep 5, 2020

@Thlb I agree completely. I was just stating that it wasn't removed; just broken. It's been broken for quite some time now and no one seems to be paying any attention to this issue since it was closed. I suppose one should re-open it, but I feel like this isn't something that the Mozilla iOS team cares much about given how it took quite a bit of back and forth to even get someone to add the feature and then it was immediately broken after the next update.

Loading

@Balooforever
Copy link

@Balooforever Balooforever commented Sep 6, 2020

Reopen, still broken ..
Time for change ..

Loading

@nook24
Copy link

@nook24 nook24 commented Sep 7, 2020

For all following this issue or find it through web search: @drixter figured out how to restore the custom sync server feature on current iOS versions of Firefox.

Original post: #6535 (comment)

Summary: #6535 (comment)

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Linked pull requests

Successfully merging a pull request may close this issue.