Proposal: Seamless Sign-On for OpenStreetMap users from 3rd party apps

[stalker314314]

Problem

Hi all,

Context: Map builder (OSM map editor) team at Microsoft is trying to improve user experience during signup process. We’d like to propose couple of modifications to OSM code base for more streamlined single sign-in with 3rd party identity providers (Google, Facebook, Microsoft, Wikipedia, GitHub). The specific scenario of interest are the 3rd party applications acting on behalf of users, typically by submitting modifications to the map (map editor applications), one of which is our Map builder. Due to complex process involving multiple forms to fill and navigating up to 3 applications, it is anticipated that the number of users drops significantly before completing their first-time contribution.

We are ready to invest engineering resources in developing and implementing the proposal. Our hope is that this would benefit both signups coming from inside OSM website as well as from any other third-party signup process!

Description

We are focusing here on:

• Reducing the information required from user to fill in when the information is available from IdP (Identity Provider),
• Removing the email verification step when it is not necessary,
• Reducing the number of steps required to complete the process.

We propose to do this gradually, together with inputs from you, in one or more PRs, and here is short breakdown of proposed changes:

• Pre-populate email field when using "Login with Microsoft" and "Login with GitHub"
• Improvements to Login and Sign-Up Screens
• Introduce "Themed layouts" for Sign-up and Login views.

We would like to hear your ideas, suggestions, and concerns in the comments before we dive in into implementation!

Here are all these changes in more tech details

---

1. Pre-populate email field

Currently, only "Login with Facebook" and "Login with Google" pre-populate email field in Sign-up form.
The omniauth plugin currently used for "Login With Microsoft" is omniauth-windowslive and it does not properly parse the information received from the identity provider. It leaves the email field empty, even though email is available. It appears that replacing omniauth-windowslive with alternative plugins providing similar functionality would properly parse the tokens.

• Use alternative plugin for "login With Microsoft" (modify config/initializers/omniauth.rb, update Gemfile).
• Verify migration path for the users with existing tokens obtained from omniauth-windowslive.
• If necessary, submit PRs for appropriate omniauth plugin strategies.
• Create a new plugin if updating strategy is necessary and existing plugins are no longer maintained.
• Investigate if GitHub and OpenMedia providers provide verified email addresses.
• Update condition in UsersController.auth_success
• Investigate if OSM can fetch some of the user's profile properties from IdP in run time instead of keeping their copies in its database, to avoid keeping stale information.

This effort will not introduce OSM database schema modifications (maybe some new boolean field, but we will consult with you and be upfront on this if that is a case).

2. Improvements to Login and Sign-Up Screens

The proposal would include a number of improvements for Login and Sign-Up screens.

• Combine "Sign-up" view with "Login" as a separate tab.
• Sign-up view displays links to Contributor terms and terms of use, instead of displaying scrollable text box for Contributor terms.
• Remove "Repeat email" field in Sign-up view.
• Re-organize Sign-up view to reduce vertical space required.
• Ensure the screens can be displayed in browser popups.

We do not anticipate any database schema modifications for this part.

You can see current behavior, proposed behavior and also proposed signup/login on OSM website in screenshot section (watch out, images are pretty large).

Notable differences from existing flow

• Improved design of OSM sign-in and sign-up pages.
• When using 3rd party Identity Provider:
  • The email verification steps would be skipped if the IdP provides an email address,
  • User does not enter the password.
• The text of “Contributor Terms” is not included in the popup screens. Instead, only link is presented, similar to link to “Terms of use”, with a note that setting up the account implicitly assumes that user agreed to the terms.

Example workflow

As an example, consider a user working with an OSM editing web application. At some point they want to submit a modification to the maps. User does not yet have an OSM account but may have an account with one of identity providers supporting OpenID.

• Web application redirects to combined OSM Login/Signup page. OSM Login/Signup page may display different themes based on an optional hint provided by client application. Mock screens display Google and Microsoft themed OSM login page. Since user does not have OSM account, they click on “Sign Up” button.
• “Sign Up” form displays additional fields required for user to sign up to OSM using username/password:
  • “Confirm password” and “Display Name” text fields and links to “OSM terms of use” and “OSM contributor terms” documents, required for OSM accounts using username/password.
  • “Sign up with…” buttons for all supported identity providers.
    If signing with 3rd party IdP, the user does not need to fill in any fields in the form. They simply click one of “Sign up with…” buttons. OSM redirects to the appropriate IdP (Identity Provider) login page.
• IdP presents their login page, user logs in with IdP.
• IdP presents authorization request for access rights requested by OSM (typically email and public profile). User confirms the authorization. User is returned to OSM.
• OSM presents “Welcome” page where user can modify their display name if desired. Display name is pre-populated from the information obtained from user’s profile provided by IdP.
• OSM presents “Authorization Required” page to allow the application to modify the map. User clicks “Authorize.”
• Finally, the trust is established, and the contribution can be made.

3. Introduce "Themed layouts" for Sign-up and Login views.

The idea is that a 3rd party application would be able to trigger a themed layout, perhaps by providing a URI parameter. Themed layout would display one of the "Sign-in with" on more prominent place.

OSM would display non-themed layouts of the views. You can see how themed layout would look like in screenshots above.

Screenshots

Current behavior:

Proposed behavior:

Proposed signup/login on OSM website:

[tomhughes]

There's an awful lot of text here, and images that are frankly too big to really be able to comprehend, but from an initial attempt to parse it all a few comments:

• Adding more cases where we pre-populate email is fine, subject to questions around email validation but obviously that is no issue for the Microsoft provider assuming it always returns Microsoft addresses
• Moving the contributor terms to a link will require agreement from LWG and I think you're unlikely to get it
• The repeat email field is there for a reason - to reduce the number of people that have to get me to fix their email address when they mistype it
• Merging login and signup is fine by me - nobody understands at the moment that social signup is actually better done from the login page than the signup page
• Theming is something we can discuss - it's not something that has come up before and I need to think about it

One big issue is that our login/signup flows are very complicated and fragile and we have pretty much imposed a moratorium on making significant changes to them until we can fix that.

We have long wanted to move to using devise for handling this and have kind of said we're not going to make any changes until that is done.

[gravitystorm]

@stalker314314 thank you for these proposals, on the whole they look fine to me. In particular, I'm keen that the full end-to-end process of both signing up for an account, combined with authorisation for the app, works smoothly.

We propose to do this gradually, together with inputs from you, in one or more PRs,

Great! Definitely more than one PR involved here 😄

A few small notes from me, based on the initial designs:

• I repeat what tom said, I severely doubt that removing the terms from the signup process will be approved. It's also worth noting that the three languages presented are there for specific legal reasons. It would be best to assume that this is non-negotiable for now, and dealt with entirely separately from the rest of the proposals
• Is it possible to collaborate on these screenshots, or in some way on the workflows? I have comments on some parts of these, but I don't have any suggestions as to how best to make them. However, I can make them here in text if necessary.
• Please update the screenshots to use openstreetmap.org rather than .com 😄 Also I doubt that we'll have a separate domain for signup. Having the incorrect urls in the screenshot isn't important, but it is distracting when reviewing them.
• I'd like to see some ideas for how we manage the OSM-specific onboarding (currently the "welcome" screen) when users sign-up-and-authorise. There's two goals here - successful redirect to the application, plus also providing the onboarding information to the user.
• For the omniauth stuff, it would be great to have the email parsing fixes available upstream. Is there any reason not to make PRs for omniauth-windowslive? Perhaps that should be worked on first, since I know other projects can take some time to review/merge/release, and so it's good to get that stuff sorted early, long before PRs are made here.

[tomhughes]

Well omniauth-windowslive hasn't had a release since 2017 and the last commit was 2018 so it's not clear how active it is.

I did look into this a bit recently to see if there was a better option, and they seem to be surprisingly thin on the ground for a major identity provider.... There is omniauth-microsoft_graph and omniauth-microsoft-office365 but I confess to be unclear on whether they are really Office specific or are actually generic Microsoft authenticators...

[gravitystorm]

Well omniauth-windowslive hasn't had a release since 2017 and the last commit was 2018 so it's not clear how active it is.

Sure, but r2 had no activity since 2017 either (and, like omniauth-windowslive, no open issues nor open PRs), but I raised an issue last week and it has been fixed already! So it's always worth trying, before deciding to look elsewhere.

[milan-cvetkovic]

@tomhughes @gravitystorm, thank you for your input,

To introduce myself, I am working with @stalker314314 in Map builder team at Microsoft Serbia.

Integrating devise is not a trivial task, so we might start with simpler PRs.

Initially we would propose a PR for resolving the first point of the proposal,
namely fixing the extraction of the fields for "Login with Microsoft".

There is omniauth-microsoft_graph and omniauth-microsoft-office365 but I confess to be unclear on whether they are really Office specific or are actually generic Microsoft authenticators...

As @tomhughes mentions, since omniauth-windowslive is not recently updated,
there is a chance that it won't be fixed any time soon.

It appears that omniauth-microsoft-office365 can be used to login to Microsoft accounts out of the box. I would like to verify what the impact is to existing users who already gave their authorizations, and if it would be possible to migrate their authorizations.

At this time we would also make an effort to look into the reasons why "Login with GitHub" does not recognize the email, although this may be by design - email addresses are not really publicly available on GitHub.

Moving the contributor terms to a link will require agreement from LWG and I think you're unlikely to get it.

Agreed, we are planning to start discussions with LWG about sensitive aspects of this proposal.

Is it possible to collaborate on these screenshots, or in some way on the workflows? I have comments on some parts of these, but I don't have any suggestions as to how best to make them. However, I can make them here in text if necessary.

I am not aware of a publicly available tool to allow collaboration. I will spend some time looking into it. In the mean time, all comments/suggestions are welcome.

[tomhughes]

On email addresses I think I was a bit mistaken in what I wrote before.

We always take the address from third party providers, assuming that the omniauth plugin gives us one.

What we don't always do is skip validation of the address - we originally only did that for google on the grounds that a google account and a gmail address are directly connected so if the user has authenticated the email must be valid.

I was at some point persuaded to extend that to Facebook but I really don't think we should have done because they only know that it was valid at some point in the past.

[milan-cvetkovic]

Please update the screenshots to use openstreetmap.org rather than .com

@gravitystorm I have updated the images to fix the addresses in address bar and to label the individual screens with a number to allow collaborating here in text.

[tordans]

I am not aware of a publicly available tool to allow collaboration. I will spend some time looking into it. In the mean time, all comments/suggestions are welcome.

I put the last images in a google presentation which allows public commenting https://docs.google.com/presentation/d/18xvz32cWYKGoUazE6PlXRQYb6sjUIuLjXDb3AADWMjY/edit#slide=id.p

---

@milan-cvetkovic what are your thoughts on how to handle the configuration settings that are needed for the branded flow? There are different ways to go with different pros/cons. For exampled…

• Build the secondary, custom flow in separate routes and templates. That allows full flexibility, but is also the hardest to maintain
• Build a config object (yml file? db?) that allows to modify the (new) views (but use the same routes)
• Create URL params that allow to modify the (new) views (using the same routes) — likely the least favorable since some URL params need to be checked against an allow-list anyways (like the redirect url) and the changes required would clutter the URL quite a bit.

[milan-cvetkovic]

@tordans the intent of branded flows is probably only relevant when invoked during the setup phase from 3rd party app, so the URL generated by the app itself should indicate which flow preferred.

I didn't put too much effort in how to implement this feature - initially I had URL parameters in mind, and a single layout with alternate elements.

[milan-cvetkovic]

I am not aware of a publicly available tool to allow collaboration. I will spend some time looking into it. In the mean time, all comments/suggestions are welcome.

I suggest we use Figma, tool that we used to author the screenshots, since it offers easy zoom-in/out, and allows modifications of the design:

• Anyone can view the diagram.
• People who have Figma account can leave comments or ask owner for permission to edit.
• The ownership can be transferred to a different person, so the diagrams stay around longer than the original owner.

https://www.figma.com/file/RqpiQmshkhjTew2Y1MvDn9/OSM-SSO?type=design&node-id=3%3A3490&mode=design&t=Dk7hr35VQWraaysh-1

[tordans]

I suggest we use Figma, tool that we used to author the screenshots, since it offers easy zoom-in/out, and allows modifications of the design.

Thanks, I just tried it and it works great.

I updated my comment above.

---

Looking at the flow again @milan-cvetkovic I think there is an area that can be improved from a UX point of view if the right code path can be found: Screen 201 is the page after it was checked if the user is currently logged in or not. However, we don't know if the user has an account already. It looks like you assume that most users of your app will have an OSM account, because you link to the sign up page. Quite a few users will just add there data to this form, even if they are not signed up. The error case would be to show "unknown user/password".

I wonder how this error case can be improved.

• Some apps say "there is no email address for this account, do you want to sign up". However that makes it easier to mass check email-addresses. We would have to check if we consider this an issue (or maybe its possible already via some flow).
• One could make the error message a bit more special like "Unkown user/password. Do you want to sign up" and create a link that will auto-fill the email+passwort on the signup page. This way the data given by the users will not be lost

Another thing could be to make the page 201 even more focussed on the OSM social login and make sure that the flow behind this handles both cases (sign up and sign in) the same way. For example to hide the input fields behind a "or use email+password instead" link.

[milan-cvetkovic]

Please note that you will need to login to Figma in order to leave and read comments.

[milan-cvetkovic]

I have created PR 4169 to address pre-populating the email address and auto-validating it when populated during "Login with Microsoft." This PR does not introduce any UX modifications - we can discuss details in the PR itself.

[milan-cvetkovic]

@tordans

Another thing could be to make the page 201 even more focussed on the OSM social login and make sure that the flow behind this handles both cases (sign up and sign in) the same way. For example to hide the input fields behind a "or use email+password instead" link.

The idea is that 3rd party apps can chose which flavour, if any they want to display as signin/signup screens. The diagram currently shows only flavoured screens, but If no flavour is specified by 3rd party app, regular OSM login/signup (103) screen would be displayed.

[tordans]

The idea is that 3rd party apps can chose which flavour, if any they want to display as signin/signup screens. The diagram currently shows only flavoured screens, but If no flavour is specified by 3rd party app, regular OSM login/signup (103) screen would be displayed.

@milan-cvetkovic I understand that. My suggestion was meant to (a) improve the issue caused by having separate sign in / sign up screen and (b) improve conversion by nudging people even more to use the MS login (which we can assume they have, given where they came from…).

[milan-cvetkovic]

@tordans

(a) improve the issue caused by having separate sign in / sign up screen

If the user enters incorrect credentials, OSM currently displays login page with a simple message "Sorry, could not log in with those details." Adding a link to sign-up page with pre-populated fields from the previous unsuccessful attempt would be a nice addition.

(b) improve conversion by nudging people even more to use the MS login (which we can assume they have, given where they came from…)

I am not sure if it would be good to hide all the remaining fields when signing in with preferred identity provider. We should keep all login options open - allow users to login with whatever identity provider they prefer, even though the 3rd party software suggests provider "X". Maybe hiding behind "Sign in with username/password" link is too aggressive.

You do have a point that the flow behind "Login with" buttons on both login and sign-up pages really is identical, but I am not sure how to better communicate that to users.

[simonpoole]

@milan-cvetkovic hi, is this project still alive, as in does your team still intend to work on this?

[tomhughes]

That's what #4455 is?

[milan-cvetkovic]

The design diagram was updated to add globe image to sign up and log in screens, with "SSO Proposal v2" page, selectable from the menu in the top-left corner of figma doc:

Here is the direct link to the page: SSO Proposal v2