On-boarding: Use curated list from XMPP Providers project

[Echolon]

Hello,

Here is my suggestion to make use of the curated server list from the XMPP Providers project. Transparency: Yes it is another project where I am involved in.

The advantage is that it can simply ready via JSON and, that's what I highly recommend, can make use of guiding users directly to the right support address if they have one of those domain. It can also offer many other options, as randomized distribution of user but also allocation based on e.g. users choice by country / suggestions based on IP / device language. We can also automatically register the user to the relevant support chat / or admin JID.

If you clone the repository and run the script you get three categories. Category A has the best ranking and also only list servers which support IBR (in-band registration). The output lists only list the servers an no details, but if it is of interest to make use of some details, those need to be exported as well. Bear in mind that the C lists might have servers with bad properties of any kind. So I do not recommend to take the bare list.

Also happy to hear your feedback to the project. Let me know if there are technical questions.

[Echolon]

Reference implementation: UWPX/UWPX-Client#159
Blabber.im: https://codeberg.org/kriztan/blabber.im/commit/b00ec1e5f6c4f745cacc3b48dc392136690d7f26

[Echolon]

Entries are frequently updated and accessible via data.xmpp.net now:

https://data.xmpp.net/providers/v1/providers-A.json

From the invent.kde code repository:
providers-B.json
providers-C.json

Category A servers satisfy all criteria best.

Please do not use any C servers for user on-boarding as long as you have thoroughly reviewed them yourself!

[Echolon]

Related with some ideas #128

• Introduce user to XMPP with one screen "page" (within the first start of Monal)
• Show reference to provider's support website if the password forgotten (hard coding in the beginning I'd suggest)
  • That can be taken from the providers list, too!
• Let them pic an avatar
• More distance between the fields
• Add a text which tells the user that they might should create an account: You don't have an account?
• Move the first two fields + button a bit more upwards

• Suggestion how it could be drawn:
  Enter:
  __ your nickname __ @domain.im
  Once they finally created their account, confirm in an extra screen, like: "Hurray! Your ID is your nickname@blabber.im Have lots of fun!"
• Other process: Choose your name - This can be set before the ID creation. The ID could also be suggested as 'Name' + random 3 digits number (Name: Bob, ID: bob543@domain.im)
• May inform them about specific expectations like:
  • Minimum length of 2 characters
  • This is case insensitive: macbeth is not the same as MacBeth or Macbeth.
  • Characters not allowed: " & ' / : < > @
• https://blog.prosody.im/great-invitations/

[Echolon]

rel. #442

[melvo]

Users want to have an easy onboarding (login / registration). An easy registration is feasible by suggesting XMPP providers for account registration. Once the onboarding is accomplished, users want to add new contacts easily. Suggestions for the XMPP provider's domain while typing the chat address can speed up that process.

The new project XMPP Providers tackles both tasks. It provides a curated list of providers. In contrast to similar projects, XMPP Providers categorizes providers so that client developers do not have to filter the list on their own. Additionally, it helps providers to improve their properties and integrate badges on their websites. That empowers the whole system of XMPP providers, users and client developers.

There are already projects using XMPP Providers. It would be great to see Monal joining them!

[Echolon]

A short reminder to the devlopers: I know people like to take the general C list with no or self made filters.

We highly recommend to actually follow the recommendation and only take A servers. We invested time to review each server and the ranking in the end has its reason. There are C servers which should currently not be recommended to anyone.

[Echolon]

Entries are frequently updated and accessible via data.xmpp.net now:

https://data.xmpp.net/providers/v1/providers-A.json

From the invent.kde code repository: providers-B.json providers-C.json

Category A servers satisfy all criteria best.

Please do not use any C servers for user on-boarding as long as you have thoroughly reviewed them yourself!

I have updated the links for B & C

[Echolon]

https://providers.xmpp.net/

[FriedrichAltheide]

We will continue to maintain our own list of XMPP servers for registration

[melvo]

@FriedrichAltheide Our goal is to have a well-maintained data base for as many clients as possible. That way, the client developers do not have to do that on their own. Could you explain what you expect from XMPP Providers in order to integrate it in Monal?

[melvo]

Where is your current list located and how can it be used at the moment? I could not find that feature in the latest release.

[tmolitor-stud-tu]

Since XMPP providers were published we had many discussions with other XMPP users, server administrators and developers as well as client developers. Within this limited sample, we encountered people misunderstanding the grading. This resulted in discussions what XMPP providers is trying to solve and for whom.

While some may argue that an objective list of XMPP servers is beneficial both, to end users choosing a server and to client developers integrating more servers, we figured that it opened more question than it answered. We will discuss some of our various reasons, that were discussed within our team as well as in several MUCs by several people in the past, below.

1.     The recommendations on the website [0] are not filtered by country. In Germany e.g., only EU based servers should ever be part of the ranking. We therefore would need to filter the server list ourselves resulting in different server selections in our app and the webpage. Therefore, users could wonder why we selected different servers than advertised officially on the providers.xmpp.net subdomain.

2.     The ranking is incomprehensible to users. They will presumably only take a look at the category A and B. Servers like conversations.im or trashserver.net that are well known, established and reliable servers are downgraded to C and D. In case of conversations.im the FAQ [1] states: “Providers in this category cannot, if no registration is supported, or should not, because of bad properties, be used for registrations. […]”. If we have a look at the reasons for the downgrade:

• Sharing files is allowed up to an unknown size or less than 1 MB
• Shared files are stored up to an unknown size or less than 1 MB

There are no valid reasons to not considering a server just because this information is unknown. At least not in our humble opinion, because server admins may decide to not publish this information.

3.     The complete process is not fully automated. Only because a server is good / bad now does not mean that it is good / bad in the future. This especially applies for the max upload size. One of the reasons that perfectly fine servers are downgraded now. Therefore, a new concept would be required to probe the server regularly. But there is no easy way to probe e.g., the BUS-Factor. So, there is no real way to fully automate the process. Therefore, information collected manually should either be removed completely, or at least not used for grading decisions like the upload quota.

4.     But even if the process were automated, we, as client devs, would still have to ask server operators for permission before adding a server to our in-client list. Why? Because we do not want to hit servers with huge load. As stated in [2] we cannot rely on anybody ever asking the server operators if they even want to be on the list (conversations.im, for example, does not want to be on this list). We therefore recommend removing all servers with unclear permissions from the list.

5.     Let’s assume that all these problems are magically gone, and every server operator granted permission for any client to use their server with any load. Would be great right? Well: no. We would still need to check if we know the server operator, if they use up-to-date software, how good the server is maintained, how fast they react to problems, how involved they are within the XMPP community… Why? Because we want to know the server operators. We want to ensure that we only recommend servers that are definitely compatible with Monal or at least responsive if problems occur.

Again, let's assume for a moment that all servers work perfectly fine, are updated asap and have enough free resources to handle many new users. Would be great again, right? Unfortunately not, as described above server admins were not asked before they were added to the list.

So can we at least rely on the correctness of the provided information? Maybe, but maybe not.

Let's have a look at a server, that we knew before and therefore did not select at random: anoxinon.me

On the xmpp provider page for this server it states that it is “Run privately” [3]. If we have a look at the source of the information [4] it clearly states that they are “[…] gemeinnütziger, nicht-kommerzieller Verein […]” [5] in other words they are not run privately, but by a non-profit association. So even the already provided information is not in a state where we could trust them. We would therefore need to recheck all provided information before usage in our client anyway.
We informed one of the five members of the upstream repository about this mismatch just after the official XMPP providers release.

So the remaining question is: What does the XMPP providers list solve? It gives users a hint of security, stability, availability!? Not really. It only provides vague knowledge to the current compatibility. That’s all. But new XMPP users will see this official list with rankings as the official guide to good and awful servers. So at least from our view the list is at least for XMPP users misleading and the grading should be removed unless there are real reasons to not recommend a server.

Conclusion: Monal will not (or at least for the foreseeable future) use XMPP providers. We will instead keep on using well established servers within the EU with a good reputation, with server admins that update their servers asap and help debug errors. And of course we always ask the server admins for permissions before adding their server to our list. Some operators do not want more load on their servers and some want a direct contact in case that something has to be discussed. We believe that this approach is the way to go to improve the overall experience for our users.

Friedrich Altheide, Thilo Molitor

[0] https://providers.xmpp.net/

[1] https://providers.xmpp.net/faq/#client-to-server

[2] https://invent.kde.org/melvo/xmpp-providers/-/issues/33

[3] https://providers.xmpp.net/provider/anoxinon.me/

[4] https://invent.kde.org/melvo/xmpp-providers/-/commit/f16fe977d18895acd8537535794fc8c772e166d7

[5] https://anoxinon.de/ueber-uns/

[tmolitor-stud-tu]

Where is your current list located and how can it be used at the moment? I could not find that feature in the latest release.

https://github.com/monal-im/Monal/blob/develop/Monal/Classes/RegisterAccount.swift#L30