Idea for the new location stack

[ale5000-git]

@mar-v-in
Is it possible to make it hybrid? (by including the most important backends inside but still allow external backends).

• Although backends are limited in new Android versions it is possible to unlock limitations by placing xml files and overlays in the system partition.
• Dropping backends completely waste a lot of works by other developers regarding backends.
• Some backends can never be integrated because they contains less privacy oriented code but still it may be useful for some people.
• The Mozilla backend may not working correctly in some countries, this isn't related to the new code but related to the Mozilla database.

[mar-v-in]

• I recognize that a lot of developers put efforts into creating backends for UnifiedNlp, however I don't see a way that allows us to continue using them. Some of the code certainly can be directly integrated into GmsCore, so it's not lost entirely.
• I am soon going to add more features to the location stack like activity recognition and geofencing. These require a reliable network location stack. The issues I've seen with the multi-process approach in the past would very likely cause severe issues to those features as well.
• I am aware that some backends for non-free network location servers have had better quality than the Mozilla database can provide. The best way to resolve this issue IMO is that we work on improving the Mozilla database. I plan to add features to make it easy for microG users to contribute to the database.

[Iey4iej3]

Is it possible to make these backends as some sort of plugins? I know nothing on Android programming, but I see that Termux has a couple of plugins such as Termux:API and Termux:Boot. I suppose that the main APP and plugins could cooperate more intimately than arbitrary two APPs.

BTW, I opened another issue for LocalNLPBackend: #1941

[CoelacanthusHex]

* I am aware that some backends for non-free network location servers have had better quality than the Mozilla database can provide. The best way to resolve this issue IMO is that we work on improving the Mozilla database. I plan to add features to make it easy for microG users to contribute to the database.

There are also some great NLP modules that are open source and have no cloud function, such as https://github.com/Helium314/Local-NLP-Backend

[ale5000-git]

@mar-v-in
This is my idea:

Even if backend cannot be kept how they were before isn't possible to still allow external "backend" in another way?

Most of the code could be in GmsCore.
For example the external "backend" instead of request GPS position or the Wifi scan directly could request it to GmsCore (using an API provided by GmsCore protected with a custom GmsCore permission) in a way that Android doesn't detect the external components as origin but only GmsCore so the external "backend" doesn't need to request Android location permissions (or others).

So the backend itself just contains the code to construct the location, for example the Déjà Vu Location Service could be modified to remove the code to acquire directly the WLAN/WiFi AP and mobile/cellular tower data but instead get them from GmsCore.
Even the part that passively monitors the GPS should be delegated to GmsCore.

[ale5000-git]

I don't know in details what was used before but to communicate between apps there is: Android IPC Mechanisms

[mar-v-in]

@ale5000-git the problem is not only the backend accessing wifi information / doing wifi scan, but also that it runs in a separate process. Technically, GmsCore could load code from the backend into its own process, but that can easily become a nightmare when it comes to security, as the loaded code would get all permissions from GmsCore and access to all its private data (e.g. the backend would get access to the super token for accessing users' Google accounts). We did use AIDL IPC before, but that requires multi-process.

---

In the end, we've generally seen three kinds of backends:

• Backends that get information about the current location from some external service (Mozilla, Baidu, Trains)
• Backends that get information about the position of wifi / cell tower from some external service and then do something to locally calculate the position (OpenCellId, radiocells.org, Apple, Local Wifi, Local GSM, Merged)
• Backend that learn wifi / cell tower locations when GPS is available and then do something to locally calculate the position (Deja Vu, Local NLP)

Basically all backends fall in one of the three categories and that's also the only ones that make sense. However that also means that a lof of work was duplicated and/or features are missing in some of the backends that would've made sense and are already implemented in other backends of the same kind. Example: It doesn't make sense to try to localize based on Wi-Fi networks that are known to regularly change their location (like those in public transport). Some backends thus got code to filter those out, others didn't - and those that did probably have different lists of such wi-fi networks.

Having everything in one codebase will likely improve the overall quality, so I also do see some advantages of migrating everything in a single codebase (beside the purely technical requirements).

In case anyone wonders what the current implementation in GmsCore does:

• Fetches and long-term caches locations of cell towers from Mozilla
• Fetches wi-fi based location from Mozilla
• Fetches wi-fi location from supported train wi-fi networks

I'd like to add:

• Import cell/wi-fi database
• Integrated download of full cell database from Mozilla
• Learn cell/wi-fi location when GPS is available
• Upload/contribute data learned from GPS to Mozilla
• UI for manual report/correction of wrong data

With those things added, the GmsCore implementation would essentially end up including the features of most backends we've had before. Synergies in code will hopefully increase quality while keeping the codesize maintainable (e.g. learning wi-fi locations by GPS for local storing isn't too different from learning them to add to Mozilla database)

[Iey4iej3]

but also that it runs in a separate process.

@mar-v-in Is it possible to run in the same process (despite two packages)? I mean, just as how Termux plugins seem to work?

[Iey4iej3]

• Learn cell/wi-fi location when GPS is available

• Upload/contribute data learned from GPS to Mozilla

I am using TowerCollector for this now. Maybe this codebase could be adapted.

[ale5000-git]

While I agree that have everything together reduce duplication, but it also complicate the external additions.
Have to recompile the entire GmsCore for every single change is a big complication (not to mention that the signature change also cause issues, and you will lost app data), also it require the developer to understand GmsCore code that is harder compared to a single separate backend.

Learning wi-fi locations by GPS for local storing is really more important in my opinion because it can work without internet, and also help the ones that doesn't trust external servers for location (this could also be expanded to locally storing Mozilla responses in cases the user want both).

Also another problem is app data, for various test something I need to wipe GmsCore app data and if the database is inside it, this will make it more annoiyng.

Also what about Apple backend? Some people use it because it is more precise.

In summary: the current way improve the situation but I think it could be better to also find an additional way to allow external addition without the previous issues.

[chirayudesai]

I'd like to add:

@mar-v-in is there anything external contributors can start helping with in this list already?

[chirayudesai]

Another thing to note is that some of the backends use a license that may be incompatible with GmsCore's Apache2.

In the short term this means one can't easily copy over any code from those repos if and as needed.

In the grand scheme of things it'd mean either re-licensing by their authors, or a rewrite, as feasible.

[hieu-van]

I support this. Here for example, in Hanoi, VN, the radius of accuracy is never lower than 50 km, and the exact current location has never been less than 200 meters off the correct location.

Although Apple and Mozilla backends were often slow to acquire location, at least they were fairly accurate.

(Trying not to rant but) how can they even allow such absurd accuracy 🥲 The acquired location should be dismissed if the accuracy radius is more than 50 meters.

[Iey4iej3]

in Hanoi, VN, the radius of accuracy is never lower than 50 km, and the exact current location has never been less than 200 meters off the correct location.

BTW, currently, you could use Tower Collector to collect and improve the data.

Although Apple and Mozilla backends were often slow to acquire location, at least they were fairly accurate.

I don't understand this comment. If I understand correctly, the new location stack is using Mozilla location, but uniquely Mozilla.

[hieu-van]

Although Apple and Mozilla backends were often slow to acquire location, at least they were fairly accurate.

I don't understand this comment. If I understand correctly, the new location stack is using Mozilla location, but uniquely Mozilla.

I used Apple UnifiedNlp Backend before the location stack changed. It was good.

[tazer4]

I don't know much about programming, but is it possible to add it back the way it was switchable by advanced option for those who are using older versions of android? Freely switching between the new stack and old stack allowing for nlp backends on android 11 and under for example?

[Iey4iej3]

I don't know much about programming, but is it possible to add it back the way it was switchable by advanced option for those who are using older versions of android? Freely switching between the new stack and old stack allowing for nlp backends on android 11 and under for example?

This seems to be a reasonable workaround to avoid that everything "suddenly" breaks.

[ale5000-git]

It isn't possible because the location code was rewritten and the old code removed, having both previous and current code would require a lot of duplicated code and would cause a lot of bugs.

[Epicanis]

• The best way to resolve this issue IMO is that we work on improving the Mozilla database. I plan to add features to make it easy for microG users to contribute to the database.

Ended up in this thread trying to find a solution to that. It would be nice to have a way to start getting the wifi-based database updated again. I moved 2500km months ago and now whenever I try to get a location from Mozilla it tries to put me 2500km away, where I used to live. (This seems to be causing problems getting a GPS fix, system is having trouble figuring out which satellites it should be able to see based on a vastly wrong initial location, maybe?)

It's not clear if Mozilla's wifi database is maintained - I've tried submitting anonymous location updates for my wifi's BSSIDs but it doesn't seem to make any difference. It doesn't seem like there are any wifi-collecting clients any longer (Mozilla Stumbler is dead and gone, and Tower collector only seems to collect cell tower info.)

I tried to add LocalNLP to resolve the problem, noticed it was silently blacklisted (installs but can't be accessed), and went searching.

If microG was able to successfully correct/update Mozilla's wifi-based location database, that'd be a nice solution. (Adding LocalNLP-like functionality would be very nice as well.)

[Hoerli1337]

[bege10]

• Upload/contribute data learned from GPS to Mozilla

...and to OpenCellID as TowerCollector does.

The Local GSM backend with a local database filled with data from Mozilla and OpenCellID did a very goog job for me.

[IzzySoft]

Import cell/wi-fi database

That just made my heart jump: like @bege10 I'm currently using the Local GSM backend with a local database filled with data from Mozilla and OpenCellID, and create that database locally on my computer (it is then pushed to all devices running microG). With that available on-device, there should be an option to turn off MLS network calls entirely (thinking of roaming costs, for example). Contributing learned data to not only MLS but also OCI would be a nice thing, too – though I'd accept the argument that one could use Tower Collector for OCI (which I do anyway – especially on my "OSM mapping sprees").

[draekko]

For what its worth and my $0.02 being forced to use network based and accessing Mozilla data sucks. i'd rather have at the very least a localized method instead and reading from a local database as other backends did. I am not a fan of Mozilla at all. I have no intentions of cooperating with them either by having them collect data in some fashion. Plus i usually run in airplane mode 98% of the time and only have it on when i need to contact the outside world. In my use case i have lost a lot of functionality. Not a happy camper.

[derMart]

I also got hit by the fact that external location providers are not available anymore as I use my own to protect my privacy.
That external providers are not available anymore is a large privacy problem IMHO. While merging every possible thing into one codebase has advantages, a modular approach also has. One of them is that people could choose from a lot of possible options or implement their own without needing to recompile and merge with the existing code base where maintainers need to approve the changes and also take time to maintain them. This is a large effort.
I would love to see a direction where microg might provide a good internal location service out of the box (with of course please incorporating code from now external providers) but also has the ability to include external location providers.
This was IMHO one of the strength of microg project so far.

@mar-v-in Could you elaborate a bit more, what technical problems existed with the multi-process approach as you described above?

[Mer0me]

I was looking for a solution to track my children's phones without exposing their location to the entire universe. Unfortunately, Mozilla Location Service thinks my local GSM antennas are far away from home, hundred of kilometers. When Nextcloud PhoneTrack App can't receive GPS signal anymore, it relies to the mobile network position given by MLS, so the phone location tracking is impossible.
Is there a way to prevent the MLS to be used when GPS is not available ? I would prefer an "unknown position", than a false one.

[bege10]

I was looking for a solution to track my children's phones without exposing their location to the entire universe. Unfortunately, Mozilla Location Service thinks my local GSM antennas are far away from home, hundred of kilometers. When Nextcloud PhoneTrack App can't receive GPS signal anymore, it relies to the mobile network position given by MLS, so the phone location tracking is impossible. Is there a way to prevent the MLS to be used when GPS is not available ? I would prefer an "unknown position", than a false one.

Doesn't it help to disable MLS in the microG location settings?

[Mer0me]

Nope, maybe some sort of cache somewhere ? All is disabled in the microg location settings and OSMAnd continue to believe it's in the same place, hundreds of km from me...

[Sapiosenses]

@mar-v-in Could you elaborate a bit more, what technical problems existed with the multi-process approach as you described above?

In the commit comments for the GmsCore v0.2.28 changes, it was mentioned that the problem was modern android ROMs with very aggressive power-management tactics that kept killing the external UNLP backend system processes, making them unreliable.

So apparently the only reliable tactic to avoid that problem with modern android ROMs is to make such processes part of a privileged/system app (eg GmsCore), which does not have the same sort of restrictions applied to it.

Now we have the problem of the Mozilla location service being retired permanently in June 2024, so lots of changes will be happening in terms of UNLP very soon.

#2237