Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bookmark synchronization/online backup #622

Open
Tracked by #2082
matrixik opened this issue Jun 18, 2021 · 85 comments
Open
Tracked by #2082

Bookmark synchronization/online backup #622

matrixik opened this issue Jun 18, 2021 · 85 comments
Labels
Bookmarks and Tracks Bookmarks, imported tracks, and KML, KMZ, KMB, GPX, GPZ import or export Epic A large body work

Comments

@matrixik
Copy link

matrixik commented Jun 18, 2021

I see that it was disabled because of proprietary server #23 (comment)

Maybe it would be developed to use other opensource backend? I'm thinking it would be nice to be able to send backups to external git repository.

I see two pluses:

  • anyone could host it anywhere (private server or private repo on Github etc.)
  • if I remove something (or kid do something...) by accident then I can revert this change in repo and force import

Minus:

  • need git (don't know if there are libraries that you could import in app)

Also #613 could be stored in it.

@GLLM
Copy link

GLLM commented Jun 18, 2021

Just my 2 cents ... this could be made compatible with Nextcloud :)

@biodranik
Copy link
Member

This feature should be user-friendly, with privacy in mind. You can help us to plan it :)

@Surendrajat
Copy link

Surendrajat commented Jun 19, 2021

An automatic backup feature to a location of user's choice using Android's Storage Access Framework or something similar is one way to go. It's easy to use and compatible with any cloud sync/drive app that can either sync a local dir or provide it via SAF. That way it won't require adding support for some cloud service directly into the app.
Few of the apps that support this feature are Aegis, Wallet and Super Backup (closed-source).

@jancborchardt
Copy link
Contributor

Hi from Nextcloud! :) cc @eneiluj and @tacruc of the Nextcloud Maps app: https://github.com/nextcloud/maps/

We would love to help make it possible to sync favorites between Organic Maps and Nextcloud Maps so people have a great open source way to save their private maps info. We could even do more sync like searches and routes in the future.

@eneiluj and @tacruc can explain the more technical aspects of Nextcloud Maps and how we could sync favorites with Organic Maps. :)

@pataquets
Copy link

Just dropping https://remotestorage.io/ as a backend to consider.

@tacruc
Copy link

tacruc commented Aug 11, 2021

HI together,
I wanted to give the Nextcloud maps favorite API description:
https://raw.githubusercontent.com/nextcloud/maps/master/openapi.yml
Which can be inspected using e.g. https://petstore.swagger.io/.
I'm happy to help, if there are any questions or problems.

@penguin86
Copy link

HI together,
I wanted to give the Nextcloud maps favorite API description:
https://raw.githubusercontent.com/nextcloud/maps/master/openapi.yml
Which can be inspected using e.g. https://petstore.swagger.io/.
I'm happy to help, if there are any questions or problems.

Thank you very much, tacruc! It's exactly what I was searching for!
I recently switched to Nextcloud and have the urgent need to open my imported geobookmarks in Organic Maps. I tried to clone the OM sources and build the app, just to have an idea of the complexity of the implementation, but didn't succeed... I'm a little rusty at Android development. Maybe I'll re-try in the future.

As a temporary solution, thanks to your API definition, I was able to develop a very basic app that allows to open the Nextcloud Maps geobookmarks in any android app supporting Geo URI (my use case is Organic Maps, but Google Maps and many others are supported...). Don't expect bells and whistles, it's the bare minimum to get me on the road again, but I hope it can be useful to someone.

Screenshot

If anyone is interested, it can be downloaded from the github page and soon will be available on F-Droid.

I also hope the source code can be useful as an example to integrate Nextcloud Geobookmarks support in Organic Maps. In case anyone would like to try and needs any clarification on my code, I will be glad to help!

@keunes
Copy link

keunes commented Sep 16, 2021

Hi @matrixik @tacruc @GLLM and all others interested in this,

Since I understood feature requests had to go in the 'discussion' section, I wrote a feature request for synchronisation with Nextcloud Maps. Happy that there's an issue for this as well :) (more visibility, easier to track).

Thanks @penguin86 for adding the reference to this issue in the discussion thread, and for creating the Nextcloud Maps client. I hope it'll serve as inspiration and help for implementing synchronisation in Organic Maps!

I want to have a single map app that offers offline service, allows me to submit map contributions and does routing. Organic Maps has all that and (I couldn't try it out, but I suspect that) the Nextcloud Maps client won't be able to do all that. So synchronisation between Nextcloud Maps and Organic Maps would be the best solution for me.

Unfortunately I don't have development skills, so I can only encourage others :)

@ElTopo
Copy link

ElTopo commented Oct 9, 2021

Another idea: optinally, use a user-defined folder for saving/updating bookmark lists as text file(s), so the user can use a private git repo for commit/checkout/share bookmark lists.
The user can use github private repo for sharing bookmark lists between devices.

@biodranik
Copy link
Member

Why git repo? What are the benefits?

@ElTopo
Copy link

ElTopo commented Oct 9, 2021

@biodranik I found that git is better than general clouds (e. g. NextCloud) for syncing text files, and I automatically get the features such as keeping file histories, merging changes, etc., plus there are lots of git services that provide free private repos.

@biodranik
Copy link
Member

Do you plan to edit and merge it manually?

@ElTopo
Copy link

ElTopo commented Oct 9, 2021

@biodranik if OM uses text file for bookmarks, yeah I will. I have some old backups of places from other map apps I always wanted to merge into OM's lists.

@biodranik
Copy link
Member

KML uses XML format, it's not funny to merge manually.

@vbertola
Copy link
Sponsor

I recently migrated from the old Maps.me and I am a bit worried since I discovered that OrganicMaps does not store bookmarks in a place on the phone where I can just copy and backup the files manually; in my experience, Maps.me used to crash relatively often and sometimes it would destroy the KMB file currently in use, deleting dozens of bookmarks at once. So it was pretty important to backup bookmarks often.

On how to do it, another option would be to have a way for OrganicMaps to store or automatically export bookmarks into a user-defined public folder on the phone, either as its primary bookmarks storage or as a form of automated backup whenever a bookmark is updated. This way, users could use their own synchronization/cloud app to sync that folder to a server, without the need for OrganicMaps to support and implement many possible protocols and systems for cloud storage.

@biodranik
Copy link
Member

It's an Android 11 requirement, that we can not store bookmarks in the user-accessible folder anymore. Maybe it could be possible to create a button with "manual backup to the specified folder", which will request permissions from the user. But it should be investigated first. @rtsisyk what do you think?

@cxt666
Copy link

cxt666 commented Nov 1, 2021

One thought: Would it be feasible to start with an option to store a Sync File (CSV, KML, KMZ,...) on a configurable storage via FTP/SCP/WEBDAV. The Storage target might be provided by the user (Server,Path, User, Pass) in the first step, maybe later a "native" Solution can be integrated.

@biodranik
Copy link
Member

The default solution should work for most users and be as simple as possible. That's why "geeks-only" solutions are not in the priority (and they actually may require a lot more work).

The most complex part is authentication. It would be great to reuse an OSM account, for example, but forcing people to register in OSM for backups is also not good.

Do you have any ideas on how exactly the sync/backup should behave for non-technical users?

@uschmelmer
Copy link

The default solution should work for most users and be as simple as possible. That's why "geeks-only" solutions are not in the priority (and they actually may require a lot more work).

The most complex part is authentication. It would be great to reuse an OSM account, for example, but forcing people to register in OSM for backups is also not good.

Do you have any ideas on how exactly the sync/backup should behave for non-technical users?

I know a lot of non-technical users having access to some nextcloud instance (hosted by family members, friends, university, company...). So i think having the option to sign into you nextcloud (with maps app enabled) and have a builtin sync is very user-firendly and would also satisfy a lot of us geeks.
Best Regards,
uschmelmer

@biodranik
Copy link
Member

There are several important parts for sync (remember, we want to support non-geeks too!).

  1. Auth. The idea is to use some very simple auth approach (ideally reuse OSM auth where possible, for example). We want to avoid running and supporting our own auth server and store user credentials. This is the toughest question.
  2. Encryption. Ideally bookmarks should be E2E encrypted. But there is a question about sharing with other people.
  3. Storage. Probably we can use Cloudflare or some other cloud provider, bookmarks mostly don't take much space.
  4. Merging conflicts. If you edited bookmarks on one device and then on another one, there could be conflicts and they should be properly resolved.
  5. Sync strategy. Save and upload whole file on every modification? Do not use kml format on the server, but some simplified key/value storage?
  6. Sharing. The coolest part, with the idea to use the same bookmark lists/sets shared between your devices and between other users too (your spouse, for example).
  7. Web bookmarks editor support (for the future).

@jancborchardt
Copy link
Contributor

@biodranik what could be useful and keep the maintenance work off of your side is to allow people to connect their accounts:

  • Nextcloud / WebDAV
  • Dropbox
  • Google Drive
  • etc
  • (Export bookmarks to storage could also be an option if people don’t want/have cloud services)

In addition to what @tacruc said at #622 (comment), for Nextcloud support you could use our single sign on library for mobile: https://github.com/nextcloud/Android-SingleSignOn/
So either Nextcloud could be used with the Maps app API, or even just as a data dump via WebDAV connection (like any other provider). When using the Maps app API, it could also work with our sharing functionality for what you mention in point 6.

@biodranik
Copy link
Member

This approach is only feasible if we somehow implement all auth-related stuff on the web, with only a few lines of code to open webview and get auth results back in the app.
Also, it requires implementing local merging of conflicts, as I wrote above. And we should stick to the current KML files format because these storages work with files only.

@pastk pastk added the Bookmarks and Tracks Bookmarks, imported tracks, and KML, KMZ, KMB, GPX, GPZ import or export label Feb 11, 2022
@xplosionmind
Copy link

@adegans your tone and your attitude are very disturbing and definitely not polite. Please, consider writing the same thoughts but in a more respectful and empathetic way. Frustration is not an excuse for being so aggressive.

You have the right to voice your thoughts and what you are thorn by, but nothing excuses using strong and assertive words.

@adegans
Copy link

adegans commented Sep 22, 2023

@xplosionmind sigh is all I can say to that.

If voicing frustrations and ideas is too "strong and assertive" for you guys then I have nothing else to contribute here.

@Friptick
Copy link

@xplosionmind Your own criticism of @adegans comes across as sanctimonious and censorious. This forum is not (yet) a kindergarten with a speech code.

@adegans is not being "aggressive". He is being entitled. He is seeing this project in terms of "us and you" ("You have some sort of plan", "we can wait for you to do your magic", etc). But again, adegans, why are you expecting anything at all from people who are completely unpaid volunteers? Put yourself in their shoes. You're not paying a penny and you're not contributing code. So even if they don't have a plan, what's that to you? It's the entitlement that irritates me. There is no customer relationship to be served here. As users, we are just takers. We are owed nothing at all.

If the frustration of not being in a commercial relationship is so bad, maybe go find a paid product to buy. Coz whining here is at best useless and at worse (as you can see) it will just annoy the people who are doing the actual work that we all want to benefit from.

@adegans
Copy link

adegans commented Sep 23, 2023

If being frustrated about the lack of progress or even a vague promise/milestone for a popular feature request is being entitled then so be it.

All I was pointing out and asking for is some clarity while providing logical input/suggestions on how this feature could work.

If that's being entitled then your last few posts whining about my supposed entitlement are no better.
Your as in the project, not you/us/we/whoever since you even interpret that wrongly from my posts.

So have fun hiding behind your "it's free so shut up" nonsense as I'm sure this topic would be just as lengthy and slow as when everyone paid $10 or even $20 for the app to buy it. With the minor difference that perhaps, just maybe there would be a roadmap with this item on it.

And with that, I sign off. Clearly my input and opinions are not wanted here. Waste of time. I wish you all a good day and sorry for trying to make suggestions and asking about things... Or, in your interpretation - being entitled. sigh

@alensiljak
Copy link

and sorry for trying to make suggestions and asking about things

One thing I will never understand is that instead of waiting for a year (or whatever), you or anyone else could have learned enough about Android and cloud development to start working on this feature if it is so important. Perhaps even complete it.
Why didn't you?

@Kassagi
Copy link

Kassagi commented Feb 17, 2024

Maybe add synchronisation with Proton Drive? Or create QR code to share?

It depends how in app it is stored. Can you provide me sample of bookmark I backend (I mean like app store it)

@tobiaalberti
Copy link

Could this maybe be implemented via the introduction of Organic Maps accounts, using Organic Maps servers with data that's end-to-end encrypted by default, or would this be too big an endeavour given the project's current resources/priorities?

@arigit
Copy link

arigit commented May 13, 2024

Could the OSM account be used for this? since the app already has and auth workflow for it.
Otherwise, the platform's native way to do this for those that are OK with it (google account, apple etc depending on the phone OEM) would probably be fine for 99% of the user base

@biodranik
Copy link
Member

Yes, of course. The main issue is not with the way to authenticate, but with other things )

CC @kirylkaveryn

@HansCronau
Copy link

@arigit, when you ask "Could the OSM account be used for this?", do you mean for authentication or for data storage/syncing?
Because I highly suspect that OSM accounts are not meant to be your personal data storage.

You suggest that 99% of the user base would be OK with "platform native" storage. I wonder if that is accurate, because one of the main reasons to use Organic Maps/OSM is to cut dependency on Google/Apple.

@HansCronau
Copy link

HansCronau commented May 15, 2024

Reading the discussion on this issue (and the title) I see three different use cases, in increasing complexity:

  1. one time export: for when users want to
    • move/copy bookmarks to another device (and/or possibly another app) once
    • share (a finished set of) bookmarks with another person (who may use another app) once
    • create a very simple restore point/backup once
  2. periodic backup: similar to 1, but with the important differences that
    • the goal of 2. would to be able to restore a recent or past state of the Organic Maps app/bookmarks, possibly on another device, but not to get bookmarks to another person or app. (E2E encryption would be possible for 2. but not for 1.)
    • this would require a "set and forget" solution, which ideally would at least allow existing sync-tools to upload the data to a safe location
  3. active sync: for when users want to view and edit bookmarks on different devices simultaneously, and possibly (or even probably?) in different apps and with different people
    • This is the hard one. While 1. and 2. could be kept as simple as (periodically) exporting a (KML or GPX) file to local storage and letting a human or generic sync-tool do the rest, this one requires as least some authentication and back-end architecture. @biodranik already wrote a comment on that.
    • While it may also the most complete and noob friendly option, it is less clearcut and deserves some serious user research into which devices, apps, and people users actually want to sync to. The answers to this could for example make the difference between going for an official OrganicMaps back-end (which could be made very noob friendly, but would also be a lot of work, possibly for nothing if users want sync to another app) and choosing an existing back-end solution like Nextcloud Maps (which could mean less development and maintenance for Organic Maps, but might be useless if most users don't want to get themselves a Nextcloud account).

Because 1. and 2. could be combined in a single solution that would be clearcut compared to 3. and retain some value by itself even if 3. someday becomes reality, I would suggest splitting off "backup and export" to its own issue and leaving this issue to focus on the bigger solutions to 3. active sync. @biodranik

@kirylkaveryn
Copy link
Contributor

kirylkaveryn commented May 15, 2024

Reading the discussion on this issue (and the title) I see three different use cases, in increasing complexity:

  1. one time export: for when users want to

  2. periodic backup: similar to 1, but with the important differences that

  3. active sync: for when users want to view and edit bookmarks on different devices simultaneously, and possibly (or even probably?) in different apps and with different people

  1. Is already implemented with export all bookmarks feature.
  2. Is is under development on iOS using the iCloud to skip authentication at all. You can check the related PR [ios] Base implementation of iCloud Synchronization #7641
    This implementation have some limitation, but covers the wide range of use cases.

@arigit
Copy link

arigit commented May 16, 2024

@HansCronau I think you are right RE OSM not supporting this.
RE an OS-native way to persist a list of favorites, on android many apps support the "backup to the google account" option, it's available to any app, and it's encrypted as per the api description. The concern of many of us with googlemaps is about sharing your location continuously, learning your places and behavior etc. None of this applies to backing up a list of coordinates which they cannot read or process (at least in theory), I'd think.

Maybe a solution like @kirylkaveryn mentions but for android, using google drive, could be a useful trade-of,

@Friptick
Copy link

For this project I cannot understand why anyone would think a solution involving proprietary Apple or Google software is a solution at all. It's not. This projects styles itself as user-centric and privacy-respecting. If people want Google and Apple spyware mapping, they have it already. They're not here.

@biodranik
Copy link
Member

  1. Most users don't care, they just want to backup and sync out of the box.
  2. For those who care, a custom solution is needed.

@Friptick
Copy link

So, apparently:

"No ads, no tracking, no data collection, no crapware."

is now:

Most users don't care, they just want to backup and sync out of the box.

For a project that calls itself FOSS and talks up privacy, the right solution to this problem is not to prop up the Big Cloud duolopy. It is surely either:

  1. P2P sync (i.e., make the data syncable via a file or folder endpoint - that's it)
  2. A FOSS cloud tool, i.e. Nextcloud

Actually, it's both of those things. The first is the one I want and it is not so hard to do. OsmAnd does it already.

@kirylkaveryn
Copy link
Contributor

kirylkaveryn commented Jun 24, 2024

solution involving proprietary Apple or Google software is a solution at all

The question is not about the tools that should be used for syncing the data but is about giving a user the choice and not and not making a decision for him in this case and be fair.

The current solution with iCloud for ios is fully opt-in and doesn't force users to enable it. Even though it notices users that it is an iCloud but not just "some sync that is best for you".

99% of iPhone users want to sync files using iCloud out of the box, without sharing any data with some thirdparties or making additional authentication. I think it is OK to give them this opportunity.

OsmAnd does it already.

Personally, I don't want to create an additional account in osmand backup system.

@HansCronau
Copy link

solution involving proprietary Apple or Google software is a solution at all

The question is not about the tools that should be used for syncing the data but is about giving a user the choice and not and not making a decision for him in this case and be fair.

While this is not plain wrong and may in fact seem very reasonable, I feel it overlooks the fact that there's the question of what choices to prioritise (and whether to develop any of the features discussed in this thread in the first place).

99% of iPhone users want to sync files using iCloud out of the box, without sharing any data with some thirdparties or making additional authentication. I think it is OK to give them this opportunity.

I understand that you are using a figure of speech with the "99%" statistic, but it makes me wonder if there is any real data on what Organic Maps users actually want. We're not talking about any iPhone user here. We're talking about Organic Maps users that (for one reason or another) also use an iPhone. A 80%+ statistic would frankly surprise me.

Also note that iCloud/Apple is a third party too.

@Friptick
Copy link

@kirylkaveryn

Personally, I don't want to create an additional account in osmand backup system.

Neither do I and I never mentioned OsmAnd's silly in-house backup thingy.

What I said is that OsmAnd makes a P2P endpoint available. So that users can use any sync tool to get the data in and out of another app.

This is what I do. Currently I sync a GPX file from QLandKarte (desktop) to OsmAnd. And it works, sort of. (The problem is colors and symbols, because GPX does not standardize them and so individual apps encode them in their own special ways.)

To get back to the point, I generally agree with @HansCronau's analysis. IMO this whole issue needs 2 separate solutions, to cover both normie users and power users:

  1. some kind of easy-peasy one-click cloud-based solution (personally I would prefer a FOSS project to support other FOSS projects, i.e. Nextcloud, rather than boost a corporate duopoly, but I understand that there is a pragmatic calculation to make here)
  2. a simple file or folder than can be synced 2 ways and will keep the state of the OM data up to date - i.e.:
  • the bookmark state changed? the file is updated
  • the file changed? the bookmark state is updated

@pastk
Copy link
Contributor

pastk commented Jun 24, 2024

99% of iPhone users want to sync files using iCloud out of the box

Surely many people would use it and will be very happy with it.

Some apps go a long way to provide users with the best possible privacy-respecting options, while other apps do less.

There are many "grades":
(1) A mainstream app that doesn't care about privacy, collects user data, etc.
...
(5) Doesn't collect data itself, but by default integrates with popular non-privacy friendly services.
(6) Integration with non-privacy friendly services is opt-in.
(7) Integration is opt-in and the app warns users about potential privacy implications.
...
(10) Several integration options with open privacy-respecting and self-hosted services only, all potential privacy risks are accurately disclosed and advice given how to mitigate risks, etc. etc.

While an integration with a mainstream service will be certainly enjoyed by many users - it would certainly dilute the "privacy champion" image OM has/had.

I'm writing it not as a critique (and I admire all the effort you've recently put into the iOS version! incredible!).

Its just to better understand why people might have a different perspective on it.
And to reflect where on this scale OM currently is and where it wants to be..

@kirylkaveryn
Copy link
Contributor

There are many "grades":
(1) A mainstream app that doesn't care about privacy, collects user data, etc.
...
(5) Doesn't collect data itself, but by default integrates with popular non-privacy friendly services.
(6) Integration with non-privacy friendly services is opt-in.
(7) Integration is opt-in and the app warns users about potential privacy implications.
...
(10) Several integration options with open privacy-respecting and self-hosted services only, all potential privacy risks are accurately disclosed and advice given how to mitigate risks, etc. etc.

Can the mainstream app (I think that mainstream != bad and FOSS != geeky) be respectful for both users who really care a lot about data privacy the users who just want things to work?

IMO the best solution will allow users to make a choice between the grades you describe, will respect their choice and will not force to use smth they do not want.
I understand the critics about the proprietary apples/google sync solutions.
And I'll do my best to help to implement additional sync provider to ios so user can select between at least two options.

@pastk
Copy link
Contributor

pastk commented Jun 24, 2024

So, apparently:

"No ads, no tracking, no data collection, no crapware."

is now:

Most users don't care, they just want to backup and sync out of the box.

This kind of mismatch between people's expectations and the reality of what OM actually is - is not uncommon lately, unfortunately.

I believe its largely because the common description of OM found in stores, on GH, web-site, etc. promotes an image which is far from being accurate.

I've created a discussion about this issue:
https://github.com/orgs/organicmaps/discussions/8559

@arigit
Copy link

arigit commented Jun 24, 2024

@pastk I think the argument above is outside of the context of the discussion. The original ask is to allow the list of favorite or saved locations to synchronize using the mobile OS tools offered to apps. So that if the mobile gets stolen or swapped, the data stays with you.
I do not think anyone is asking to share current or even past location with the OS vendor, so no added exposure to tracking. Further, the data in the favorites file could be encrypted with a user password for those that want that (like firefox does with with the password file). Also, in addition to using the OS services, conceivably other platforms could also be supported like webdav or other popular self hosted platforms, for those that want to do that (probably a small fraction of the userbase would use this though)

@pastk
Copy link
Contributor

pastk commented Jun 25, 2024

I do not think anyone is asking to share current or even past location with the OS vendor, so no added exposure to tracking.

Well actually I think that generally bookmarks are a more sensitive information than pure location data (depending on how one uses bookmarks of course). As basically bookmarks are location data (where one has been) often combined with comments which could be very sensitive like with who and where one met, where do friends live, etc.)

Further, the data in the favorites file could be encrypted with a user password for those that want that

It should be encrypted by default, unless a user explicitly opted-in to e.g. use the cloud to share their bookmarks with others (usually it'd be some selected "public" lists, not all bookmarks).

@kirylkaveryn bookmarks are encrypted when synced to iCloud, right?
(sorry I haven't been following the PR)

@tobiaalberti
Copy link

I don't know where the Organic Maps team stands on the idea of adding paid features. But I believe people would be willing to pay something for a well-built option to synch bookmarks across devices that's end-to-end encrypted by default (via a formal account like Firefox does it, or without like Brave does it).

I also believe end-to-end encryption should be central to any such effort in order to stay true to Organic Maps' current pro-privacy and anti-surveillance stance.

@trymeouteh
Copy link

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bookmarks and Tracks Bookmarks, imported tracks, and KML, KMZ, KMB, GPX, GPZ import or export Epic A large body work
Projects
Status: No status
Development

No branches or pull requests