Implement Export All Bookmarks buttons for easier backup and share

[biodranik]

Currently, uninstalling Organic Maps also deletes all user-created bookmarks and tracks (see #3504). Users who did not backup their bookmarks, are losing them forever.

Because of the privacy (no EXTERNAL_STORAGE permission), and Storage Scope restrictions introduced in Android 11, it is not possible to store bookmarks in a separate, non-removable directory. iOS has a similar situation.

Proper online backup discussed in #622 is needed in the future, but implementing a privacy-focused, easy-to-use one-button-exports-all backup (with later import/restore) scenario is critically important.

Possible implementation:

All bookmarks and tracks can be exported as a KMZ file (zip archive) with multiple KML files inside it, thus they can be imported not only in Organic Maps but also opened and edited by any other software that supports valid KML specification.

Related issues:

• [android] User-selectable folder for bookmarks - adds online backup (NextCloud, Google Drive, etc.) #1871
• [ios] Bookmarks synchronization into iCloud #2678
• [android] Give valuable names to bookmarks files #585
• Merge duplicated KML/GPX folders and features on import #2220

[biodranik]

Files should be exported/imported in Google Earth compatible format, like this one:
BACRNKMZ.kmz.zip

https://developers.google.com/kml/documentation/kmzarchives

[quite]

(optionally) automatically writing out to a bookmarks file (of choice? picked by the user using Storage Access Framework?) whenever bookmarks are changed would be really practical. Then user can have it backed up by their own means. And this is fairly small to implement

[pastk]

(optionally) automatically writing out to a bookmarks file (of choice? picked by the user using Storage Access Framework?) whenever bookmarks are changed would be really practical. Then user can have it backed up by their own means. And this is fairly small to implement

Yeap this idea is discussed in #1871 (comment)

[cyber-toad]

@biodranik I had an idea to look into this task and would like to confirm couple of tech details before starting to go deeper.
I discovered a method BookmarkManager::PrepareToSaveBookmarks - looks like it is responsible for fetching all data (points, tracks) by categoryid (that is an item in Bookmark & Tracks list) and generating kmz file name for it.
So we need to implement something close to this method that will iterate for all categories and combines to the single kmz? I see that there are some tech items that you need to take into account for such file: https://stackoverflow.com/questions/74776494/how-to-add-multiple-kmls-into-a-kmz but is my understanding correct, that import of multiple tracks from single kmz is now supported in OM?
Also a one UI question - the "Export all" button should be at the bottom of the screen together with "Create new list" and "Import" items?

[pastk]

Another option is to save each bookmark list into its own kmz file - its a more flexible approach as e.g. a user might want to send only one list to someone.
A single kmz with multiple kml inside is good for a backup and sync only.

[dvdmrtnz]

Can Organic Maps use the "In my iPhone" folder to store the bookmarks files like Osmand does? Then it would be trivially easy to copy and paste all bookmark files to make backup copies.

[brunetton]

Another option is to save each bookmark list into its own kmz file - its a more flexible approach as e.g. a user might want to send only one list to someone.
A single kmz with multiple kml inside is good for a backup and sync only.

Indeed. I plan to use Synching to sync bookmarks to my computer, and I have to sync some lists with my girlfriend, but not all lists. I would really appreciate to have multiple KML files, one per list!

[biodranik]

Another option is to save each bookmark list into its own kmz file - its a more flexible approach as e.g. a user might want to send only one list to someone.

@pastk How is it different from our current implementation?

[biodranik]

Let's focus on this issue, and keep privacy in mind: exporting all bookmarks/lists at once, and then properly importing all bookmarks/lists at once. Use-case: easy backing up bookmarks or easy sharing them with another user/to a new device.

@cyber-toad

1. UI buttons are very easy to implement, let's leave them out of the scope for now while thinking of a better UX.
2. It may make sense to start from implementing import of multiple KML files inside a properly created Google Earth KMZ file. In this way, the required file structure will be learned in advance. Maybe there are some other apps that generate such files too. An important part here is to support the official KML specification for compatibility and to allow easy bookmark creation/management on desktops.
3. There is a related issue with the import at the moment: duplicate lists are not filtered and are always imported. Then users need to manually delete such duplicates. You may hit this scenario when you start testing. It would be great to implement a check if the already present list is exactly the same, or (ideally) already has all bookmarks from an imported list. An appropriate message can be shown. This issue is not a blocker, we may track it separately and implement it at any time.
4. Export should be done to a valid Google Earth KMZ/KML format, its structure is described in docs. It requires a specific file/folder structure.
5. Files are already stored internally in KML. A possible implementation may try to iterate them and properly add them into a zip archive. Please check if our current Zip wrapper API can handle that, and let me know if it can't. Btw, the current zip implementation is very old and doesn't compile on Windows. Maybe we need to migrate to minizip-ng or to miniz. It also can be tracked in a separate issue.
6. Another possible implementation is to iterate over categories in memory. Need to research which one would be better.
7. Tests are important. An example file from Google Earth can be stored in the repo, then imported by a test, exported, and then stored in the repo and byte-compared later when tests are run on different platforms.

[pastk]

Another option is to save each bookmark list into its own kmz file - its a more flexible approach as e.g. a user might want to send only one list to someone.

@pastk How is it different from our current implementation?

The current implementation requires a user to export each file manually which is very tedious. Here I suggest to export all (or multiple selected) files via a single button press, but they will be separate files still.

[cyber-toad]

The current implementation requires a user to export each file manually which is very tedious. Here I suggest to export all (or multiple selected) files via a single button press, but they will be separate files still.

So we have 3 options here:

1. Export all data to a single kmz: (+) easy to store/backup single file, for example user can send it to email/messenger and that's it (-) all tracks are in single file, user can share only all of them at once
2. Export all data at once to specific folder with separate file per bookmark item (+) user has access to separate tracks for sharing/post-processing (-) it could be more difficult to backup all data as user need to work with multiple files
3. Export all data with separate file per bookmark item and all added to the single zip archive (+) single file for backup, can unpack it and share/process separate items (-) need to add support of import from .zip to OM

I think that if we want to target just a technical backup, option (1) is easiest, but if we need partial sharing/post processing of data separate files in (2) or (3) might be more convenient for the user.

Let me know what do you think.

[biodranik]

I was talking about option (4): kmz file with all lists in it as separate kml files (not a single file). It is officially supported by Google Earth. An index doc.kml file should be created for that, you can read about it here: https://groups.google.com/g/kml-support-getting-started/c/OlZbJWcmRkw/m/n53XIErHluwJ

We can make it more flexible as pastk mentioned, by allowing to export any selected lists (not just one and not just all).

[cyber-toad]

@biodranik I see, thanks for the details.

So looks like the plan for this item is:

Minimal set of items required for release:

• Add support of import of multiple kml's stored inside single kmz according to spec https://groups.google.com/g/kml-support-getting-started/c/OlZbJWcmRkw/m/n53XIErHluwJ. Each nested kml should generate separate bookmark.
• Add support of import of all bookmarks as separate kmls to a single kmz file as mentioned in spec above
• Tests: verify that for chain [source -> import -> export -> target] target matches source
• 'Export all' button

Nice to have:

• Allow to select bookmarks for batch export
• Duplicate checking for imports to avoid duplicated bookmarks when file is re-imported

There is a tech question here - we need somehow to identify that this is a multi-kml kmz and instead of passing it to xml parser unpack inner kmls to some temporary path and then use existing code that do 'import all' for this folder?

[cyber-toad]

There is a tech question here - we need somehow to identify that this is a multi-kml kmz and instead of passing it to xml parser unpack inner kmls to some temporary path and then use existing code that do 'import all' for this folder?

[biodranik]

A zip scan should be performed to decide if there's one or more kml files inside. Note that a single kml can also contain images folder. We'll support it too.

[biodranik]

Thanks to @cyber-toad importing a KMZ with many KML files inside is already implemented. To finish this important backup task:

1. Implement a single KMZ export with all KML files inside bookmarks folder.
2. Add Export/Backup All button in iOS
3. Add Export/Backup All button in Android

[rtsisyk]

Are multiple files in KMZ already supported? I also linked #2220 as relevant.

[biodranik]

@rtsisyk yes, in #6613

[cyber-toad]

Thanks to @cyber-toad importing a KMZ with many KML files inside is already implemented. To finish this important backup task:

1. Implement a single KMZ export with all KML files inside `bookmarks` folder.

I started reading a code to implement this and have a question. For now most of the methods prepareCategoryForSharing/PrepareFileForSharing/SharingResult class/etc have categoryId as a single parameter.

What do you think is a better way to rework them to mark that multi-export is required: pass some special value (like -1), add new method/classes without argument for batch export or rework old methods to accept list of categoryId (and populate it with all ids in Android/iOS code)? Technically all of them will work, just want to review these ideas before implementation is started.

[biodranik]

@cyber-toad thanks for asking!

Looks like some functions are already working with collections (PrepareToSaveBookmarks, GetFileForSharing). It may make sense to add a new iOS/Android method like "ExportAllBookmarksAndTracks", which internally will get all collection ids and properly prepare a single KMZ file by using existing methods with patches to accept vectors/lists of collections. WDYT?

[rtsisyk]

Meanwhile, OpenTracks expects doc.kml in KMZ files and don't import it otherwise.

[rtsisyk]

What do we plan to do with GPX? Also pack to KMZ?

[biodranik]

Internally, we store everything in KML. So there is no issue with exporting of imported GPX tracks that will be in KML format. For exporting, we can either export a plain GPX (with a separate button in the UI), or export GPZ (a zip of the GPX, need to check which apps support it).

[cyber-toad]

All PRs are merged, looks like issue could be closed (I don't have permissions for it).

[biodranik]

@cyber-toad thank you very much and congrats for closing this epic issue! Some users demand exporting GPX now...