Offer to download and install offline maps

[andrixnet]

I would like to request a feature that allows in-app download of offline maps (to make it easier and much more user friendly) for the mapsforge offline function.
Maps to be used: official mapsforge per-country maps : http://download.mapsforge.org/maps/
With their permission if required+obtained

An implementation similar to that of OSMAND map manager.

I have personally seen a good number of users (especially young would-be geocachers) discouraged by the complexity of steps involved in obtaining and installing an offline map.

Of course, with a warning about potential additional costs if using GSM data connection.

[Lineflyer]

I am with you when it comes to having more user friendlyness. However I doubt mapsforge would allow us as their server is explicitly stated as no server for mass downloading.

I could more think about publishing a good tutorial.

[AndersBillLinden]

I do not understand why the user has to specify a directory for offline maps, as he/she will probably not care where on the file system those will be stored. The same is probably true for "map themes directory".

[SammysHP]

Maps and themes can be shared between different apps.

[AndersBillLinden]

It would then be a great thing to have some folder name proposed/autogenerated by cgeo?

[andrixnet]

@AndersBillLinden please also take into consideration these facts:

1. map files may be quite large (hundreds of MB or more)
2. there is a good number of low and mid-range devices on the market that have either small storage space or a combo of several storage spaces with different sizes. The system default one may not be the one with the largest amount of available space. Also there may be an SDCard present bringing a significant storage boost.
   Automated choice by c:geo may not always be the buest guess or the user's preference. Thus the user should be given the possibility to choose.
3. Having the map files inside c:geo's own directory tree may restrict access to them to other apps.

One suggestion: choose storage device with most free space available (by default) but allow user to change it manually.
Autogenerate "maps/mapsforge" or "maps/osm" or simply "maps" (?)

[andrixnet]

Consider also the restrictions for apps to use SDcard on Android 5 and 6.
For example, even on non-rooted devices, Total Commander requests permission and will work on whole SDcard, so it is possible.
Look at OSMAND and it's options to allow you to set the preferred location for map data files for inspiration.

[SammysHP]

I vote against this request. There are many different sources for Mapsforge maps and there is no convention for a common directory. We can provide help in the FAQ or maybe even in-app (a link in the preferences to the FAQ), but we should not provide a function to directly download maps.

[andrixnet]

You could talk to Mapsforge people and offer something in the line of http://download.mapsforge.org/ (rather a fast mirror as suggested) in a user friendly way. (and I mean as c:geo function, rather then a FAQ item that gets used by at most some IT guys).

[SammysHP]

You want to host a mirror of mapsforge.org, OpenAndroMaps, Freizeitkarte and maybe some more? Great!

[andrixnet]

Well, maybe I can host such a mirror.

All jokes aside, the essential first step was meant to ask the c:geo people to ask the mapsforge people if and how such a thing might be acomplished with existing resources first, if not, what projected resources might be needed and the feasibility to acomplish this task.
A positive answer with existing resources would be very good news for c:geo.
A positive answer with additional resources necessary should result in a draft of required resources, something like:

• about X GB hosting needed
• estimate of Y single map file downloads per day

Based on this, one or more mirroring service might be set up much easier.

I am talking only about Mapsforge. Other map sources indeed should be noted as alternate maps and remain specified in the FAQ.

[SammysHP]

Most people that I know don't use the "official" Mapsforge maps. Including other providers would be mandatory.

[Lineflyer]

Consider also the restrictions for apps to use SDcard on Android 5 and 6.
For example, even on non-rooted devices, Total Commander requests permission and will work on whole SDcard, so it is possible.
Look at OSMAND and it's options to allow you to set the preferred location for map data files for inspiration.

I do not understand this comment.
You can have the offline maps stored on your SDcard also on Android 4.4+ and c:geo can access it there. c:geo has read permission for the SDcard and that is enough for this purpose and works flawless as tested and in use by mself.

Regarding us hosting an own mirror/download server:
I think this is a bit too much for our small team in terms of administrational effort and costs.
Furthermore as also @SammysHP stated, this will then demand to even include more and other sources over time.

[SwissPoPo]

It is not easy for everyone to download a map and extract it to the correct folder. An integration into cgeo would be helpful.

Some apps have already made an integration.
Take a look at the download page of Freizeitkarte or OAM. There are special download URI for these apps.
Oruxmaps, Backcountry Navigator, BikeComputer allows to download maps or themes. On selection the app will download the map/theme and extract it to correct configured folder.
Locus allows to instruct the app what do to. Example: Map-download, extract, activate map and Theme-download, activate theme.

It is also possible to integrate the full repository to:

• autodownload or offer a local map (BoundingBox)
• offer a list all maps inside the app
  http://repository.freizeitkarte-osm.de/repository_freizeitkarte_android.xml

[Lineflyer]

It is also possible to integrate the full repository to:
autodownload or offer a local map (BoundingBox)
offer a list all maps inside the app
http://repository.freizeitkarte-osm.de/repository_freizeitkarte_android.xml

If we implement anything for offline map management at all, this is IMHO more reasonable than hosting own servers/content.
I quickly checked freizeitkarte-osm.de but could not find any in detail statement about whether we have to ask for permission to link to their servers or not.

[andrixnet]

@Lineflyer I never meant to imply that c:geo team hosts offline maps mirror repositories by themselves.

@Lineflyer Android 5+ has introduced limitations on the permissions apps have to write to SD_card. They are limited to certain dedicated directories. Lifting this limit mostly requires root permissions. However some apps are specially written to ask and obtain permission (like Total Commander file manager). Again, I am describing this not as a developer, but from end-user point of view.

While on these systems c:geo will have write permission in a certain directory (and deeply nested in the directory tree), it won't have permission everywhere.
It's an additional user complication and hassle to download map files and make sure you place them in the right directory for c:geo.
Since we are talking about a scenario where one can get some maps through simple UI in c:geo, but also use alternate maps as downloaded files from other sources, it is clear that c:geo needs both read and write permission and that the place where they are store should be reasonably user friendly to work with...
I've had the same problem with OSMAND that only allowed me to store map files in certain directory on Android 6, but not in directory of my choosing.

[SammysHP]

While on these systems c:geo will have write permission in a certain directory (and deeply nested in the directory tree), it won't have permission everywhere.

But it has read permission. The user can place the file where he wants. Problems would only occur if c:geo would try to write a file (e.g. after downloading the map). The current situation is much easier to handle and should be simple enough for any user to perform.

[andrixnet]

Indeed.

[pstorch]

I would like to start on a download feature for Maps from http://www.openandromaps.org/ as mentioned on the mailing list.
Implementing an action similar to the Locus actions http://docs.locusmap.eu/doku.php?id=manual:advanced:customization:actions but simpler. I think we just need the URLs of the files (Maps and Themes) to download. No destination or post processing (after element). Just download the files and use them.
Regarding the destination I suggest using a directory next to the GeocacheData directory, e.g. Maps. The user has already the option to put it on internal or external storage. That would apply then for the map files, too. The current map directory might not be writable as mentioned above. So we would end up with two map directories:

1. a c:geo map directory (the one next to GeocacheData) which we know is writable and where we can download the maps
2. an extra map directory (the current setting) which might be a read-only directory from another app like Locus.

Suggestion for the action:

<?xml version='1.0' encoding='UTF-8'?>
<cgeoActions>
  <map src="http://download.openandromaps.org/maps/canada/Newfoundland.zip"/>
  <mapTheme src="http://www.openandromaps.org/wp-content/users/tobias/Elevate4.zip"/>  
</cgeoActions>

There could be one or many map and mapTheme elements.

With an action URL like cgeo-actions://http/www.openandromaps.org/wp-content/library/xml/Germany_Mid.xml

[Bananeweizen]

We could also just implement the locus scheme, and ignore everything we don't need. Pro: More simple for the websites. Con: We might catch more links than we intended (with non supported map formats on other sites etc.). I generally don't like the approach to force the website to implement yet another generator per new app.

[pstorch]

We could just use the Locus one, but it feels a bit like hijacking.
Defining a standard scheme for map and theme downloads would be nice though.

[SammysHP]

There are already download methods for Locus and Orux and I'd call the Locus-way a rather generic implementation. +1 for using it.

http://freizeitkarte-osm.de/android/de/mitteleuropa.html

[leobut]

Since c:geo is using mapforge, why not use the mapforge maps from mapforge.org (http://download.mapsforge.org/)? They have pretty much everything covered, but I don't think these maps are proper topo maps.

For topo maps I would suggest using the maps from openandromaps if possible: http://www.openandromaps.org/en/downloads/countrys-and-regions

[pstorch]

@ControlTheBit from the Mapsforge download page:

This server has limited capabilities and is not suitable for mass or automated downloads. Do not point any widely distributed apps to this server for map download. 

[leobut]

@pstorch that's right. But check out the second link:

Mirror Rechenzentrum der Hochschule Esslingen, University of Applied Sciences (fast)
--> http://ftp-stud.hs-esslingen.de/pub/Mirrors/download.mapsforge.org/

There is no warning about anything. Maybe one could just drop 'em a few lines via mail and ask for permission?

[SammysHP]

Once c:geo is able to handle the described XML files, they can provide such files if they want.

[pstorch]

While still thinking about it. Somehow as a user I would prefer to initiate the download directly in the app. Without opening the browser, navigating to a map and open the download link with the app again. So ideally I would download an index document (XML or json) with all the available Maps, present it to the user and let him/her choose one to download. Without leaving the App.

Regarding the Locus way: when we implement it, wouldn't a user expect it to work the same way? With destination and post processing actions?

[Bananeweizen]

Regarding the Locus way: when we implement it, wouldn't a user expect it to work the same way?

No, they wouldn't. Users don't know about these details, if I understood it right how that is used in Locus. For a user, it is clicking on a link, and some time afterwards a new map is downloaded and available. The XML, the steps etc. are never presented in the UI, or are they?

[pstorch]

You can also download tracks and waypoints with it, when I understand it correctly. But we would only support maps and themes.

[SwissPoPo]

If you reuse the Locus xml files please consider that they could contain proprietary locus themes, which are not compatible with cgeo. A separate xml for cgeo would be better.

Locus actions can also active a map and/or theme.

[SammysHP]

they could contain proprietary locus themes, which are not compatible with cgeo

Oh, right! I absolutely forgot that.

[pstorch]

Mirror Rechenzentrum der Hochschule Esslingen, University of Applied Sciences (fast)
--> http://ftp-stud.hs-esslingen.de/pub/Mirrors/download.mapsforge.org/

There is no warning about anything. Maybe one could just drop 'em a few lines via mail and ask for permission?

I've contacted Hochschule Esslingen. Let's see if they allow the usage of their mirror from within c:geo. I'll keep you updated.

[SammysHP]

Please don't include a list of available map sources! Why should we prefer a specific map? Let us just include a way to download maps via some sort of API (e.g. one of the described methods above).

[leobut]

Please don't include a list of available map sources! Why should we prefer a specific map? Let us just include a way to download maps via some sort of API (e.g. one of the described methods above).

I agree. The user should not care about where the map comes from.

What is interesting for the user though are the following things:

• Topo map yes/no --> if yes, maybe different distances between elevation lines?
• Country / region
• Size of the map in MB/GB

[SammysHP]

You misunderstood me. I meant that we should not include any list at all! There is not one single map source, there are many and we should not prefer one of them.

[samueltardieu]

There is not one single map source, there are many and we should not prefer one of them.

Why not? We are not a governmental body which has no right to favour a source over another. We can, and we do, make choices. Offering an "easy" source and allowing others to be used does not deprive the user of any possibility. What is the problem in recommending a map source, or in using a source as an easy example of what can be retrieved?

As an illustration, Debian package "build-essential" which is assumed to be present when building Debian packages contains "gcc" and does not contain "clang". And this is not a problem, since "clang" can be installed in addition or in replacement to "gcc".

[SammysHP]

OK, but then let's include at least OpenAndroMaps, Freizeitkarte and basic Mapsforge.

[pstorch]

We got an answer from hs-esslingen.de: we are allowed to use their Mapsforge maps download mirror as much as we want.

[SammysHP]

Can you please ask Freizeitkarte and OpenAndroMaps as well?

One open issue: Updates. We would need to find a way to detect updated maps and overwrite the previous map (even if the filename has changed).

[pstorch]

We are already in contact with the admin of OpenAndroMaps. I can ask for Freizeitkarte as well.
OpenAndroMaps and Freizeitkarte use the approach with the custom actions, means the user selects the map on the browser and the special link is redirected to c:geo App the map.
The FTP Mirror for the Mapsforge maps would be different. There we would need to crawl the directory and present the user a list of available maps. Or as @mucek4 mentioned on the dev list, we could maintain and host a list our self with the links to the maps.

Map updates (with different filenames, really?) needs to be handled, yes.

[pstorch]

Just saw that Freizeitkarte has also an index document with all available maps. Which we could read in our app. http://repository.freizeitkarte-osm.de/repository_freizeitkarte_android.xml

[SammysHP]

I thought they contain the version in the filename, but they are organized in directories: http://download.freizeitkarte-osm.de/android/

[pstorch]

Got a reply from Klaus from freizeitkarte-osm.de. His proposal is to use the repository xml file and present a list of the available maps directly in our app for download. This is more user friendly than telling the users to use their Browser to navigate to their website, choose the map they want and click on the action-url, which then opens our app again.
He also gave the hint, that we should take care that the Freizeitkarte maps need the right theme to look properly. At the moment we don't have a connection between maps and themes and don't know which ones fit together.
The more I think about it the more I want to discard the idea with the action-urls. I'll ask OpenAndroMaps if they can also provide such a repository file. For the mapsforge mirror from hs-esslingen.de we can either load the directory index and generate such a list from it or use the proposal from @mucek4 and create such a file our self. But I would prefer to scan the directory index, to also know the current timestamp of the map and the size. I think we should present these information which each map to the user.

[pailakka]

I'm producing Mapsforge compatible topographic maps from Finland (http://kartat.hylly.org, sorry only Finnish atm). It would be great for the Finnish users to include these to cgeo. I'm not however sure if it is feasible to include all of the "small" map providers to the suggested scheme of repository xml's.

These Finnish topo maps definitely require a specific theme to go with so it would be great if there is a) a way through indents to download the map file via a link b) associate map files with compatible themes

[pstorch]

Ok, there are two good reasons to keep the action-url method. With this we can support more map providers (like Finland maps from @pailakka) and it's required by OpenAndroMaps. If we would include the OpenAndroMaps directly in c:geo, the users wouldn't see the donation button on their homepage. Which would result in less donations and put the project at risk. This is a totally valid point.

So we can offer direct download in app for the Mapsforge and Freizeitkarte maps and support action-urls for all others. Still the question if we should invent our own action-url scheme or just support the locus action-urls. The latter would instantly open up all map providers which already have the locus action-urls, but we won't support it 100% the same way as locus (like the special locus themes).

[rsudev]

IIRC there are also action-urls for OruxMaps. They use the standard mapsforge library, therfore their themes are compatible.
But to avoid any interference, shouldn't we rather create our own?

[andrixnet]

Please take a look at OSMAND application and how it presents map managing, downloads and updates to the user (as user interface). I believe it is a good inspirational source for UI. c:geo would want to add an extra layer for map type choice, once a country is selected (as in mapsforge, topo, etc)

[andrixnet]

Regarding directories and permissions,
I am now using an S6 with Android 7.
For example, without root and any special tweaks, TotalCMD app asks for permission and I grant it to access all files (read an write) on the storage. (S6 only has internal emulated storage, no card slot).

Regardless where c:geo stores it's internal data, since map files definitely are LARGE files, the user might want to be in control where to store them, and I mean both storage device and directory.
And IMO would be best if manual addition of a map file to that directory continues to be recognized and used by c:geo (and mark it as non-online source so not to try to manage updates).

[andrixnet]

Since different map repos offer different listing help, I would suggest a robot on c:geo (or other) website running on cron, to inspec each repo and generate a centralized and unified listing that c:geo can then use to directly download files, rather then implenting repo1, repo2, repo3, etc listing specs.

[pstorch]

Since different map repos offer different listing help, I would suggest a robot on c:geo (or other) website running on cron, to inspec each repo and generate a centralized and unified listing that c:geo can then use to directly download files, rather then implenting repo1, repo2, repo3, etc listing specs.

Yes, somewhere we have to abstract the different map listings from repo1, repo2, ...
But I doubt that it is good idea to maintain a separate piece of infrastructure and a codebase for it.

[fm-sys]

Let's close this as it is implemented now...