Places: Improve details of the default world map

[lastzero]

As a user of the free Community Edition, I would like to have more detail included in the basic world map so that I can better browse my pictures by location.

• Since more than 95% of users opt for our completely free version without signing up as a sponsor, we unfortunately can no longer offer the high-quality commercial maps to everyone. Kindly note that they were also not listed as a free feature in our overview in the past, but were still available for evaluation purposes.
• To improve the situation, it would be possible to add more details like cities and lakes to the basic maps. The only problem is that we don't have the resources to do that due to our large backlog of feature requests. So this would be a perfect task for contributors! ❤️
• You can find the existing maps as JSON file in this directory: https://github.com/photoprism/photoprism/tree/develop/assets/static/geo

---

We are aware that advanced users could register "non-commercial test accounts" instead, but we think that would not be completely fair and MapTiler could then no longer offer them to those in need. Keep in mind that we have many more users than other open source projects that might encourage their users to do this. Likewise, using the OpenSteetMap development API is discouraged for consumer applications like ours, although some projects do it anyway. See our FAQ: https://docs.photoprism.app/getting-started/faq/#are-the-keys-for-using-interactive-world-maps-provided-free-of-charge

[filipnyquist]

I know that Naturalearth provide some public domain maps , might be worth look into https://github.com/nvkelso/natural-earth-vector

[lastzero]

We also created the existing GeoJSON file from public data. I can't remember the exact steps, but maybe there are hints in the Git commit messages or old GitHub comments. Probably downloaded individual files specifying the land, borders, and sea, and then merged them using a command line tool.

There are many websites and repos to choose from (but always check the license before downloading):

• http://geojson.xyz/
• https://geojson-maps.ash.ms/
• https://github.com/nvkelso/natural-earth-vector
• https://github.com/simonepri/geo-maps

👉 it might be worth automating this with a script so that it can be easily updated/improved later without having to start from scratch (again)

[lastzero]

For more clarity, we have renamed the included offline maps to "Low Resolution" in the "Settings" drop-down menu (previously "Basic").

[maciejla]

Hi, I think it would be nice to add a "Custom tileserver URL" option :) that way everyone can host their own tileserver e.g. only for their country
Also, I have some experience with OpenStreetMap, I'll try to generate a geojson that's feasible for this kind of use, but it may be hard to fit a map of the whole world into a few MB :)

[lastzero]

Hi, I think it would be nice to add a "Custom tileserver URL" option :) that way everyone can host their own tileserver e.g. only for their country

Generally yes, although hosting custom maps is a bit advanced. Considering all the other feature requests, we can't make this our highest priority. Feel free to remind us later next year! 😉

Also, I have some experience with OpenStreetMap, I'll try to generate a geojson that's feasible for this kind of use, but it may be hard to fit a map of the whole world into a few MB :)

Thanks, that's awesome! The JSON is not so big anymore when it is gzip compressed. Maybe there is also a way to use multiple files for lazy loading? This is the UI library we use: https://maplibre.org/

[maciejla]

Good news 1 - I was able to add some city names (place=city in OSM) without drastically changing the file size (1.6MB, 380KB gzipped)

Bad news 1 - I don't think it's possible to go much further with one single geojson file

Bad news 2 - I don't know how to do it better

Good news 2 - I'm willing to learn how to do it :)

So, I'm publishing the resulting geojson here - feel free to use it if you want :)
I'll try to figure out some kind of lazy loading now :)

(sorry for sending it in a zip, github doesn't accept json in issues for some reason)
world2.zip

[maciejla]

My workflow in short:

osmium tags-filter planet-latest.o5m  n/place=city -o test4.osm
osmfilter test4.osm --keep-tags="all name=" --drop-version --drop-author -o=test5.osm
osmtogeojson -m test5.osm > a.geojson

then delete osm id's, as they aren't needed in this case:

import json

with open('a.geojson') as f:
    data = json.load(f)

for feature in data['features']:
    del feature['id']
    del feature['properties']['id']

with open('b.geojson', 'w') as f:
    json.dump(data, f)

then merge with the existing file:

import json

with open('b.geojson') as f:
    data = json.load(f)

with open('world.json') as f:
    data2 = json.load(f)

data['features'].extend(data2['features'])

with open('c.geojson', 'w') as f:
    json.dump(data, f)

then minify:

jq -c . < c.geojson > d.geojson

[lastzero]

Thanks! I'll replace the file and add a script with the commands when I have a bit of time 💐

[lastzero]

I replaced the file and tested it but could not see any cities. So I assume that either the code must be changed or the layer is not visible for some reason?

[maciejla]

Whoops, my fault. I'll try to fix that now :)

[maciejla]

Honestly, I don't know how to make the geojson working with photoprism :(
However, I'm currently experimenting with vector tiles, and here's what I found out so far:

• It's quite easy to generate a vector map based on OSM: https://openmaptiles.org/docs/generate/generate-openmaptiles/
• It's quite easy to serve this map on demand: https://github.com/maptiler/tileserver-gl
• mbtiles file for Poland with MAX_ZOOM=7 takes 13.8MB, can be overzoomed quite reasonably
• If I'm estimating correctly, mbtiles file for the whole planet would take about 650MB. It seams reasonable to ship with photoprism / as a additional download
• Unfortunately, in order to generate a mbtiles file it's required to import OSM files into a postgresql database
• The import process takes quite some time (haven't measured it unfortunately)
• After the import, tiles can be quickly regenerated (270s for Poland on i7-3770 w/ 16G of RAM)

I'll try to create tiles for a bigger area (Europe probably), if that works out reasonably I'll try to create a whole planet :)

[maciejla]

I'm willing to generate and update these files as OSM data gets updated if the feature finally makes it into photoprism :) MapLibre seems to support vector tiles, so I'll try to display the map I'm currently serving with tileserver-gl in photoprism soon :)

[lastzero]

I know about OpenMapTiles, which are free maps provided by MapTiler. However, in the past there was a catch in the license so we could not use them. In the few minutes I just had, I can't find that anymore, but more research needs to be done to see if it's OK to use it in a production app (and bundle it), and how much storage that would require (so if our users would be OK with that too). Finally, it requires work on our end (or your contribution).

[maciejla]

Didn't knew about that, I'll check the license! Thank you for the warning :)

[lastzero]

We could also host these maps for our users (including CE), but that would not be offline like the current basic maps and also slower (not local, no CDN).

[maciejla]

I have read OpenMapTiles license, and it seems we can use it. However, it would be nice if someone with actual legal knowledge confirms that
From what I can see, code is licensed under BSD 3-clause, so that's not a problem. The style is licensed under CC-BY 4.0, so I think it's okay to use it:

The cartography and visual design features of the map tile schema (also known as the "look and feel" of the map) are licensed under the Creative Commons Attribution 4.0 license. To view a copy of the license, visit http://creativecommons.org/licenses/by/4.0/.

Products or services using maps derived from OpenMapTiles schema need to visibly credit "OpenMapTiles.org" or reference "OpenMapTiles" with a link to http://openmaptiles.org/.

For a browsable electronic map based on OpenMapTiles and OpenStreetMap data, the credit should appear in the corner of the map. For example:

© OpenMapTiles © OpenStreetMap contributors

For printed and static maps a similar attribution should be made in a textual description near the image, in the same fashion as if you cite a photograph.

Exceptions to OpenMapTiles attribution requirement can be in a written form granted by MapTiler (info@maptiler.com).
The project contributors grant MapTiler AG the license to give such exceptions on a commercial basis.

The catch in the license you are talking about probably refers to ready-to-use tiles at https://data.maptiler.com/downloads/planet/ (it's required to select "non-commercial personal project" or "evaluation and education purpose" in order to download old tiles for free, otherwise it costs $4000)
However, if we are generating them ourselves, it should be fine

[lastzero]

The catch in the license you are talking about probably refers to ready-to-use tiles at https://data.maptiler.com/downloads/planet/ (it's required to select "non-commercial personal project" or "evaluation and education purpose" in order to download old tiles for free, otherwise it costs $4000)

Yes, I think I got that far and then stopped because we didn't have the time or resources to render our own tiles. While the Community Edition could be considered non-commercial, German law might see it differently and it's not really personal anymore either.

Do you have any idea what hardware is required to do this, how long it takes, and how much storage is needed (for generation and the final tiles that can be distributed)?

[maciejla]

For Poland, my PC (i7-3770, 16GB of RAM, SSD) can do it. For the whole planet, I don't have an idea, but that's what I'm trying out right now :))

[lapo-luchini]

@maciejla if it needs more RAM I have 32GB… unfortunately I have little free time to experiment myself, but if you have a workflow to execute, I can certainly help.

[kovlo]

I can offer 70+GB RAM with ~20x2GHz threads (E5-2620). If you set up a docker or VM or script (for Ubuntu any version), I can run it for you to do some experiments.

[maciejla]

Thank you for offering so much resources! I also have access to 24c 128GB RAM server at school, so if the admin agrees, that's something I can experiment on :)

Here are my experiments so far (i7-3770, 16GB of RAM, 1TB Silicon Power SSD):

Whole download + import + tile-generation process for Africa (5.6GB .osm.pbf file) took about 8h 30m
For Asia (11.4GB .osm.pbf) I decided to break it down into steps:

• make import-osm took 41150s (about 11h 30m)
• make import-sql took 54295s (about 15h)
• make generate-tiles-pg took about 1h 16m
• all the other steps took just a few minutes

As for needed disk capacity after importing Africa and Asia (17GB .osm.pbf total):

• postgresql database (/var/lib/docker/volumes/openmaptiles_pgdata) takes 172GB of disk space
• openmaptiles cache directory takes 17GB of space

Resulting mbtiles file takes 72.5MB

Extrapolating that to whole planet (67GB .osm.pbf) means about 745GB of disk space is required, and that the resulting mbtiles file will take 286MB

EDIT: Above (striked-out) calculations are wrong. See next comment

Unfortunately, I think the biggest bottleneck I have in this whole process is the disk speed (often ~60% of CPU time is "wasted" in iowait state). I'm going to try enabling zstd compression in btrfs, as that may greatly reduce the amount of data that's needed to be read from disk (and of course, that means I don't have to move files out of that SSD 😅)

I'll also run a public tileserver-gl instance serving vector tiles out of the mbtiles file that I have generated :)

[maciejla]

Whoops, it seems the postgresql database gets cleaned when executing make import-osm. Discovered that a bit late 😅 It also seems I miscalculated the approximation of full planet mbtiles filesize.

However, I have successfully generated a mbtiles file for geofabrik extracts of Africa, Antarctica, Asia, Australia and Oceania. I have also successfully merged resulting mbtiles with tile-join (https://github.com/mapbox/tippecanoe)

Now I'm going to generate mbtiles for geofabrik extracts of Central America, Europe, North America and South America. Then I'll join them, and we'll have a full planet mbtiles file :) When I have the full planet file, I'll try to integrate it with PhotoPrism :)

[maciejla]

Confirmed, it is way faster with disk compression enabled (compress=zstd in btrfs). Previously, make import-osm took 11h 30m for 11.4GB .osm.pbf Asia file. Now, it took 5h 9m for 12GB North America file :)

[maciejla]

Only geofabrik extract left is Europe :) and I need some help :)
Unfortunately, I think 16GB of RAM is not enough to import and generate mbtiles for Europe (26.2GB .osm.pbf file) - my PC starts swapping which slows down the import process a lot

So, @lapo-luchini or @kovlo - I'll be grateful if you could execute following workflow:

• install docker and docker compose - depending on the distro, in debian sudo apt install docker.io docker-compose
• git clone https://github.com/openmaptiles/openmaptiles
• cd openmaptiles
• make
• ./quickstart.sh europe geofabrik

That will automatically create docker container and volume with the database, download the europe.osm.pbf file (26.2GB) from Geofabrik, import it into postgresql database and generate tiles. Then, send me the resulting tiles.mbtiles file, I'll merge it with the rest of the files I have generated and we will have the whole planet done :))

I don't know exactly how much resources is required, but it should be between 250-500GB of disk space, the more RAM the better. It will probably take a few days

Thank you very much! If you have any questions - feel free to ask :)

[lastzero]

@maciejla @HeikoGr @graciousgrey I feel like we can go with that, what do you think?

Maybe the colors could use a little more love. Buildings and street names are also missing. However, a more detailed mbtiles file with zoom level 15 should be available for testing soon. Funnily enough, the "original" mbtiles file from OpenMapTiles did not render correctly, while all self-rendered files you provided worked perfectly.

@HeikoGr Although I appreciate the idea behind pmtiles, using this format seems to cause a bit more overhead in terms of code size (additional deps required), development and testing effort. Detailed maps would be too big to host locally anyway and most users don't care where it's hosted, they just want it for free.

So I think we'll serve the maps from our server, and additionally put a global CDN in front of it to get the best possible performance. This doesn't cost us anything (at least for now), because we have enough free credits from users who signed up through the "Bunny CDN" affiliate link in our docs, see https://docs.photoprism.app/getting-started/using-a-cdn/.

[HeikoGr]

Looks good to me! Any optimization should be made afterwards, as this is an important feature.

pmtiles would also be hosted on a central server. But it is only one plain file without tile server. It's a very interesting approach, but you are right - you need another js library on the client side.

[lastzero]

An updated development preview will be available for testing soon! Unfortunately I could not configure the CDN due to problems with the host headers (the JSON response contains absolute links including the domain name).

[benjaminjonard]

I don't know if it is of any help, but someone regularly uploads updated mbtiles files here https://osm.dbtc.link/index.php?b=mbtiles&s=size&r=1

Edit : I could make it work with street names and buildings

[lastzero]

I don't know if it is of any help, but someone regularly uploads updated mbtiles files here https://osm.dbtc.link/index.php?b=mbtiles&s=size&r=1

Thanks for letting us know! That might have been helpful, but we now have maps with very high details (~160 GB compressed) and also can adjust the colors if needed, so I think that's as good as it gets:

[lastzero]

Based on feedback we received yesterday, the "Low Resolution" option has been re-added to the settings menu:

Kindly note that it will only be available if you have experimental features enabled under Settings > Advanced (reload the app for the settings to take effect). This is because we have received feedback/bug reports from users who felt that there is a problem with these simple offline maps as they do not contain any details other than land, water, countries and borders. On the other hand, it's worth having an option that works completely offline and improving it as time allows.

[lastzero]

I've spent a bit more time tweaking the colors. Also, I noticed that a few points of interest are missing on the map because they don't have a sprite (icon) specified. This might be something that contributors can help with later on.

[lastzero]

@maciejla @HeikoGr An update preview image with the build id 230606-dda00ba23 is now available for final testing. I've also configured a CDN for the map tiles, so performance should be much better for most users. If nobody complains, we will release this in the next few days, probably tomorrow.

[lapo-luchini]

Detailed maps would be too big to host locally anyway and most users don't care where it's hosted, they just want it for free.

I'd love (and prefer) to use self-hosted "17GB z12" files up there if/when possible, but that works too, in the meantime. I'd also love to help code the feature, but unfortunately I have too little time for my current "opensource pet projects" already so I fear that won't happen soon. 🥲

[lastzero]

@lapo-luchini I can imagine that a locally hosted option might be preferable for a few users who have particularly high privacy requirements, don't need too much detail and frequent updates, have enough storage, don't use the maps from the public internet, or optionally have an internet connection with high-speed upload, their own CDN, or host their instance with a cloud provider (in which case the privacy advantage is somewhat mitigated). Note that we don't make money from advertising or selling data to advertisers, so we neither have a team to analyze your data nor customers to buy it. IMHO this should impact the choices and tradeoffs you want to make.

Edit: That being said, you are of course welcome to fork and customize our software for your own needs, including to use other maps.

[lastzero]

Final testing for the upcoming release has started and the release notes have been updated to document all changes:

• https://docs.photoprism.app/release-notes/#development-preview

[Jeff254]

Am very excited about this release. It makes the Community Edition a very usable..

…

On Wed, Jun 7, 2023, 11:56 AM Michael Mayer ***@***.***> wrote: Final testing for the upcoming release has started and the release notes have been updated to document all changes: - https://docs.photoprism.app/release-notes/#development-preview — Reply to this email directly, view it on GitHub <#2998 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHNPNQW7EKVNXPDB23ABNRDXKA62BANCNFSM6AAAAAAS47KPTA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[loulou91]

Thanks you so much for this really great improvement of the CE edition!
For me it's a welcome gift ;-)

[gitinthebus]

I'm so glad this was finally addressed in community edition. The map was the main feature I chose photoprism over alternatives and that bare map before, without any names or features, was completely useless.

[HeikoGr]

I was curious if you have updated the maps recently and discovered that you now use api.maptiler.com instead of your own maps-server since #2106

@lastzero
to avoid an unnecesary invoice from maptiler:
was this by accident or a planned change?

[lastzero]

@HeikoGr The map style selector we added in #2106 is only visible if your instance got a valid maps key from us, e.g. because you signed up as a member. The free default map (see screenshot below) is loaded from our CDN, so you don't need access to the MapTiler API.

[HeikoGr]

I see. I thought the "default" in the selector was the old low res geo.json (because my browser wasn't fast enough with rendering).