(A) "size" erroneous in map-catalog.json (B) NEW OSM dataset from Oct/Nov 2020 (C) Prep for future datasets incl Satellite Photos

[holta]

@georgejhunt

1. Should we fix "size": 16841445376 that appears 3 times in http://download.iiab.io/content/OSM/vector-tiles/map-catalog.json which is apparently drawn from https://github.com/iiab/maps/blob/master/public/2021-01-27/map-catalog.json ?

   • Line 21 under osm-planet_z0-z10_2019.mbtiles
   • Line 47 under satellite_z0-z9_v3.mbtiles
   • Line 75 under osm_africa_z11-z14_2019.mbtiles

2. Is 2020-10-06/modify_catalog.py the most recent and best way of doing this?

   Do you have any recommendations here, in general?

3. In contrast, 16841445376 appears only once (on Line 124, under osm_africa_z11-z14_2019.mbtiles) in /library/www/html/common/assets/adm-map-catalog.json ~= https://github.com/iiab/iiab-admin-console/blob/master/roles/common/files/map/adm-map-catalog.json — apparently because it's auto-created by something like https://github.com/iiab/iiab-admin-console/blob/master/roles/cmdsrv/files/scripts/osm-vect_v2_finish_install.py ?

4. NEW OpenStreetMap datasets from Oct/Nov 2020 are now posted to https://archive.org/details/osm-vector-mbtiles that we should take a look at in coming months (when we can!) in service to low-income communities worldwide and their students who will greatly benefit...

   We should lay out a rough plan for 2022 so these communities get what they want (is Sentinel satellite data...essentially auto-updated at the moment?!)

Thanks!

Historical footnote:

• PR How can IIAB operators install the latest /library/www/osm-vector/maplist/assets/regions.json ? (catalog of downloadable maps, e.g. from d.iiab.io) [document workarounds for IIAB 7.0] #1612

Tangentially related:

• [OSM] Failed to install/reinstall OpenStreetMap on iiab 7.2 version via ./runrole #3064
• PR osm_vector_maps was repeatedly failing to install, so use 'timeout: "{{ download_timeout }}"' with get_url (in 10 roles!) [& change IIAB's 'download_timeout: 200' to 100 sec] #3083

[tim-moody]

there are two catalogs because the osm module and adm cons do the math differently, and the difference with respect to size reflect the different math.

[holta]

there are two catalogs because the osm module and adm cons do the math differently, and the difference with respect to size reflect the different math.

Thanks for context. I haven't dug into the details, but just FYI "size": 16841445376 is marked as identical in both files (all 3 + 1 = 4 cases) so this doesn't look like a math issue...?

1. "size": 16841445376 appears 3 times in map-catalog.json
2. "size": 16841445376 appears once at adm-map-catalog.json#L124

PS I revised the quick explanation of adm-map-catalog.json in point 3. above, and it's probably not 100% perfect (!) but hopefully at least clarifies a bit — towards helping everyone sort this all out for 2022 & Onwards.

[tim-moody]

I'm not clear on what the issue is here. There could be as many as three.

1. "size": 16841445376 has the wrong value

What should the value be and why do you think so? Is it just that three things probably don't have the same size?

2. This size appears three times in one catalog and once in another

3. The tiles files could be upgraded.

Is there also a question about synchronization between the repo and d.iiab.io?

[tim-moody]

The last question is why does it matter?

[holta]

The 3 issues in the subject line (A), (B) and (C) are to get offline communities and offline students of all kinds ready for the maps they need in 2022 and many years beyond.

Item (D) is definitely less important, but could be added as appropriate: should the stale "url" field be removed from iiab-catalog.json in future — if perhaps this information from soon-to-be-5-years-ago (perhaps) proves no longer irrelevant? An example from https://github.com/iiab/maps/blob/master/public/2021-01-27/map-catalog.json#L78 is here:

      "url": "https://archive.org/download/en-osm-omt_africa_2017-07-03_v0.23/en-osm-omt_africa_2017-07-03_v0.23.zip",

Item (E) that @tim-moody (almost but not quite!) suggests above: Should iiab-catalog.json itself possibly be hosted on GitHub, if not now, then perhaps sometime in the future? Visible versioning might greatly help field communities in future, to show lineage of different .mbtiles files and Map Pack offerings available to them each year and in recent years. (Possibly not needed in 2022? But curatorial/library-oriented implementation gurus greatly appreciate this kind of choice + transparency of this nature...if and when it's achievable!)

Item (F) can happen along the way (the social glue that holds everything together) and is implicit, and rarely talked about (until it's too late, and then "Where Are The Docs?" is suddenly the most popular question, when they don't exist...)

Specifically: which community docs and/or pipeline automation might be most important to make this quality mapmaking welcoming to new talent contributors over the coming ~5 years, helping @georgejhunt and others make this extremely valuable contribution fresh every 1-2 years or so, and for many years to come?

RECAP: Routine technical glitches can easily be solved now or in due course. At the same time, the higher-level purpose should be protecting our foundation so that in 5 years from now...the extremely useful foundation we've built over the past 5 years...is honored and disseminated increasingly broadly (and inevitably in new ways, that we cannot yet predict, as 2020 becomes 2030...)

[holta]

Some catalog and hosting/speedup and menu-def cleanups may remain in coming days/weeks, e.g. relating to things like:

• Download speed for maps -- Bluehost vs Digital Ocean [OSM] #3111

But the larger task is now done!

Thanks to all the work below, from @georgejhunt and @tim-moody:

• PR Programs used in the generation of 2020 version maptools#4
  • https://github.com/iiab/maptools/blob/main/README.md from
  • https://github.com/georgejhunt/maptools/blob/main/README.md
• PR Maps7.3--Uses 2020 OSM data and 2020 satellite imagery maps#45
  • SEE package-lock.json & package.json within:
  • https://github.com/iiab/maps/tree/master/osm-source/pages/webpack
• PR Branch Maps7.3 -- includes 2020 OSM data and 2020 satellite images #3165
  • Code for generation is a Jupyter Notebook at https://github.com/georgejhunt/maps/blob/maps7.3/osm-source/jupyter-code/maps.ipynb
  • For "2020" (this version) https://github.com/georgejhunt/maps/blob/maps7.3/osm-source/jupyter-code/README.md
• PR upgrade maps to 2020 tiles iiab-admin-console#478

[holta]

Just to clarify now that the above PR's are all merged, progress is ongoing here:

• PR host maps on github #3192
• PR GitHub host maps#54
• Garbage in what should be a source file resources/regions.json & discussion of cities1000.sqlite maps#55
• PR Upgrade presets to 2020 maps iiab-admin-console#479
• PR only one map url iiab-admin-console#481
• PR patch older map menu defs iiab-admin-console#482
• PR change default map source to contabo iiab-admin-console#483
• PR remove maps regions.json iiab-admin-console#484

[holta]

Huge progress continues:

• PR remove maps regions.json iiab-admin-console#484
• PR Revert "install fonts to '/viewer/assets/'" #3196
• PR upstream fonts.css does not have mapping to 'common' #3197

[holta]

Question(s):

• Is geodb.pl the only (or most important) thing left we need to migrate from tools.unused to https://github.com/iiab/maptools ?
• Discussion of how to evolve our ~26MB cities1000.sqlite for City Search is ongoing — with initial notes here:
  • Garbage in what should be a source file resources/regions.json & discussion of cities1000.sqlite maps#55
• Probably there are other items on the checklist arising from Thursday morning's call (http://minutes.iiab.io) that I'm forgetting !

[holta]

Subtle font adjustments yesterday and today — Etc:

• PR host maps on github #3192
• PR Revert "install fonts to '/viewer/assets/'" #3196
• PR upstream fonts.css does not have mapping to 'common' #3197
• PR Revert "upstream fonts.css does not have mapping to 'common'" #3198
• PR import iiab version for comparision. maps#56
• css issue current master maps#57
• PR copy fonts.css to maps assets #3203
• PR osm-vector-maps: Lint + clarify for readability e.g. map_installer_url -- and speed up initial install of 48MB planet_z0-z6_2020.mbtiles + 25MB satellite_z0-z6_2020.mbtiles #3204
• PR Refine /usr/bin/iiab-install-map-region for *2020*.mbtiles & Contabo maps#58
• PR Begin updating osm-vector-maps/README.md e.g. "How to upgrade IIAB Maps" (ENTIRELY AT YOUR OWN RISK) #3205
• PR Updated & Friendlier docs/README.md [IIAB Maps technical summary, OSM] maps#59
• PR Copy geodb.pl for cities1000.sqlite ⁠— from https://github.com/iiab/maps/blob/master/tools.unused/geodb.pl maptools#5
• Can we improve IIAB Maps city search in 2022? [OSM] #3206
• Please clarify the use of osm_size within the code base. [& discussion of init.json browser viewport, centered on lat/long + zoom-level] maps#60
• ol.js & olms.js looking for a .map file? iiab-admin-console#485
• PR add osm constants #3207
• PR use iiab constants and whitespace maps#61
• PR RFC Osm pylint maps#62

[tim-moody]

still needed?

[holta]

Good enough!