tilelive-tmsource : Cannot set property 'id' of undefined.

[init-js]

I downloaded a single city's worth of data from Mapzen from which to generate a vector tile map. The data is the "raw pbf" data export option file, and mapzen extracts data from OSM.
( https://mapzen.com/data/metro-extracts/ )

I assumed I'd be able to substitute that file in ./data directory instead of downloading the albania default. I'm doing the manual steps, except I'm skipping the import-natural-earth, import-water, and import-osmborder. It's not really explained anywhere, as far as I can tell, why those steps are necessary -- what if I was making a non-earth map?

Anyways, at the point where it's generating the files, i'm getting what looks to be a missing 'id' field error, or a missing configuration where an id is supposed to be. Is this due to an unexpected input format of the files, or is it more bad yaml configuration?

$ docker-compose run generate-vectortiles
WARNING: Dependency conflict: an older version of the 'docker-py' package may be polluting the namespace. If you're experiencing crashes, run the following command to remedy the issue:
pip uninstall docker-py; pip uninstall docker; pip install docker
WARNING: The MIN_ZOOM variable is not set. Defaulting to a blank string.
WARNING: The MAX_ZOOM variable is not set. Defaulting to a blank string.
WARNING: The BBOX variable is not set. Defaulting to a blank string.
Pulling generate-vectortiles (openmaptiles/generate-vectortiles:0.1.1)...
0.1.1: Pulling from openmaptiles/generate-vectortiles
8ad8b3f87b37: Pull complete
751fe39c4d34: Pull complete
ae3b77eefc06: Pull complete
7783aac582ec: Pull complete
393ad8a32e58: Pull complete
63ac16987eeb: Pull complete
98e749c71626: Pull complete
a53b3b4bb02b: Pull complete
df9cf2ba76cb: Pull complete
Digest: sha256:466e4b18e15e909a5d32b0699039e53db8450e45b8a4ef37238affd0ff4225d3
Status: Downloaded newer image for openmaptiles/generate-vectortiles:0.1.1
/usr/local/lib/node_modules/tilelive/bin/tilelive-copy:100
   if (err) throw err;
TypeError: Cannot set property 'id' of undefined
  at /usr/local/lib/node_modules/tilelive-tmsource/index.js:184:20
  at tryToString (fs.js:414:3)
  at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:401:12)

[ImreSamu]

I assumed I'd be able to substitute that file in ./data directory instead of downloading the albania default

have you tried this : QUICKSTART.md#using-your-own-osm-data ?

[init-js]

did a "make clean-docker", then make remove-docker-images just to have a clean slate

arranged ./data/ so that it only contained my pbf file, with the expected extension name:

$ ls -l data/
total 936
-rw-rw-r-- 1 jslegare jslegare 942243 Apr 22 18:33 squamish.osm.pbf

If I look at the geojson for the same extracted map, it has a total of about 10000 features at most. I'm setting it up with a narrow zoom range 15 to 18.

https://www.openstreetmap.org/#map=15/49.6869/-123.1488&layers=C

That same map, converted from GeoJSON with tippecanoe, yields an mbtiles file that is about 1MB in size for these zoom levels.

$ cat .env
POSTGRES_DB=openmaptiles
POSTGRES_USER=openmaptiles
POSTGRES_PASSWORD=openmaptiles
POSTGRES_HOST=postgres
POSTGRES_PORT=5432
QUICKSTART_MIN_ZOOM=15
QUICKSTART_MAX_ZOOM=18
DIFF_MODE=false

Then, I run ./quickstart.sh squamish as your link suggests. I'm getting, first:

WARNING: The MIN_ZOOM variable is not set. Defaulting to a blank string.
WARNING: The MAX_ZOOM variable is not set. Defaulting to a blank string.
WARNING: The BBOX variable is not set. Defaulting to a blank string.

Then, after a while, it aborts with:

-------------------------------------------------------------------------------------
====> : Start generating MBTiles (containing gzipped MVT PBF) from a TM2Source project. 
  : TM2Source project definitions : ./build/openmaptiles.tm2source/data.yml 
  : Output MBTiles: ./data/tiles.mbtiles  
  : Source code: https://github.com/openmaptiles/generate-vectortiles 
  : We are using a lot of Mapbox Open Source tools! : https://github.com/mapbox 
  : Thank you https://www.mapbox.com !
  : See other MVT tools : https://github.com/mapbox/awesome-vector-tiles 
  :  
  : You will see a lot of deprecated warning in the log! This is normal!  
  :    like :  Mapnik LOG>  ... is deprecated and will be removed in Mapnik 4.x ... 
.IOError: [Errno 2] No such file or directory: u'././data/docker-compose-config.yml'

Sure enough, that file doesn't exist. Reading the script, it does look like docker-compose-config.yml is only created if a map is downloaded (make download-geofabrik area=${osm_area}), which isn't the case if you bring your own data (BYOD).

so I did make download-geofabrik area=british-columbia, just to get that config. That is missing from the instructions -- and it's almost treacherous, because had I not found that out, wouldn't it have computed tiles for my data with the bounding box from the default albania (or from a previous run)?

cat ./data/docker-compose-config.yml
version: "2"
services:
generate-vectortiles:
environment:
BBOX: " -141.0085209, 46.2501694, -113.2258330, 61.3374141"
OSM_MAX_TIMESTAMP : "2017-04-22T20:36:48Z"
OSM_AREA_NAME: "british-columbia" <-- I edited that to "squamish", in case.
MIN_ZOOM: "15"
MAX_ZOOM: "18"

Looks like the zoom levels carry over from .env at least.

Ran quickstart.sh squamish again after that, only to find that content of ./data had been deleted between the last run and this one.

$ ls -l data/
total 60
-rw-r--r-- 1 root root 47179 Apr 22 20:20 geofabrik.yml

so it tried to download the squamish area from geofabrik, which doesn't exist. I'm looking into what it is exactly that deleted stuff in my data folder -- everything except that geofabrik.yml file.

[init-js]

ugh.

make download-geofabrik area=foo clears all files in ./data/ before downloading.

then, ./quickstart.sh bar will ALSO delete all files in ./data/ if ./data/bar.osm.pbf doesn't exist.

That's not very resistant to typos.

[ImreSamu]

@init-js:

The current BYOD documantation is not perfect ( 😞 )

My suggestion for generating Squamish, British Columbia data :

0. Start from a clean state and with the default settings

1. Load bigger data from geofabrik , and generate zoom 0-7

./quickstart.sh british-columbia

2. modify .env file

QUICKSTART_MIN_ZOOM=0
QUICKSTART_MAX_ZOOM=14

3. copy and modify the config file

cp ./data/docker-compose-config.yml squamish-config.yml

and modify MAX_ZOOM and BBOX to Squamish

so cat squamish-config.yml is :

version: "2"
services:
  generate-vectortiles:
    environment:
      BBOX: " -123.3909, 49.6143, -122.9720, 49.8411 " 
      OSM_MAX_TIMESTAMP : "2017-04-22T20:36:48Z"
      OSM_AREA_NAME: "Squamish"
      MIN_ZOOM: "0"
      MAX_ZOOM: "14"

4. up Postgres

docker-compose up -d postgres

5. generate tiles

docker-compose -f docker-compose.yml -f ./squamish-config.yml run --rm generate-vectortiles

6. Set metadata for mbtiles .

Currently this is using QUICKSTART_MAX_ZOOM value ! ( from .env )

docker-compose run --rm openmaptiles-tools generate-metadata ./data/tiles.mbtiles

7. modify permissions of the mbtiles

docker-compose run --rm openmaptiles-tools chmod 666 ./data/tiles.mbtiles

8. test

make start-tileserver

9. Open from browser:

• http://localhost:8080/styles/osm-bright/#12.67/49.7096/-123.1424

10. Stop Postgres

docker-compose stop postgres

[ImreSamu]

@init-js

If I look at the geojson for the same extracted map, it has a total
of about 10000 features at most. I'm setting it up with a narrow zoom range 15 to 18.

Zoom level = 14 contains "everything" ,

• see : Needed map zoom to do Generating? #203
• and check your area with overzooming : (z18 from z14 )
  • http://localhost:8080/styles/osm-bright/#18.43/49.70346/-123.14617