New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tilelive-tmsource : Cannot set property 'id' of undefined. #227

Closed
init-js opened this Issue Apr 22, 2017 · 5 comments

Comments

Projects
None yet
3 participants
@init-js

init-js commented Apr 22, 2017

I downloaded a single city's worth of data from Mapzen from which to generate a vector tile map. The data is the "raw pbf" data export option file, and mapzen extracts data from OSM.
( https://mapzen.com/data/metro-extracts/ )

I assumed I'd be able to substitute that file in ./data directory instead of downloading the albania default. I'm doing the manual steps, except I'm skipping the import-natural-earth, import-water, and import-osmborder. It's not really explained anywhere, as far as I can tell, why those steps are necessary -- what if I was making a non-earth map?

Anyways, at the point where it's generating the files, i'm getting what looks to be a missing 'id' field error, or a missing configuration where an id is supposed to be. Is this due to an unexpected input format of the files, or is it more bad yaml configuration?

$ docker-compose run generate-vectortiles
WARNING: Dependency conflict: an older version of the 'docker-py' package may be polluting the namespace. If you're experiencing crashes, run the following command to remedy the issue:
pip uninstall docker-py; pip uninstall docker; pip install docker
WARNING: The MIN_ZOOM variable is not set. Defaulting to a blank string.
WARNING: The MAX_ZOOM variable is not set. Defaulting to a blank string.
WARNING: The BBOX variable is not set. Defaulting to a blank string.
Pulling generate-vectortiles (openmaptiles/generate-vectortiles:0.1.1)...
0.1.1: Pulling from openmaptiles/generate-vectortiles
8ad8b3f87b37: Pull complete
751fe39c4d34: Pull complete
ae3b77eefc06: Pull complete
7783aac582ec: Pull complete
393ad8a32e58: Pull complete
63ac16987eeb: Pull complete
98e749c71626: Pull complete
a53b3b4bb02b: Pull complete
df9cf2ba76cb: Pull complete
Digest: sha256:466e4b18e15e909a5d32b0699039e53db8450e45b8a4ef37238affd0ff4225d3
Status: Downloaded newer image for openmaptiles/generate-vectortiles:0.1.1
/usr/local/lib/node_modules/tilelive/bin/tilelive-copy:100
   if (err) throw err;
TypeError: Cannot set property 'id' of undefined
  at /usr/local/lib/node_modules/tilelive-tmsource/index.js:184:20
  at tryToString (fs.js:414:3)
  at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:401:12)
@ImreSamu

This comment has been minimized.

Collaborator

ImreSamu commented Apr 22, 2017

I assumed I'd be able to substitute that file in ./data directory instead of downloading the albania default

have you tried this : QUICKSTART.md#using-your-own-osm-data ?

@init-js

This comment has been minimized.

init-js commented Apr 23, 2017

did a "make clean-docker", then make remove-docker-images just to have a clean slate

arranged ./data/ so that it only contained my pbf file, with the expected extension name:

$ ls -l data/
total 936
-rw-rw-r-- 1 jslegare jslegare 942243 Apr 22 18:33 squamish.osm.pbf

If I look at the geojson for the same extracted map, it has a total of about 10000 features at most. I'm setting it up with a narrow zoom range 15 to 18.

https://www.openstreetmap.org/#map=15/49.6869/-123.1488&layers=C

That same map, converted from GeoJSON with tippecanoe, yields an mbtiles file that is about 1MB in size for these zoom levels.

$ cat .env
POSTGRES_DB=openmaptiles
POSTGRES_USER=openmaptiles
POSTGRES_PASSWORD=openmaptiles
POSTGRES_HOST=postgres
POSTGRES_PORT=5432
QUICKSTART_MIN_ZOOM=15
QUICKSTART_MAX_ZOOM=18
DIFF_MODE=false

Then, I run ./quickstart.sh squamish as your link suggests. I'm getting, first:

WARNING: The MIN_ZOOM variable is not set. Defaulting to a blank string.
WARNING: The MAX_ZOOM variable is not set. Defaulting to a blank string.
WARNING: The BBOX variable is not set. Defaulting to a blank string.

Then, after a while, it aborts with:

-------------------------------------------------------------------------------------
====> : Start generating MBTiles (containing gzipped MVT PBF) from a TM2Source project. 
  : TM2Source project definitions : ./build/openmaptiles.tm2source/data.yml 
  : Output MBTiles: ./data/tiles.mbtiles  
  : Source code: https://github.com/openmaptiles/generate-vectortiles 
  : We are using a lot of Mapbox Open Source tools! : https://github.com/mapbox 
  : Thank you https://www.mapbox.com !
  : See other MVT tools : https://github.com/mapbox/awesome-vector-tiles 
  :  
  : You will see a lot of deprecated warning in the log! This is normal!  
  :    like :  Mapnik LOG>  ... is deprecated and will be removed in Mapnik 4.x ... 
.IOError: [Errno 2] No such file or directory: u'././data/docker-compose-config.yml'

Sure enough, that file doesn't exist. Reading the script, it does look like docker-compose-config.yml is only created if a map is downloaded (make download-geofabrik area=${osm_area}), which isn't the case if you bring your own data (BYOD).

so I did make download-geofabrik area=british-columbia, just to get that config. That is missing from the instructions -- and it's almost treacherous, because had I not found that out, wouldn't it have computed tiles for my data with the bounding box from the default albania (or from a previous run)?

cat ./data/docker-compose-config.yml
version: "2"
services:
generate-vectortiles:
environment:
BBOX: " -141.0085209, 46.2501694, -113.2258330, 61.3374141"
OSM_MAX_TIMESTAMP : "2017-04-22T20:36:48Z"
OSM_AREA_NAME: "british-columbia" <-- I edited that to "squamish", in case.
MIN_ZOOM: "15"
MAX_ZOOM: "18"

Looks like the zoom levels carry over from .env at least.

Ran quickstart.sh squamish again after that, only to find that content of ./data had been deleted between the last run and this one.

$ ls -l data/
total 60
-rw-r--r-- 1 root root 47179 Apr 22 20:20 geofabrik.yml

so it tried to download the squamish area from geofabrik, which doesn't exist. I'm looking into what it is exactly that deleted stuff in my data folder -- everything except that geofabrik.yml file.

@init-js

This comment has been minimized.

init-js commented Apr 23, 2017

ugh.

make download-geofabrik area=foo clears all files in ./data/ before downloading.

then, ./quickstart.sh bar will ALSO delete all files in ./data/ if ./data/bar.osm.pbf doesn't exist.

That's not very resistant to typos.

@ImreSamu ImreSamu self-assigned this Apr 23, 2017

@ImreSamu

This comment has been minimized.

Collaborator

ImreSamu commented Apr 25, 2017

@init-js:

The current BYOD documantation is not perfect ( 😞 )

My suggestion for generating Squamish, British Columbia data :

0. Start from a clean state and with the default settings

1. Load bigger data from geofabrik , and generate zoom 0-7

./quickstart.sh british-columbia

2. modify .env file

QUICKSTART_MIN_ZOOM=0
QUICKSTART_MAX_ZOOM=14

3. copy and modify the config file

cp ./data/docker-compose-config.yml squamish-config.yml

and modify MAX_ZOOM and BBOX to Squamish

so cat squamish-config.yml is :

version: "2"
services:
  generate-vectortiles:
    environment:
      BBOX: " -123.3909, 49.6143, -122.9720, 49.8411 " 
      OSM_MAX_TIMESTAMP : "2017-04-22T20:36:48Z"
      OSM_AREA_NAME: "Squamish"
      MIN_ZOOM: "0"
      MAX_ZOOM: "14"

4. up Postgres

docker-compose up -d postgres

5. generate tiles

docker-compose -f docker-compose.yml -f ./squamish-config.yml run --rm generate-vectortiles

6. Set metadata for mbtiles .

Currently this is using QUICKSTART_MAX_ZOOM value ! ( from .env )

docker-compose run --rm openmaptiles-tools generate-metadata ./data/tiles.mbtiles

7. modify permissions of the mbtiles

docker-compose run --rm openmaptiles-tools chmod 666 ./data/tiles.mbtiles

8. test

make start-tileserver

9. Open from browser:

10. Stop Postgres

docker-compose stop postgres

@ImreSamu

This comment has been minimized.

Collaborator

ImreSamu commented Apr 25, 2017

@init-js

If I look at the geojson for the same extracted map, it has a total
of about 10000 features at most. I'm setting it up with a narrow zoom range 15 to 18.

Zoom level = 14 contains "everything" ,

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment