Skip to content

WIP: Add Dockerfile and a script to do initial data bootstrap#1254

Merged
iandees merged 9 commits intomasterfrom
iandees/dockerize
Nov 15, 2017
Merged

WIP: Add Dockerfile and a script to do initial data bootstrap#1254
iandees merged 9 commits intomasterfrom
iandees/dockerize

Conversation

@iandees
Copy link
Copy Markdown
Member

@iandees iandees commented May 17, 2017

Comment thread scripts/docker_boostrap.sh Outdated
-d "$POSTGRES_ENV_POSTGRES_DB" \
-c "create extension if not exists postgis; create extension if not exists hstore;"

/usr/bin/wget https://s3.amazonaws.com/metro-extracts.mapzen.com/new-york_new-york.osm.pbf
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this be a variable so I could, say, load up San Francisco instead?

How might we support multiple metro extracts?

Comment thread scripts/docker_boostrap.sh Outdated
--cache 1024 \
--style osm2pgsql.style \
--hstore-all \
new-york_new-york.osm.pbf \
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would need to get the filename part of the earlier variable per comment https://github.com/tilezen/vector-datasource/pull/1254/files#r117051094.

@iandees
Copy link
Copy Markdown
Member Author

iandees commented Nov 14, 2017

The import script is currently failing with:

+ ./perform-sql-updates.sh -h postgres -p 5432 -U osm -d osm
Creating custom schema...
psql:schema.sql:6: ERROR:  relation "wof_neighbourhood_placetype" already exists

@iandees
Copy link
Copy Markdown
Member Author

iandees commented Nov 14, 2017

Also getting this:

Generating functions from YAML...
/usr/src/app/vectordatasource/meta /usr/src/app/data
Traceback (most recent call last):
  File "sql.py", line 1, in <module>
    from vectordatasource.meta.python import parse_layers, output_kind, \
ImportError: No module named vectordatasource.meta.python

I think part of the problem is that my .dockerignore was not ignoring the compiled stuff I had locally, so it was pulling .pyc and .egg, etc. stuff into the Docker image. When I added this stuff to .dockerignore and built the image again, it required several more changes. I'm still running into similar errors, though, so I'm not sure what's going on.

@iandees
Copy link
Copy Markdown
Member Author

iandees commented Nov 14, 2017

😭 I think the issue is that I'm building the docker image using the tilezen/vector-datasource tag and then trying to run the mapzen/vector-datasource image...

@iandees
Copy link
Copy Markdown
Member Author

iandees commented Nov 15, 2017

Part of the problem was that I was confusing the image names as I mentioned above, but I also was running into problems because the image I built pulled in compiled stuff from my Mac that it shouldn't have. I added those things to .dockerignore and got farther. But with the compiled stuff not showing up I had to actually build the vector-datasource stuff in the image, so I added that as well. The image seems to work now as a data loader. I'll continue tomorrow by adding the documentation to tilezen/tileserver#93.

@iandees iandees merged commit 9b8c675 into master Nov 15, 2017
@iandees iandees deleted the iandees/dockerize branch November 15, 2017 21:49
@guyisra
Copy link
Copy Markdown

guyisra commented Jun 12, 2018

@iandees probably a bit old, but how did you end up fixing the relation "wof_neighbourhood_placetype" already exists
seems to be caused by the CREATE TABLE in schema.sql

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants