generate-vectortiles fails #166

weatherc · 2017-02-06T16:18:47Z

Hi,

Did run the thing per the instructions "one by one". Now generate-vectortiles fails with following error:

/usr/local/lib/node_modules/tilelive/bin/tilelive-copy:100
if (err) throw err;
^

Error: Postgis Plugin: ERROR: function z(numeric) does not exist
LINE 1: ...5289e+38 3.402823466385289e+38)'::box3d, 900913), z(3.40282e...
^
HINT: No function matches the given name and argument types. You might need to add explicit type casts.
in executeQuery Full sql was: 'SELECT * FROM (SELECT geometry, class FROM layer_water(ST_SetSRID('BOX3D(-3.402823466385289e+38 -3.402823466385289e+38,3.402823466385289e+38 3.402823466385289e+38)'::box3d, 900913), z(3.40282e+38))) AS t LIMIT 0'

at Error (native)
at /usr/local/lib/node_modules/tilelive-tmsource/index.js:111:18
at Array.map (native)
at normalize (/usr/local/lib/node_modules/tilelive-tmsource/index.js:100:35)
at /usr/local/lib/node_modules/tilelive-tmsource/index.js:185:19
at tryToString (fs.js:414:3)
at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:401:12)

Any ideas?

The text was updated successfully, but these errors were encountered:

ImreSamu · 2017-02-06T17:40:02Z

./quickstart.sh is working ?

weatherc · 2017-02-06T17:59:09Z

Sorry, wrong button :)

Yes, quickstart are working.

Btw. how do i change so OSM-data are used for coastline for lower zooms?
It looks quite oddy for ex finnish archipelago when OSM-coastline kicks in at level 9 or so as its way to late. Ie. openmaptiles demo-styles have island-names showing up in areas where no island are shown etc. I should want it to kick in at level 5 or so.

ImreSamu · 2017-02-06T18:23:55Z

Yes, quickstart are working.

Good news! :)

Btw. how do i change so OSM-data are used for coastline for lower zooms?

If possible, please create a new issue.
quick answer:

You have to check the ETL diagram of the layers to understand of the data flow
And after - you have to modify the layer definitions code

weatherc · 2017-02-06T18:36:24Z

EDIT...
Quickstart worked for Albania (./quickstart) but now it fails for Europe (./quickstart europe) with this error:
====> : Drop and Recreate PostgreSQL public schema
docker-compose run --rm import-osm /usr/src/app/psql.sh -c "DROP SCHEMA IF EXISTS public CASCADE ; CREATE SCHEMA IF NOT EXISTS public; "
The MIN_ZOOM variable is not set. Defaulting to a blank string.
The MAX_ZOOM variable is not set. Defaulting to a blank string.
The BBOX variable is not set. Defaulting to a blank string.
Traceback (most recent call last):
File "", line 3, in
File "compose/cli/main.py", line 64, in main
File "compose/cli/main.py", line 116, in perform_command
File "compose/cli/main.py", line 711, in run
File "compose/cli/main.py", line 997, in run_one_off_container
File "site-packages/dockerpty/pty.py", line 334, in start
File "site-packages/dockerpty/pty.py", line 373, in _hijack_tty
File "site-packages/dockerpty/io.py", line 378, in flush
OSError: [Errno 9] Bad file descriptor
docker-compose returned -1
Makefile:85: recipe for target 'forced-clean-sql' failed
make: *** [forced-clean-sql] Error 255

ImreSamu · 2017-02-06T19:05:31Z

(./quickstart europe) with this error:

please attach the ./quickstart.log and an exact command.

And first, please test with smaller extracts.

./quickstart.sh albania is working ?
./quickstart.sh austria is working ?
./quickstart.sh germany is working ?
...

File "compose/cli/main.py", line 64, in main

Strange,

What is you docker and docker-compose version ? If not the latest , please upgrade ...
Can you retest/replicate on other environment/machine ?

weatherc · 2017-02-06T19:25:37Z

Docker version 1.12.2, build bb80604
docker-compose version 1.9.0-rc1, build 28788bd

./quickstart.sh albania => works
./quickstart.sh germany => works

The exact command for Europe was
./quickstart.sh europe
The log:
quickstart.txt

Sorry but i can't retest on other machine but this should have a lots of resources as can be seen in the log.

ImreSamu · 2017-02-06T19:51:06Z

thank you.

docker-compose version 1.9.0-rc1,

release candidate ?

Can you update the docker-compose to the latest ? I am not sure, but it can be a docker-compose problem: https://github.com/docker/compose/search?q=File+%22compose%2Fcli%2Fmain.py%22%2C+line+64%2C+in+main&type=Issues&utf8=%E2%9C%93

ImreSamu · 2017-02-06T21:34:03Z

I have installed https://github.com/docker/compose/releases/tag/1.9.0-rc1

but I can't replicate the error [ ./quickstart.sh europe -> File "compose/cli/main.py", line 64, in main ]

in my test environment:

Docker version 1.12.6, build 78d1802
docker-compose version 1.9.0-rc1, build 28788bd

# cat /etc/*{release,version}
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.1 LTS"

weatherc · 2017-02-06T21:43:50Z

I found the problem, now is it running fine.

As i can't rely on my internet-connection to the server to stay online continous for the time it takes to finish did i run it with nohup. That caused it. Once runned without nohup did it start to run OK.

ImreSamu · 2017-02-06T22:10:39Z

I found the problem, now is it running fine.

Good news!

the ./quickstart.sh is for testing ( with small extracts ) and for local development ;
not ideal for production

masoud2vg · 2019-09-19T04:48:39Z

@weatherc What did you do for z(numeric) function to fix the problem?

eyusupov · 2019-10-19T09:12:20Z

@masoud2vg I had the same issue recently and found this PR. The function is created here https://github.com/openmaptiles/import-sql/blob/master/import_sql.sh#L29 from this https://raw.githubusercontent.com/mapbox/postgis-vt-util/v1.0.0/postgis-vt-util.sql, so it could be that your import process failed at some point before that.

stevage · 2020-06-02T01:22:23Z

I have just experienced this issue.

docker-compose 1.25.4
docker 19.03.8
openmaptiles c6fff87 (May 27, 2020)

First I tried quickstart, but import-bbbike didn't work. So I began stepping through, with import-geofabrik (us-pacific).

Starting openmaptiles_postgres_1 ... done
Wait for PostgreSQL to start...
docker-compose run --rm -u 501:20 openmaptiles-tools pgwait
rm -rf data/tiles.mbtiles
echo "Generating tiles ..."; \
	docker-compose -f docker-compose.yml -f ./data/docker-compose-config.yml run --rm -u 501:20 generate-vectortiles
Generating tiles ...
Generating tiles into /export/tiles.mbtiles for zooms 0..7 inside (-180.0000000,15.5631200,-123.3180018,71.8052587) using 10 threads
Filtering deprecation warnings from the Mapnik's output.
/usr/local/lib/node_modules/tilelive/bin/tilelive-copy:100
        if (err) throw err;
                 ^

Error: Postgis Plugin: ERROR:  function z(numeric) does not exist
LINE 1: ...5289e+38 3.402823466385289e+38)'::box3d, 900913), z(3.40282e...

kochis · 2020-06-16T14:09:36Z

Can confirm, also seeing this issue

$ make generate-tiles

Starting postgres docker compose target using default image (no recreate if exists)
Starting openmaptiles_postgres_1 ... done
Wait for PostgreSQL to start...
docker-compose run --rm -u 501:20 openmaptiles-tools pgwait
Detected area=new-york based on the found data/ pbf file. Use 'area' parameter (or env var) to override.
Generating tiles into data/tiles.mbtiles (will delete if already exists)...
docker-compose -f docker-compose.yml -f data/new-york.dc-config.yml run --rm -u 501:20 generate-vectortiles
Generating tiles into /export/tiles.mbtiles for zooms 0..7 inside (-83.4785061,40.0976424,-9.5434670,49.8730330) using 10 threads
Filtering deprecation warnings from the Mapnik's output.
/usr/local/lib/node_modules/tilelive/bin/tilelive-copy:100
        if (err) throw err;
                 ^

Error: Postgis Plugin: ERROR:  function z(numeric) does not exist
LINE 1: ...5289e+38 3.402823466385289e+38)'::box3d, 900913), z(3.40282e...
                                                             ^
HINT:  No function matches the given name and argument types. You might need to add explicit type casts.
in executeQuery Full sql was: 'SELECT * FROM (SELECT geometry, class, intermittent, brunnel FROM layer_water(ST_SetSRID('BOX3D(-3.402823466385289e+38 -3.402823466385289e+38,3.402823466385289e+38 3.402823466385289e+38)'::box3d, 900913), z(3.40282e+38))) AS t LIMIT 0'

    at Error (native)
    at /usr/local/lib/node_modules/tilelive-tmsource/index.js:111:18
    at Array.map (native)
    at normalize (/usr/local/lib/node_modules/tilelive-tmsource/index.js:100:35)
    at /usr/local/lib/node_modules/tilelive-tmsource/index.js:185:19
    at tryToString (fs.js:414:3)
    at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:401:12)

TomPohys · 2020-06-16T15:07:04Z

Hi @kochis. Did you made make import-sql before make generate-tiles? Function z is one of the many functions which are imported into PostgreSQL.

kochis · 2020-06-19T17:18:56Z

@TomPohys Yep, that seemed to do the trick, thanks!

Though now I'm seeing layer_water does not exist 🤔

Error: Postgis Plugin: ERROR:  function layer_water(geometry, integer) does not exist
LINE 1: ...ELECT geometry, class, intermittent, brunnel FROM layer_wate...
                                                             ^
HINT:  No function matches the given name and argument types. You might need to add explicit type casts.
in executeQuery Full sql was: 'SELECT * FROM (SELECT geometry, class, intermittent, brunnel FROM layer_water(ST_SetSRID('BOX3D(-3.402823466385289e+38 -3.402823466385289e+38,3.402823466385289e+38 3.402823466385289e+38)'::box3d, 900913), z(3.40282e+38))) AS t LIMIT 0'

Ran the following before trying to regenerate:

make clean
make 
make import-sql

JensTimmerman · 2020-07-21T12:05:06Z

I'm seeing the same issue

tilelive-copy --scheme=pyramid --bounds=BBOX --timeout="18000000"  --concurrency="10"--minzoom=MIN_ZOOM --maxzoom=MAX_ZOOM "tmsource://$PWD" "mbtiles:///root/data/tiles.mbtiles"
/usr/lib/node_modules/@mapbox/tilelive/bin/tilelive-copy:100
        if (err) throw err;
                 ^

Error: Postgis Plugin: ERROR:  function layer_water(geometry, integer) does not exist
LINE 1: ...ELECT geometry, class, intermittent, brunnel FROM layer_wate...
                                                             ^
HINT:  No function matches the given name and argument types. You might need to add explicit type casts.
in executeQuery Full sql was: 'SELECT * FROM (SELECT geometry, class, intermittent, brunnel FROM layer_water(ST_SetSRID('BOX3D(-3.402823466385289e+38 -3.402823466385289e+38,3.402823466385289e+38 3.402823466385289e+38)'::box3d, 900913), z(3.40282e+38))) AS t LIMIT 0'

    at Error (native)
    at /usr/lib/node_modules/tilelive-tmsource/index.js:111:18
    at Array.map (native)
    at normalize (/usr/lib/node_modules/tilelive-tmsource/index.js:100:35)
    at TMSource.init (/usr/lib/node_modules/tilelive-tmsource/index.js:194:17)
    at /usr/lib/node_modules/tilelive-tmsource/index.js:184:17
    at tryToString (fs.js:457:3)
    at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:444:12)

ykalinin · 2020-07-22T19:46:54Z

Excuse me, but I've got the same issue. It happens on large amount of data. I suggest that import-sql run SQL scripts in background as well as parallel and doesn't wait properly for ending of scripts, meanwhile, what happens (if you run script directly) script water__something.sql stucks (or takes really huge amount of time with no updates) somewhere at update_tag instruction (line 400+). I will attach the exact place.

yop0382 · 2021-02-08T07:12:46Z

set MAX_PARALLEL_PSQL=1 in .env will disable // working and wait for each long script ...

weatherc closed this as completed Feb 6, 2017

weatherc reopened this Feb 6, 2017

ImreSamu closed this as completed Feb 10, 2017

jirik added this to the v3.5 milestone Mar 31, 2017

ImreSamu mentioned this issue May 24, 2017

Refactoring quickstart.sh workflow #261

Closed

5 tasks

ImreSamu mentioned this issue Jul 6, 2017

Quickstart refactoring #261 #304

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

generate-vectortiles fails #166

generate-vectortiles fails #166

weatherc commented Feb 6, 2017

ImreSamu commented Feb 6, 2017

weatherc commented Feb 6, 2017 •

edited

Loading

ImreSamu commented Feb 6, 2017

weatherc commented Feb 6, 2017

ImreSamu commented Feb 6, 2017

weatherc commented Feb 6, 2017

ImreSamu commented Feb 6, 2017

ImreSamu commented Feb 6, 2017

weatherc commented Feb 6, 2017

ImreSamu commented Feb 6, 2017

masoud2vg commented Sep 19, 2019

eyusupov commented Oct 19, 2019

stevage commented Jun 2, 2020

kochis commented Jun 16, 2020

TomPohys commented Jun 16, 2020

kochis commented Jun 19, 2020 •

edited

Loading

JensTimmerman commented Jul 21, 2020

ykalinin commented Jul 22, 2020

yop0382 commented Feb 8, 2021

generate-vectortiles fails #166

generate-vectortiles fails #166

Comments

weatherc commented Feb 6, 2017

ImreSamu commented Feb 6, 2017

weatherc commented Feb 6, 2017 • edited Loading

ImreSamu commented Feb 6, 2017

weatherc commented Feb 6, 2017

ImreSamu commented Feb 6, 2017

weatherc commented Feb 6, 2017

ImreSamu commented Feb 6, 2017

ImreSamu commented Feb 6, 2017

weatherc commented Feb 6, 2017

ImreSamu commented Feb 6, 2017

masoud2vg commented Sep 19, 2019

eyusupov commented Oct 19, 2019

stevage commented Jun 2, 2020

kochis commented Jun 16, 2020

TomPohys commented Jun 16, 2020

kochis commented Jun 19, 2020 • edited Loading

JensTimmerman commented Jul 21, 2020

ykalinin commented Jul 22, 2020

yop0382 commented Feb 8, 2021

weatherc commented Feb 6, 2017 •

edited

Loading

kochis commented Jun 19, 2020 •

edited

Loading