Missing documentation

[SomeoneElseOSM]

I've just tried it on an Ubuntu 18.04 server, and, first things first, something's needed to get "docker" installed. There are a number of different ways to do that and it'd be useful to know what you've previous done to get to the top of the README".

I did a "sudo apt install docker-compose" but "docker volume create openstreetmap-data" fails with permissions whereas "sudo docker volume create openstreetmap-data" does not. What system did you use when writing the README (i.e. on which systems could you follow the README with no changes and have everything work?)

Subsequently after

wget http://download.geofabrik.de/europe/luxembourg-latest.osm.pbf

I did

sudo docker run -v /home/blah/src/openstreetmap-tile-server/luxembourg.osm.pbf:/data.osm.pbf -v openstreetmap-data:/var/lib/postgresql/10/main overv/openstreetmap-tile-server import

which failed with

Reading in file: /data.osm.pbf
Using PBF parser.
node cache: stored: 0(-nan%), storage efficiency: -nan% (dense blocks: 0, sparse nodes: 0), hit rate: -nan%
Osm2pgsql failed due to ERROR: Read failed: Is a directory

[Overv]

Installing Docker is outside the scope of this project, but here's what I did:

1. Install Ubuntu server
2. Install Docker CE as described here
3. Add my user to the docker group with sudo usermod -aG docker myusername
4. Relogin

The fact that you get a "Is a directory" error means that the file /home/blah/src/openstreetmap-tile-server/luxembourg.osm.pbf does not exist. Make sure that the path is correct.

[Istador]

You installed docker-compose, which isn't required by this project.
(I advise using docker-compose with this project, but it requires you to write a docker-compose.yml file).

On Ubuntu the package that contains docker is called docker.io.
docker-compose depends on docker.io so you already installed it.

Personally, instead of the old docker.io package I installed docker-ce from the official docker repository
https://docs.docker.com/install/linux/docker-ce/ubuntu/#install-using-the-repository

And docker-compose via pip
https://docs.docker.com/compose/install/#install-using-pip

(The packages bundled with the distro tends to be a bit older and only allow old docker-compose file formats.)

---

docker volume create without sudo:
https://docs.docker.com/install/linux/linux-postinstall/#manage-docker-as-a-non-root-user

---

When working with docker volume mounts, an "Is a directory" error usually indicates that the source file that you are trying to mount doesn't exist.
(Because docker creates an empty directory instead of an empty file, if the file doesn't exist on the host.)

You downloaded a file named "luxembourg-latest.osm.pbf" and are trying to mount "luxembourg.osm.pbf".

You probably need to execute:

sudo docker run -v /home/blah/src/openstreetmap-tile-server/luxembourg-latest.osm.pbf:/data.osm.pbf -v openstreetmap-data:/var/lib/postgresql/10/main overv/openstreetmap-tile-server import

[SomeoneElseOSM]

Thanks both! I ended up following https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-18-04 and doing everything from the local account instead of root.

I think one of the problems before was that I did

docker run -v luxembourg-latest.osm.pbf:/data.osm.pbf ...

not

docker run -v /home/blah/luxembourg-latest.osm.pbf:/data.osm.pbf

and then later tried it with an incorrect file name, as noted above.

(I'll try things just from an Ubuntu repository later)

That then loads and the tile server runs.

[joaquinolivera]

Hi! I've tried to import the data of the region I need to set my tile server but I have the same issues.
I'm ussing docker toolbox in a windows laptop with a 18.04 WSL. I shared my folders, but nothing is working.

---

The stat command output of the path i use for the setting of the is the following:

File: /c/absolute/path/luxemburg.osm.pbf
Size: 7120 0586        Blocks: 139064     IO Block: 4096   regular file                                                                                                                                           Device: eh/14d  Inode: 6192449487655035  Links: 1 
Access: (0777/-rwxrwxrwx)  Uid: ( 1000/ myuser)   Gid: ( 1000/ myuser)      
Access: 2019-08-02 11:47:18.462086500 -0400
Modify: 2019-08-02 11:47:18.462086500 -0400    
Change: 2019-08-02 13:53:30.268887000 -0400                                   
Birth: - 

The file command output is:

/c/absolute/path/luxemburg.osm.pbf: OpenStreetMap Protocolbuffer Binary Format

As you can see i'm using the binding with:

sudo mkdir /c
sudo mount --bind /mnt/c /c

After I execute the import command,

docker run -v /c/absolute/path/luxemburg.osm.pbf:/data.osm.pbf -v openstreetmap-data:/var/lib/postgresql/10/main overv/openstreetmap-tile-server import 

even trying:

docker run -v //c/absolute/path/luxemburg.osm.pbf:/data.osm.pbf -v openstreetmap-data:/var/lib/postgresql/10/main overv/openstreetmap-tile-server import 

---

I get the following:

+ '[' 1 -ne 1 ']'
+ '[' import = import ']'
+ CreatePostgressqlConfig
+ cp /etc/postgresql/10/main/postgresql.custom.conf.tmpl /etc/postgresql/10/main/postgresql.custom.conf
+ sudo -u postgres echo 'autovacuum = on'
+ cat /etc/postgresql/10/main/postgresql.custom.conf
# Suggested minimal settings from
# https://ircama.github.io/osm-carto-tutorials/tile-server-ubuntu/

shared_buffers = 128MB
min_wal_size = 1GB
max_wal_size = 2GB
maintenance_work_mem = 256MB

# Suggested settings from
# https://github.com/openstreetmap/chef/blob/master/roles/tile.rb#L38-L45

max_connections = 250
temp_buffers = 32MB
work_mem = 128MB
wal_buffers = 1024kB
wal_writer_delay = 500ms
commit_delay = 10000
# checkpoint_segments = 60 # unrecognized in psql 10.7.1
max_wal_size = 2880MB
random_page_cost = 1.1
track_activity_query_size = 16384
autovacuum_vacuum_scale_factor = 0.05
autovacuum_analyze_scale_factor = 0.02
autovacuum = on
+ service postgresql start
 * Starting PostgreSQL 10 database server
   ...done.
+ sudo -u postgres createuser renderer
+ sudo -u postgres createdb -E UTF8 -O renderer gis
+ sudo -u postgres psql -d gis -c 'CREATE EXTENSION postgis;'
CREATE EXTENSION
+ sudo -u postgres psql -d gis -c 'CREATE EXTENSION hstore;'
CREATE EXTENSION
+ sudo -u postgres psql -d gis -c 'ALTER TABLE geometry_columns OWNER TO renderer;'
ALTER TABLE
+ sudo -u postgres psql -d gis -c 'ALTER TABLE spatial_ref_sys OWNER TO renderer;'
ALTER TABLE
+ '[' '!' -f /data.osm.pbf ']'
WARNING: No import file at /data.osm.pbf, so importing Luxembourg as example...
+ echo 'WARNING: No import file at /data.osm.pbf, so importing Luxembourg as example...'
+ wget -nv http://download.geofabrik.de/europe/luxembourg-latest.osm.pbf -O /data.osm.pbf
/data.osm.pbf: Is a directory
+ wget -nv http://download.geofabrik.de/europe/luxembourg.poly -O /data.poly
wget: unable to resolve host address 'download.geofabrik.de'
+ osmium fileinfo /data.osm.pbf
Read failed: Is a directory
+ osmium fileinfo /data.osm.pbf
+ cut -b35-44
+ grep osmosis_replication_timestamp=
Read failed: Is a directory
++ cat /var/lib/mod_tile/replication_timestamp.txt
+ REPLICATION_TIMESTAMP=
+ sudo -u renderer openstreetmap-tiles-update-expire
cat: /var/lib/mod_tile/.osmosis/state.txt: No such file or directory
cat: /var/lib/mod_tile/.osmosis/state.txt: No such file or directory
./osmosis-db_replag: line 11: 1564932767 - : syntax error: operand expected (error token is "- ")
./osmosis-db_replag: line 15: [: -gt: unary operator expected
./osmosis-db_replag: line 16: [: -gt: unary operator expected
./osmosis-db_replag: line 17: [: -gt: unary operator expected
/bin/cp: cannot stat '/var/lib/mod_tile/.osmosis/state.txt': No such file or directory
+ '[' -f /data.poly ']'
+ sudo -u renderer cp /data.poly /var/lib/mod_tile/data.poly
+ sudo -u renderer osm2pgsql -d gis --create --slim -G --hstore --tag-transform-script /home/renderer/src/openstreetmap-carto/openstreetmap-carto.lua -C 2048 --number-processes 4 -S /home/renderer/src/openstreetmap-carto/openstreetmap-carto.style /data.osm.pbf
osm2pgsql version 0.96.0 (64 bit id space)

Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=2048MB, maxblocks=32768*65536, allocation method=11
Mid: pgsql, cache=2048
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Using lua based tag processing pipeline with script /home/renderer/src/openstreetmap-carto/openstreetmap-carto.lua
Using projection SRS 3857 (Spherical Mercator)
Setting up table: planet_osm_point
Setting up table: planet_osm_line
Setting up table: planet_osm_polygon
Setting up table: planet_osm_roads

Reading in file: /data.osm.pbf
Using PBF parser.
node cache: stored: 0(-nan%), storage efficiency: -nan% (dense blocks: 0, sparse nodes: 0), hit rate: -nan%
Osm2pgsql failed due to ERROR: Read failed: Is a directory
+ sudo -u postgres psql -d gis -f indexes.sql
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
+ service postgresql stop
 * Stopping PostgreSQL 10 database server
   ...done.
+ exit 0

I erase the volumes after after every try.

[Overv]

To see what's going wrong let's do some debugging. Start the following container:

docker run -v /c/absolute/path/luxemburg.osm.pbf:/data.osm.pbf -it ubuntu bash

And inside the container's shell check what's actually being mounted at /data.osm.pbf.

[joaquinolivera]

After running:

docker run -v /c/absolute/path/luxemburg.osm.pbf:/data.osm.pbf -it ubuntu bash

and check in data.osm.pbf, the directory is empty.

Seems like docker have no access to it.

Even I put the files under `C:\Users<my-user>

[Overv]

Then I think you'll have to move the file to the actual Linux file system and not the mapped Windows one.

[joaquinolivera]

Yes, I've tried something like that also just in case.
Inside my WSL, in a directory like /home/<my-user>/ I have download my .osm file and test the import, but have the same results yet. Even with this last command, removing volumes in between tests, the data.osm.pbf is empty.

[Istador]

Then I think you'll have to move the file to the actual Linux file system and not the mapped Windows one.

That shouldn't work, as the docker service is running on the Windows host system and has only access to the windows paths. The docker process doesn't know anything about what WSL is doing internally.

Putting it under a path that is only inside the Linux subsystem, that the normal windows can't access, should fail.

---

Have you checked if C:\ is configured as a Shared Drive that docker is allowed to access?

Edit: https://blogs.msdn.microsoft.com/stevelasker/2016/06/14/configuring-docker-for-windows-volumes/

[joaquinolivera]

Have you checked if C:\ is configured as a Shared Drive that docker is allowed to access?

Yes, I have, only if I do it wrong, what could it be. I am using docker toolbox, configure the VM
shared folder, and then restart the default VM.
Here is a capture:

---

Just in case I shared C:\ with some variations in the machine folder, but none of those work.
With Auto-mount and Make permanent options.

Still this is not working

[joaquinolivera]

Well, I'm feeling pretty stupid by this, but appear you not only need to shared the folder in the VirtualBox, you need to mounted as well.

Here I let the guide that I used to solve my problem.

Thanks to all for the attention.