Providing Windows binaries/packages

[springmeyer]

In addition to OS X (#15) the osm2pgsql community of developers should provide binaries for Windows.

Let's coordinate how to get this done via this ticket.

Tasks:

• - code changes needed to support compiling under mingw, msvc, and/or cygwin
• - volunteer(s) to create binaries on windows and to test them
• - writing an easy installer using nsis that should produce a working executable without any additional installs or development toolkits and should work on 32/64 bit systems ranging from windows XP -> Win8.

Background discussions: hotosm/tech#1

[springmeyer]

/cc @onepremise - based on your build system so you have a binary yet for osm2pgsql that runs?

[springmeyer]

/cc @emirhartato, member of @hotosm who I know would love to get access to a working build to use in a workshop by the last week in April

[onepremise]

Hi Dane,

Yes I have a cygwin build which is working. I'm also getting together a
MinGW build which works with the new MinGW64 environment I've built. Let me
know which you're interested in so I can make it available.

Thanks,

Jason

On Mon, Apr 8, 2013 at 8:48 PM, Dane Springmeyer
notifications@github.comwrote:

/cc @emirhartato https://github.com/emirhartato, member of @hotosmhttps://github.com/hotosmwho I know would love to get access to a working build to use in a workshop
by the last week in April

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/17#issuecomment-16088128
.

[springmeyer]

Jason, both would be great for testing, but really whatever you have.

[emirhartato]

Hello, sorry for late reply.

This the error log for different build of osm2pgsql on Windows:
https://gist.github.com/emirhartato/5360318
https://gist.github.com/emirhartato/5360305

Build 0.69 that I got from http://tile.openstreetmap.org/osm2pgsql.zip doesn't shows any error. But the polygon sometimes doesn't work and polyline for road sometimes get messy. Here's the screenshot:

I'm running on MacOSX with 0.70, seems to work very well. But this build is not available for Windows.

[apmon]

Version 0.69 is pretty old. From what I can gather it is from sometime in 2010. That is presumably still pre 64 bit ID, so it won't work with the current planet at all, which is why it desperately needs replacing. People have compiled much newer working versions of osm2pgsql on windows though. They were just never put on tile.osm.org.

The errors shown in your logs look like configuration errors with the database parameters not correctly set. It might be possible that the defaults have changed in newer versions which is why you might need to set db parameters which you didn't use to need to do.

[emirhartato]

@apmon can you explain how to set database parameters? Hopefully if that can help solve the problem. There were 6 laptops tested and all of them have same error log. I also did a fresh install of PostgreSQL and PostGIS and create a new database, but still have the same error log.

[onepremise]

Not sure if it helps, but here are my notes on the postgres configuration:
Tune Parameters for postgresql.conf

• autovacuum = off
• checkpoint_segments = 20
• shared_buffers = 512MB # min 128kB
• work_mem = 256MB # min 64kB
• maintenance_work_mem = 512MB # min 1MB
• synchronous_commit = off

Import example:

osm2pgsql.exe -v -c -d osm -U osm -W -H localhost -P 5432 -S default.style
-s -C 1600 --hstore --number-processes=4 -r pbf
....\data\us-midwest-northeast-pacific-south-west.osm.pbf

I also have a recent build of osm2psql which works. I have some mapnik
debugging to get through on mingw64, but I'll make it all available to our
build server here:

https://vanguard.houghtonassociates.com

The next build will have a package you can download under the Artifacts tab:

https://vanguard.houghtonassociates.com/browse/OSM-OSM2PSQL-27/artifact

Is there a specific time-frame you need this?

On Thu, Apr 11, 2013 at 2:18 AM, emirhartato notifications@github.comwrote:

@apmon https://github.com/apmon can you explain how to set database
parameters? Hopefully if that can help solve the problem. There were 6
laptops tested and all of them have same error log. I also did a fresh
install of PostgreSQL and PostGIS and create a new database, but still have
the same error log.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/17#issuecomment-16218534
.

[emirhartato]

Hello...

Well, there will be a workshop in 6th May. My team will teach osm2pgsql and
we will go through OSM Tool GUI in QGIS rather than running it through
command prompt since our target participants are not an IT expert or geek.

Our team also already have Advanced OpenStreetMap documentation written by
Jeff Haack and it's available in English and Bahasa Indonesia and there's a
chapter that provide step-by-step tutorial about setting up PostGIS and
osm2pgsql. We will published it soon on learosm.org site but we can't
published it until this issue is solved.

But if that's the only way to work, it's still possible to change the
tutorial on documentation but it would be nuce if we have a working
osm2pgsql with OSM Tool GUI in QGIS for Windows.
On Apr 11, 2013 10:55 PM, "Jason Huntley" notifications@github.com wrote:

Not sure if it helps, but here are my notes on the postgres configuration:
Tune Parameters for postgresql.conf

• autovacuum = off
• checkpoint_segments = 20
• shared_buffers = 512MB # min 128kB
• work_mem = 256MB # min 64kB
• maintenance_work_mem = 512MB # min 1MB
• synchronous_commit = off

Import example:

osm2pgsql.exe -v -c -d osm -U osm -W -H localhost -P 5432 -S default.style
-s -C 1600 --hstore --number-processes=4 -r pbf
....\data\us-midwest-northeast-pacific-south-west.osm.pbf

I also have a recent build of osm2psql which works. I have some mapnik
debugging to get through on mingw64, but I'll make it all available to our
build server here:

https://vanguard.houghtonassociates.com

The next build will have a package you can download under the Artifacts
tab:

https://vanguard.houghtonassociates.com/browse/OSM-OSM2PSQL-27/artifact

Is there a specific time-frame you need this?

On Thu, Apr 11, 2013 at 2:18 AM, emirhartato notifications@github.comwrote:

@apmon https://github.com/apmon can you explain how to set database
parameters? Hopefully if that can help solve the problem. There were 6
laptops tested and all of them have same error log. I also did a fresh
install of PostgreSQL and PostGIS and create a new database, but still
have
the same error log.

—
Reply to this email directly or view it on GitHub<
https://github.com/openstreetmap/osm2pgsql/issues/17#issuecomment-16218534>

.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/17#issuecomment-16243529
.

[apmon]

My guess would be the important parameters to resolve your issues are the following:

"-U osm -W -H localhost -P 5432"

-W forces a password promt where you can specify the password for the user osm. However, you have to make sure that postgresql is set up for password authentication and that you have set the appropriate password for the osm user in postgresql.

QGis should not be affected by any of this, as it simply uses an existing postgis db, no matter how you created it.

I have not heard of OSM Tool GUI before, so I don't know what it does. (And google doesn't seem to give me anything useful either) But assuming it is a wrapper around osm2pgsql to make sure you don't have to work with the command line, then you would simply have to extend OSM Tool GUI to set the appropriate parameters of osm2pgsql.

Oh, reading your message again, I see OSM Tool GUI is something / a plugin of QGis. But the same should still apply, that the integration module needs to be adapted to set the correct parameters when calling osm2pgsql. Just that it then does effect QGis... ;-)

You probably want to test it on the command line first. Make sure it works, and then contact the developers of the tool gui.

[springmeyer]

@apmon - Yes, I wrote "OSM Tool GUI" a while back: it is a python wrapper that works with the 0.69 (old) osm2pgsql (https://bitbucket.org/springmeyer/hot_tools/src/2321b0d0adae/osm_tools/command_dock2.py). Getting the password prompt working was extremely hard. Also when I've tried to compile osm2pgsql the getopt code has been the most difficult to port. So I presume that something changed in the behavior of the password prompt at some point, breaking the plugin.

@emirhartato - can you focus on testing osm2pgsql from the command line ONLY. Do not test with the QGIS plugin. We want to first make sure that an updated osm2pgsql build for windows works from the command line. After that I can take a look at any updates needed for the plugin to adapt it.

@emirhartato - my sense is you need to provide the -W flag when testing to that a password prompt comes up.

[apmon]

Well, that makes the task of fixing the "OSM Tool GUI" somewhat easier, if we know who to contact and know they are still interested in fixining things... ;-)

If password prompt is causing issues, it shouldn't be too difficult to try and add a different way to supply the password to osm2pgsql. One option would be to just add it as another parameter. However, the disadvantage is that the command line arguments are listed in the process outputs like top (not sure if that is the case in windows), so the password would then be publicly visible to everyone on the computer.

But yes, we first need to know that osm2pgsql itself works before fixing the wrapper around it.

@onepremise Nice to see those efforts. Having any form of working osm2pgsql on windows is obviously a good start, but the more "native" the compile is, the better. If I understand it correctly mingw is more "native" than cygwin, so that would be preferable. If we can get it to build from a clean source in git and can then add a windows build server to the ci.osm.org, then perhaps we can prevent it from constantly breaking and make windows a properly supported platform of osm2pgsql.

[springmeyer]

+1 to an insecure and easy way to pass a password for windows users. I configure my linux/osx postgres installs to not need a password, but the postgres windows installers require a password based on how the installers work. So a backdoor here for development would be really handy.

[onepremise]

Nice to see those efforts. Having any form of working osm2pgsql on windows is obviously a good start, but the more "native" the compile is, the better. If I understand it correctly mingw is more "native" than cygwin, so that would be preferable.

Correct, mingw64 is definitely preferred, and I will providing the mingw64 build as soon as i tie up some lose ends with a Mapnik issue I'm working on in mingw 64 bit.

I'm very close to providing a stable build with MinGW-AD64S:

https://github.com/onepremise/MinGW-AD64S

Until then, I have the cygwin version, with 64bit IDs enabled, available here:

https://vanguard.houghtonassociates.com/browse/OSM-OSM2PSQL-35/artifact

Another build will follow up with the latest osm2pgsql updates.

[onepremise]

Here's a build with the latest changes rebased:

https://vanguard.houghtonassociates.com/browse/OSM-OSM2PSQL-39/artifact

[oeon]

Here are the steps I followed to get @onepremise's build to work for me on Win7 https://gist.github.com/jlar/5375480

[springmeyer]

Sweet, thanks @jlar! @emirhartato - could you try following those instructions ^^ and let us know how things work for you?

[emirhartato]

Sweet! Will test it out tonight.
On Apr 13, 2013 5:32 AM, "Dane Springmeyer" notifications@github.com
wrote:

Sweet, thanks @jlar https://github.com/jlar! @emirhartatohttps://github.com/emirhartato- could you try following those instructions ^^ and let us know how things
work for you?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/17#issuecomment-16321077
.

[emirhartato]

Ok, so I tested @onepremise build. It's still get me this error:

osm2pgsql SVN version 0.81.0 (64bit id space)
Password:
Error: Connection to database failed: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/tmp/.s.PGSQL.5432"?

[springmeyer]

@emirhartato - that error indicates usually means that postgres is not running. Can you confirm that you have postgres installed and it is running? Make sure to you can connect to postgres/postgis via pgadminIII and/or QGIS.

[oeon]

@emirhartato can you include your osm2pgsql command-code? Perhaps try specifying -H localhost (if applicable)?

[Rungee]

Hello,

Thank you guys for your valuable efforts.
But it seems that the build from @onepremise has some problems.
More exactly, on italy-latest.osm.pbf I caught an unhandled exception which was somewhere in the processing of relations (17410).

I have managed to find an id of OSM object. This is a relation
http://www.openstreetmap.org/browse/relation/957103

Moreover, I would like to mention that @onepremise's Cygwin build works much slower then the one provided by Dominik Perpeet.
Here is my benchmarks for cygwin and Dominik's (http://customdebug.com/osm/osm2pgsql.zip) builds for italy-latest.osm.* files.

--------------------------- Cygwin ---------------------------
.bz2
Node 94.1k/s Way 3.97k/s Relation 29.66/s

.pbf
Node 205.8k/s Way 5.01k/s Relation 10.82/s

--------------------------- Dominik ---------------------------
.bz2
Node 136.4k/s Way 12.03k/s Relation 133.92/s

.pbf
Node 213.2k/s Way 14.91k/s Relation 130.9/s

[springmeyer]

@Rungee - thanks for the feedback. I think at this point we are aiming at something that just works, performance concerns, while important, can come later. I wonder if the unhandled exception can be explained by the different binary or by a difference in the osm2pgsql version being packaged? Do you have the ability to contact Dominik to see what git revision his build is from?

/cc @apmon - what do you think about pushing the output of git describe into the reported osm2pgsql version? Note for git describe to work an annotated tag is needed git tag -a. It looks like your v0.82.0 tag is annotated nicely, but wanted to confirm this.

[emirhartato]

@jlar @onepremise
actually it works, but for some place for some reason, polygons didn't show up and polyline looks really weird.

Error message for version 0.8:

C:\osm2pgsql>osm2pgsql -c -d tes_maning -U postgres -W -H localhost -P 5432 -S C
:\osm2pgsql\default.style E:\Peta\shp_raung_2013\KRB_Raung\raung_barat.osm
osm2pgsql SVN version 0.80.0 (32bit id space)

Password:
Using projection SRS 900913 (Spherical Mercator)
cygwin warning:
MS-DOS style path detected: C:\osm2pgsql\default.style
Preferred POSIX equivalent is: /osm2pgsql/default.style
CYGWIN environment variable option "nodosfilewarning" turns off this warning.
Consult the user's guide for more details about POSIX paths:
http://cygwin.com/cygwin-ug-net/using.html#using-pathnames
Setting up table: planet_osm_point
NOTICE: table "planet_osm_point" does not exist, skipping
NOTICE: table "planet_osm_point_tmp" does not exist, skipping
Setting up table: planet_osm_line
NOTICE: table "planet_osm_line" does not exist, skipping
NOTICE: table "planet_osm_line_tmp" does not exist, skipping
Setting up table: planet_osm_polygon
NOTICE: table "planet_osm_polygon" does not exist, skipping
NOTICE: table "planet_osm_polygon_tmp" does not exist, skipping
Setting up table: planet_osm_roads
NOTICE: table "planet_osm_roads" does not exist, skipping
NOTICE: table "planet_osm_roads_tmp" does not exist, skipping
Allocating memory for dense node cache
Out of memory for node cache dense index, try using "--cache-strategy sparse" in
stead
Error occurred, cleaning up

Error message for 0.81 from https://vanguard.houghtonassociates.com/browse/OSM-OSM2PSQL-39/artifact

C:\osm2pgsql>osm2pgsql -c -d tes_maning -U postgres -W -H localhost -P 5432 -S C
:\osm2pgsql\default.style E:\Peta\shp_raung_2013\KRB_Raung\raung_barat.osm
osm2pgsql SVN version 0.81.0 (64bit id space)

Password:
Using projection SRS 900913 (Spherical Mercator)
cygwin warning:
MS-DOS style path detected: C:\osm2pgsql\default.style
Preferred POSIX equivalent is: /osm2pgsql/default.style
CYGWIN environment variable option "nodosfilewarning" turns off this warning.
Consult the user's guide for more details about POSIX paths:
http://cygwin.com/cygwin-ug-net/using.html#using-pathnames
Setting up table: planet_osm_point
NOTICE: table "planet_osm_point" does not exist, skipping
NOTICE: table "planet_osm_point_tmp" does not exist, skipping
Setting up table: planet_osm_line
NOTICE: table "planet_osm_line" does not exist, skipping
NOTICE: table "planet_osm_line_tmp" does not exist, skipping
Setting up table: planet_osm_polygon
NOTICE: table "planet_osm_polygon" does not exist, skipping
NOTICE: table "planet_osm_polygon_tmp" does not exist, skipping
Setting up table: planet_osm_roads
NOTICE: table "planet_osm_roads" does not exist, skipping
NOTICE: table "planet_osm_roads_tmp" does not exist, skipping
Allocating memory for sparse node cache
Out of memory for sparse node cache, reduce --cache size
Error occurred, cleaning up

[onepremise]

err? that's strange. You're missing some tables in your database. Did you
deploy pgSnapshot Schema for OSM? Did it complete successfully? Also, might
want to reduce your memory usage or at least provide your mem usage via
comandline, -C 1600 is what i used. Hope that helps!

On Thu, Apr 18, 2013 at 5:19 AM, emirhartato notifications@github.comwrote:

@jlar https://github.com/jlar @onepremisehttps://github.com/onepremise

actually it works, but for some place for some reason, polygons didn't
show up and polyline looks really weird.

[image: F5EF59E9-94CA-4C6D-9A25-E7747B2F48FB]https://f.cloud.github.com/assets/1563226/395674/93f6c656-a808-11e2-9633-688fd907b163.png

Error message for version 0.8:

C:\osm2pgsql>osm2pgsql -c -d tes_maning -U postgres -W -H localhost -P
5432 -S C
:\osm2pgsql\default.style E:\Peta\shp_raung_2013\KRB_Raung\raung_barat.osm
osm2pgsql SVN version 0.80.0 (32bit id space)

Password:
Using projection SRS 900913 (Spherical Mercator)
cygwin warning:
MS-DOS style path detected: C:\osm2pgsql\default.style
Preferred POSIX equivalent is: /osm2pgsql/default.style
CYGWIN environment variable option "nodosfilewarning" turns off this
warning.
Consult the user's guide for more details about POSIX paths:
http://cygwin.com/cygwin-ug-net/using.html#using-pathnames
Setting up table: planet_osm_point
NOTICE: table "planet_osm_point" does not exist, skipping
NOTICE: table "planet_osm_point_tmp" does not exist, skipping
Setting up table: planet_osm_line
NOTICE: table "planet_osm_line" does not exist, skipping
NOTICE: table "planet_osm_line_tmp" does not exist, skipping
Setting up table: planet_osm_polygon
NOTICE: table "planet_osm_polygon" does not exist, skipping
NOTICE: table "planet_osm_polygon_tmp" does not exist, skipping
Setting up table: planet_osm_roads
NOTICE: table "planet_osm_roads" does not exist, skipping
NOTICE: table "planet_osm_roads_tmp" does not exist, skipping
Allocating memory for dense node cache
Out of memory for node cache dense index, try using "--cache-strategy
sparse" in
stead
Error occurred, cleaning up

Error message for 0.81 from
https://vanguard.houghtonassociates.com/browse/OSM-OSM2PSQL-39/artifact

C:\osm2pgsql>osm2pgsql -c -d tes_maning -U postgres -W -H localhost -P
5432 -S C
:\osm2pgsql\default.style E:\Peta\shp_raung_2013\KRB_Raung\raung_barat.osm

osm2pgsql SVN version 0.81.0 (64bit id space)

Password:
Using projection SRS 900913 (Spherical Mercator)
cygwin warning:
MS-DOS style path detected: C:\osm2pgsql\default.style
Preferred POSIX equivalent is: /osm2pgsql/default.style
CYGWIN environment variable option "nodosfilewarning" turns off this
warning.
Consult the user's guide for more details about POSIX paths:
http://cygwin.com/cygwin-ug-net/using.html#using-pathnames
Setting up table: planet_osm_point
NOTICE: table "planet_osm_point" does not exist, skipping
NOTICE: table "planet_osm_point_tmp" does not exist, skipping
Setting up table: planet_osm_line
NOTICE: table "planet_osm_line" does not exist, skipping
NOTICE: table "planet_osm_line_tmp" does not exist, skipping
Setting up table: planet_osm_polygon
NOTICE: table "planet_osm_polygon" does not exist, skipping
NOTICE: table "planet_osm_polygon_tmp" does not exist, skipping
Setting up table: planet_osm_roads
NOTICE: table "planet_osm_roads" does not exist, skipping
NOTICE: table "planet_osm_roads_tmp" does not exist, skipping
Allocating memory for sparse node cache
Out of memory for sparse node cache, reduce --cache size
Error occurred, cleaning up

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/17#issuecomment-16566100
.

[springmeyer]

@onepremise. Notice that his visual screenshot is for really old osm2pgsql while the recent build is running out of memory and never finishing. @apmon can you advise if this seems odd? @emirhartato can you tell us how much memory your machine has and how large this specific osm extract is?

[apmon]

Osm2pgsql uses virtual memory fairly wasteful and kind of assumes that you have 64 bit address space and an efficient memory management. So there are a number of places, where it just allocates something like 2GB of memory with a good chance of never writing to that memory and thus never needing that amount of physical memory. However, depending on your over commit settings of your OS and how well your OS handles those situations, this might not work well.

As the error message sais, you should try the parameter "--cache-strategy sparse" which is less wastefull of virtual memory and thus might work (It roughly limits itself to what is specified in the -C parameter).

[emirhartato]

okay, update..
so it's work pretty well https://vanguard.houghtonassociates.com/browse/OSM-OSM2PSQL-39/artifact

I'm adding -H localhost and -W on my command line to get it work

Thanks everyone

[springmeyer]

@emirhartato - thanks for the update. What about the -C flag and/or the --cache-strategy sparse flag? It seems like you will need to specify those to avoid over memory usage.

@emirhartato - also, what are the next steps for you? are you good to go for your workshop, or do you need the QGIS plugin updated to also pass custom flags?

[ohadmanor]

alex85 hi,

I'm fairly new in the OSM2PGSQL I'm still wondering around, and I have some question:

1. can you explain the use of flat-nodes?
2. why Multithreat does't work with windows?

thank you for your time and effort, your work is excellent!!

[Andre-J]

@alex85k thanks, it works for me. Can this version be declared stable enough to find its way into http://wiki.openstreetmap.org/wiki/Osm2pgsql#Binary ? The server mentioned there does not respond anymore.

[alex85k]

@Andre-J : The current msvc version does not support flat-nodes and parallelism properlly, which may make it hard to use on several Gb extracts. Flat-nodes is fixable (only hard to debug with 20Gb tempfiles). Adding parallelism is much harder. Currently it is based on "fork" function which has no analogs in Windows core.

I guess for now Mingw/Cygwin version by @onepremise is more universal (but slower for small extracts):
https://vanguard.houghtonassociates.com/browse/OSM-OSM2PSQL-95/artifact
(did not test myself), I guess it could be placed to Wiki.

[Andre-J]

I tested both versions from onepromise, but did not get lucky with them: http://gis.stackexchange.com/questions/118334/download-link-to-osm2pgsql/118356#118356

[onepremise]

Sorry, wasn't aware a few dependencies were missing. I can easily add
libgcc_s_seh-1.dll and libwinpthread-1.dll into the build archive.
Otherwise, you can always retrieve them from
https://vanguard.houghtonassociates.com/browse/MINGLE-MINGLE64-203/artifact.

On Fri, Nov 7, 2014 at 4:33 AM, AndreJ notifications@github.com wrote:

I tested both versions from onepromise, but did not get lucky with them:
http://gis.stackexchange.com/questions/118334/download-link-to-osm2pgsql/118356#118356

—
Reply to this email directly or view it on GitHub
#17 (comment)
.

[Andre-J]

mingle-light.zip does not contain them, and the other one is too large to download for just 2 small files.

[ohadmanor]

alex85k hi,

I'm paying around with two version one from the Sep 15 and the other one from Oct 26, the version form Oct is faster but it crashing the one from Sep is slow (4 times slower) but it didn't crash

I'm using the following command:
osm2pgsql -c -S default.style -c 30000 -d OSM -U postgres -W -H localhost -P 5432 -j --bbox 25,25,50,50 planet-latest.osm.pbf

The HW is 96 GB RAM on windows server 2008 R2

any idea why?

thanks

[alex85k]

@ohadmanor : Thank you for the feedback! October26 version is based on newly created C++ source and uses new protobuf 2.6.0 and protobuf-c, so there are many places where the bug could be introduced. One of the previous versions was a debug one by mistake (this can explain 4x speedup). I know only about some UTF-related problem (failing tests) and flat-nodes crash.

It would be great if you can reproduce a crash on some smaller (one-country) extract, then maybe we can fix it :)

[ohadmanor]

@alex85k : I think there is a memory linkage when I try to use large scale of data. on small areas it works fine but when I try to use it on large area when it gets to the way procedure it jump to maximum memory and start to do memory allocation. then it send the following error: "Error allocating ways Error occurred, cleaning up".

when I used the same procedure on the version from Sep It works fine, slow but fine!!

I hope it will help you figure up the problem.

[alex85k]

So, this is out-of-memory error... The reasons could be:

• C++ increased memory consuption (need to meaure it on same datasets)
• C++ version may potentially have memory leaks (@pnorman - have you already checked it with some tools?)
• some Windows-dependent memory allocation error ...

I have rebuilt september C-version in Release, it should be faster:
https://dl.dropboxusercontent.com/u/63393258/osm2pgsql-september-release.zip

The latest C++ version is here:
https://dl.dropboxusercontent.com/u/63393258/osm2pgsql-november-release.zip

I have updated the CMake build scripts to support latest source: alex85k/osm2pgsql-cmake@b8d6e57

@ohadmanor : could you compare the updated version and measure memory consumption, if possible? (maybe on one-country extract)

[ohadmanor]

@alex85k : I tested both versions on small files (up to 1.5GB) it works fine (No memory linkage), I'll get back my server time tomorrow and I'll test them on large files with --bbox command.

[alex85k]

@ohadmanor Thank you. Please also track memory consumption with something like http://stackoverflow.com/questions/69332/tracking-cpu-and-memory-usage-per-process

[pnorman]

Much of the content of this topic is pre-C++. I'm wondering if we should isolate out the issues that still remain and close this particular issue

[ohadmanor]

@alex85k well I tested this two versions, both works excellent with single country file the problem is with the --bbox command both version has memory leakage (I used Red-Gate) on small files up to 2 GB it can works if you try to use the planet-latest.osm.pbf it crash in same cases it take windows with it.

I can try to debug it if you send me the latest code (if you have it on VS it will be best)

[alex85k]

@ohadmanor: I have prepared a full debug testing kit - pre-packaged libraries and cmake scripts
To use, just unpack it on D: drive and call build.bat in D:\osm2pgsql\osm2pgsql-cmake .
It will create a project osm2pgsql.sln and build it. Debugging in VS works but you'll need to include D:\osm2pgsql\libs\bin into PATH or copy all dlls to osm2pgsql.exe in Debug folder.
https://dl.dropboxusercontent.com/u/63393258/osm2pgsql_full.7z

To use the pack, only Visual Studio 2013, CMake and PostgreSQL 9.3 are needed.

@pnorman: yes, maybe the new ticket will be better.

[ohadmanor]

@alex85k I tried to debug and play around with the versions,

1. the calling to ProtobufCBuffer cause to memory linkage (I don't understand why).
2. the september version is much more stable (although the process take more time).

what is the different between the september and november?

[alex85k]

@ohadmanor September version is C, November is C++-translated (huge changes in code). Windows patches are almost the same and mostly cosmetic, so there may be new problems with Linux version too.

Does the memory leaks exist on small files too?

[ohadmanor]

@alex85k to be clear the problem is only when using the -bbox command from large files (above 2GB) and only if you try to cut area larger then 10x10 degrees.
the work around is to small pieces of 10x10 degrees with -a command on each line and it works great.

[danielkastner]

@alex85k First of all: Awesome job! I tried to import some sumps with the November-Build and it always fails. I think it might be related to UTF8 as someone mentioned earlier. Here are some infos:

Command:osm2pgsql -a -S openstreetmap-carto.style --slim -d osm -U postgres --number-processes 4 -C 20000 D:\osm_data\baden-wuerttemberg-latest.osm.pbf

Error:

Reading in file: D:\osm_data\baden-wuerttemberg-latest.osm.pbf
Processing: Node(26752k 114.3k/s) Way(4312k 37.50k/s) Relation(0 0.00/s)COPY_END for COPY planet_osm_ways FROM STDIN;
 failed: FEHLER:  fehlende Daten f├╝r Spalte ÔÇ×pendingÔÇ£
CONTEXT:  COPY planet_osm_ways, Zeile 1: ÔÇ×1677278     {7540790,266813404,444491498,645213}    {"width","6","surface","paved","name","Schwabstra├ƒe","...ÔÇ£

Error occurred, cleaning up

[alex85k]

@danielkastner : Thank you for the feedback!
I have tried the same script with smaller German extract bremen-latest.osm.pbf and had no errors. Could you try it too? (I have no PC resources to process 400Mb for now).
I also had no errors processing hand-made extract with problematic way 1677278 https://dl.dropboxusercontent.com/u/63393258/map.osm.pbf

Did you set UTF8 encoding for osm database (I had)? Are you using latest 9.3 PostgreSQL?
(however, I tried with CP1251 too - no errors...)

[alex85k]

@ohadmanor: Thank you for the details. It will be harder to catch... Did your software detect memory leaks on smaller extracts with BBox? (you can try Russia extract for example - it have > 10 degrees range and reasonable file size)

[danielkastner]

@alex85k Thanks for the quick reply. I'm not sure about the charset of the db. I'll check it and give the extract you uploaded a try.

[danielkastner]

@alex85k It's been a while but now I could give the extract you uploaded a try. Unfortanetly the same error occurs:

C:\Users\kastner\Programme\osm2pgsl_nov>osm2pgsql -a -S openstreetmap-carto.style --slim -d osm -U postgres --number-processes 4 -C 20000 D:\osm_data\map.osm.pbf
osm2pgsql SVN version 0.85-cpp-win-cmake (64bit id space)

WARNING: osm2pgsql was compiled without fork, only using one process!
Using built-in tag processing pipeline
Using projection SRS 900913 (Spherical Mercator)
Setting up table: planet_osm_point
Setting up table: planet_osm_line
Setting up table: planet_osm_polygon
Setting up table: planet_osm_roads
Allocating memory for sparse node cache
Node-cache: cache=20000MB, maxblocks=2560000*zd, allocation method=8192
Mid: pgsql, scale=100 cache=20000
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels

Reading in file: D:\osm_data\map.osm.pbf
COPY_END for COPY planet_osm_ways FROM STDIN;
 failed: FEHLER:  fehlende Daten f├╝r Spalte ÔÇ×pendingÔÇ£
CONTEXT:  COPY planet_osm_ways, Zeile 1: ÔÇ×1677278     {7540790,266813404,444491498,645213}    {"width","6","surface","paved","name","Schwabstra├ƒe","...ÔÇ£

Error occurred, cleaning up

The database is using UTF8. Here is the CREATE-Statement:

CREATE DATABASE osm
  WITH OWNER = postgres
       ENCODING = 'UTF8'
       TABLESPACE = osm
       LC_COLLATE = 'German_Germany.1252'
       LC_CTYPE = 'German_Germany.1252'
       CONNECTION LIMIT = -1;

[danielkastner]

Wait, it's working. The problem was the database or, more precisely, the tables. They were created some time ago and for the update of the data now i just deleted their contents instead of dropping the database/tables. After using osm2pgsql in create-mode instead of append-mode it worked. At least until it hit the problem with the violation of unique-constraints :/

But for your part @alex85k everything works fine now. Thanks for your support and the up2date version of osm2pgsql

[pnorman]

Given the age of this issue, I'm going to go ahead and close it. If there are new specific issues in 0.88.0 or 0.89.0-dev, they should be raised as new issues.