Providing Windows binaries/packages

[springmeyer]

In addition to OS X (#15) the osm2pgsql community of developers should provide binaries for Windows.

Let's coordinate how to get this done via this ticket.

Tasks:

• - code changes needed to support compiling under mingw, msvc, and/or cygwin
• - volunteer(s) to create binaries on windows and to test them
• - writing an easy installer using nsis that should produce a working executable without any additional installs or development toolkits and should work on 32/64 bit systems ranging from windows XP -> Win8.

Background discussions: hotosm/tech#1

[springmeyer]

/cc @onepremise - based on your build system so you have a binary yet for osm2pgsql that runs?

[springmeyer]

/cc @emirhartato, member of @hotosm who I know would love to get access to a working build to use in a workshop by the last week in April

[onepremise]

Hi Dane,

Yes I have a cygwin build which is working. I'm also getting together a
MinGW build which works with the new MinGW64 environment I've built. Let me
know which you're interested in so I can make it available.

Thanks,

Jason

On Mon, Apr 8, 2013 at 8:48 PM, Dane Springmeyer
notifications@github.comwrote:

/cc @emirhartato https://github.com/emirhartato, member of @hotosmhttps://github.com/hotosmwho I know would love to get access to a working build to use in a workshop
by the last week in April

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/17#issuecomment-16088128
.

[springmeyer]

Jason, both would be great for testing, but really whatever you have.

[emirhartato]

Hello, sorry for late reply.

This the error log for different build of osm2pgsql on Windows:
https://gist.github.com/emirhartato/5360318
https://gist.github.com/emirhartato/5360305

Build 0.69 that I got from http://tile.openstreetmap.org/osm2pgsql.zip doesn't shows any error. But the polygon sometimes doesn't work and polyline for road sometimes get messy. Here's the screenshot:

I'm running on MacOSX with 0.70, seems to work very well. But this build is not available for Windows.

[apmon]

Version 0.69 is pretty old. From what I can gather it is from sometime in 2010. That is presumably still pre 64 bit ID, so it won't work with the current planet at all, which is why it desperately needs replacing. People have compiled much newer working versions of osm2pgsql on windows though. They were just never put on tile.osm.org.

The errors shown in your logs look like configuration errors with the database parameters not correctly set. It might be possible that the defaults have changed in newer versions which is why you might need to set db parameters which you didn't use to need to do.

[emirhartato]

@apmon can you explain how to set database parameters? Hopefully if that can help solve the problem. There were 6 laptops tested and all of them have same error log. I also did a fresh install of PostgreSQL and PostGIS and create a new database, but still have the same error log.

[onepremise]

Not sure if it helps, but here are my notes on the postgres configuration:
Tune Parameters for postgresql.conf

• autovacuum = off
• checkpoint_segments = 20
• shared_buffers = 512MB # min 128kB
• work_mem = 256MB # min 64kB
• maintenance_work_mem = 512MB # min 1MB
• synchronous_commit = off

Import example:

osm2pgsql.exe -v -c -d osm -U osm -W -H localhost -P 5432 -S default.style
-s -C 1600 --hstore --number-processes=4 -r pbf
....\data\us-midwest-northeast-pacific-south-west.osm.pbf

I also have a recent build of osm2psql which works. I have some mapnik
debugging to get through on mingw64, but I'll make it all available to our
build server here:

https://vanguard.houghtonassociates.com

The next build will have a package you can download under the Artifacts tab:

https://vanguard.houghtonassociates.com/browse/OSM-OSM2PSQL-27/artifact

Is there a specific time-frame you need this?

On Thu, Apr 11, 2013 at 2:18 AM, emirhartato notifications@github.comwrote:

@apmon https://github.com/apmon can you explain how to set database
parameters? Hopefully if that can help solve the problem. There were 6
laptops tested and all of them have same error log. I also did a fresh
install of PostgreSQL and PostGIS and create a new database, but still have
the same error log.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/17#issuecomment-16218534
.

[emirhartato]

Hello...

Well, there will be a workshop in 6th May. My team will teach osm2pgsql and
we will go through OSM Tool GUI in QGIS rather than running it through
command prompt since our target participants are not an IT expert or geek.

Our team also already have Advanced OpenStreetMap documentation written by
Jeff Haack and it's available in English and Bahasa Indonesia and there's a
chapter that provide step-by-step tutorial about setting up PostGIS and
osm2pgsql. We will published it soon on learosm.org site but we can't
published it until this issue is solved.

But if that's the only way to work, it's still possible to change the
tutorial on documentation but it would be nuce if we have a working
osm2pgsql with OSM Tool GUI in QGIS for Windows.
On Apr 11, 2013 10:55 PM, "Jason Huntley" notifications@github.com wrote:

Not sure if it helps, but here are my notes on the postgres configuration:
Tune Parameters for postgresql.conf

• autovacuum = off
• checkpoint_segments = 20
• shared_buffers = 512MB # min 128kB
• work_mem = 256MB # min 64kB
• maintenance_work_mem = 512MB # min 1MB
• synchronous_commit = off

Import example:

osm2pgsql.exe -v -c -d osm -U osm -W -H localhost -P 5432 -S default.style
-s -C 1600 --hstore --number-processes=4 -r pbf
....\data\us-midwest-northeast-pacific-south-west.osm.pbf

I also have a recent build of osm2psql which works. I have some mapnik
debugging to get through on mingw64, but I'll make it all available to our
build server here:

https://vanguard.houghtonassociates.com

The next build will have a package you can download under the Artifacts
tab:

https://vanguard.houghtonassociates.com/browse/OSM-OSM2PSQL-27/artifact

Is there a specific time-frame you need this?

On Thu, Apr 11, 2013 at 2:18 AM, emirhartato notifications@github.comwrote:

@apmon https://github.com/apmon can you explain how to set database
parameters? Hopefully if that can help solve the problem. There were 6
laptops tested and all of them have same error log. I also did a fresh
install of PostgreSQL and PostGIS and create a new database, but still
have
the same error log.

—
Reply to this email directly or view it on GitHub<
https://github.com/openstreetmap/osm2pgsql/issues/17#issuecomment-16218534>

.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/17#issuecomment-16243529
.

[apmon]

My guess would be the important parameters to resolve your issues are the following:

"-U osm -W -H localhost -P 5432"

-W forces a password promt where you can specify the password for the user osm. However, you have to make sure that postgresql is set up for password authentication and that you have set the appropriate password for the osm user in postgresql.

QGis should not be affected by any of this, as it simply uses an existing postgis db, no matter how you created it.

I have not heard of OSM Tool GUI before, so I don't know what it does. (And google doesn't seem to give me anything useful either) But assuming it is a wrapper around osm2pgsql to make sure you don't have to work with the command line, then you would simply have to extend OSM Tool GUI to set the appropriate parameters of osm2pgsql.

Oh, reading your message again, I see OSM Tool GUI is something / a plugin of QGis. But the same should still apply, that the integration module needs to be adapted to set the correct parameters when calling osm2pgsql. Just that it then does effect QGis... ;-)

You probably want to test it on the command line first. Make sure it works, and then contact the developers of the tool gui.

[springmeyer]

@apmon - Yes, I wrote "OSM Tool GUI" a while back: it is a python wrapper that works with the 0.69 (old) osm2pgsql (https://bitbucket.org/springmeyer/hot_tools/src/2321b0d0adae/osm_tools/command_dock2.py). Getting the password prompt working was extremely hard. Also when I've tried to compile osm2pgsql the getopt code has been the most difficult to port. So I presume that something changed in the behavior of the password prompt at some point, breaking the plugin.

@emirhartato - can you focus on testing osm2pgsql from the command line ONLY. Do not test with the QGIS plugin. We want to first make sure that an updated osm2pgsql build for windows works from the command line. After that I can take a look at any updates needed for the plugin to adapt it.

@emirhartato - my sense is you need to provide the -W flag when testing to that a password prompt comes up.

[apmon]

Well, that makes the task of fixing the "OSM Tool GUI" somewhat easier, if we know who to contact and know they are still interested in fixining things... ;-)

If password prompt is causing issues, it shouldn't be too difficult to try and add a different way to supply the password to osm2pgsql. One option would be to just add it as another parameter. However, the disadvantage is that the command line arguments are listed in the process outputs like top (not sure if that is the case in windows), so the password would then be publicly visible to everyone on the computer.

But yes, we first need to know that osm2pgsql itself works before fixing the wrapper around it.

@onepremise Nice to see those efforts. Having any form of working osm2pgsql on windows is obviously a good start, but the more "native" the compile is, the better. If I understand it correctly mingw is more "native" than cygwin, so that would be preferable. If we can get it to build from a clean source in git and can then add a windows build server to the ci.osm.org, then perhaps we can prevent it from constantly breaking and make windows a properly supported platform of osm2pgsql.

[springmeyer]

+1 to an insecure and easy way to pass a password for windows users. I configure my linux/osx postgres installs to not need a password, but the postgres windows installers require a password based on how the installers work. So a backdoor here for development would be really handy.

[onepremise]

Nice to see those efforts. Having any form of working osm2pgsql on windows is obviously a good start, but the more "native" the compile is, the better. If I understand it correctly mingw is more "native" than cygwin, so that would be preferable.

Correct, mingw64 is definitely preferred, and I will providing the mingw64 build as soon as i tie up some lose ends with a Mapnik issue I'm working on in mingw 64 bit.

I'm very close to providing a stable build with MinGW-AD64S:

https://github.com/onepremise/MinGW-AD64S

Until then, I have the cygwin version, with 64bit IDs enabled, available here:

https://vanguard.houghtonassociates.com/browse/OSM-OSM2PSQL-35/artifact

Another build will follow up with the latest osm2pgsql updates.