Skip to content
This repository was archived by the owner on Jul 24, 2021. It is now read-only.
This repository was archived by the owner on Jul 24, 2021. It is now read-only.

Osmosis crashes when loading planet.osm #4597

@openstreetmap-trac

Description

@openstreetmap-trac

Reporter: marcelo[at]picoli.com.br
[Submitted to the original trac issue database at 10.15am, Thursday, 27th September 2012]

I've been trying to load the full planet.osm data file into postgreSQL for more than two months now... and ran into a completely new problem.

The first time I tried was with a planet.osm from the beginning of August/2012, and it loaded all the nodes, ways and relations, but when it was loading the current tables, a power failure resetted the machine and I was left with only the history tables loaded. Tried loading them by hand since the history tables were the same as the current tables (no changes were applied). Had other problems not related, and gave up.

I decided to download a new version of the planet.osm, this one is from 02/Sep/2012. Unpacked everything, deleted all the tables, and tried:

osmosis -v --read-xml file=g:\OSM\planet-latest.osm --buffer bufferCapacity=100000 --log-progress --write-apidb host=localhost database=postgres user= password=

Osmosis version was 0.40.1, and postgreSQL version was 9.1

It ran for about 6 days, loaded all nodes, and at about 90% of the ways, it crashed. The message for this error was not saved, but the most critical part ("cannot have morre than 2^32-1 commands in a transaction") is the same.

Then, postgreSQL 9.2 was released, and I decided to upgrade both postgreSQL (to 9.2) and osmosis (to 0.41 - the latest at the time). Also got and used the latest schema version. Ended doing a clean postgreSQL install, and tried again with the command:

osmosis -v --fast-read-xml file=g:\OSM\planet-latest.osm --buffer bufferCapacity=100000 --log-progress --write-apidb host=localhost:5433 database=postgres user= password=

After one week, crashed with the same error (the full message is attached):

ERRO: no pode ter mais do que 2^32-1 comandos em uma transao
Onde: comando SQL "SELECT 1 FROM ONLY "osm"."ways" x WHERE "way_id" OPERATOR(pg_catalog.=) $1 AND "version" OPERATOR(p
g_catalog.=) $2 FOR SHARE OF x"

Translation:

ERROR: Cannot have more than 2^32-1 commands in a transaction Where: SQL command is "SELECT 1 FROM ONLY "osm"."ways" x WHERE "way_id" OPERATOR(pg_catalog.=) $1 AND "version" OPERATOR(p
g_catalog.=) $2 FOR SHARE OF x"

Something changed (or grew) beyond some internal limit in postgreSQL and now it is impossible to load the full planet.osm data file.

I do not have the time available to retry loading an older planet.osm and see if the error appears again.

I believe that osmosis shouldn't encapsulate ALL the commands for the FULL planet.osm load in a single transaction (or at least I should have the option of not doing it). I also believe that committing the nodes, then the ways and finally the relations would be only a temporary measure, since the database WILL grow to more than 2^32-1 nodes and (given enough time) to more than 2^32-1 ways. Or, maybe postgreSQL should start using a 64 bit value for the commands in a transaction...

Metadata

Metadata

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions