OpenStreetMap Carto 4 #161

Closed
pnorman opened this Issue May 10, 2017 · 9 comments

Comments

Projects
None yet
4 participants
@pnorman
Contributor

pnorman commented May 10, 2017

The schema change has been merged to osm-carto master, and we expect to release a version with the changes later this month. Because this requires a reimport, I've opened this issue to track any planning and questions.

The rendering of 4.0.0 and 3.3.0 will be compatible, as will the next two minor releases after it. This gives time to reimport without needing to do all the servers at once.

  • Space requirements should be within 5% of the existing schema
  • Import time might take about 4 hours longer
  • osm2pgsql 0.86.0 or later is required, but I'd recommend the latest release you can easily get through backports
  • there's probably an interaction with #143
@tomhughes

This comment has been minimized.

Show comment
Hide comment
@tomhughes

tomhughes May 10, 2017

Member

When you say we can easily get osm2pgsql 0.86.0 through backports do you mean it's already in xenial-backports or just that we should be able to backport it easily?

Member

tomhughes commented May 10, 2017

When you say we can easily get osm2pgsql 0.86.0 through backports do you mean it's already in xenial-backports or just that we should be able to backport it easily?

@tomhughes

This comment has been minimized.

Show comment
Hide comment
@tomhughes

tomhughes May 10, 2017

Member

Scratch that I see we already have 0.88.1 anyway.

Member

tomhughes commented May 10, 2017

Scratch that I see we already have 0.88.1 anyway.

@pnorman

This comment has been minimized.

Show comment
Hide comment
@pnorman

pnorman May 10, 2017

Contributor

When you say we can easily get osm2pgsql 0.86.0 through backports do you mean it's already in xenial-backports or just that we should be able to backport it easily?

No, I'm suggesting you get the most recent release you can. I haven't been following Ubuntu versions, but it looks like it is 0.88.1

Contributor

pnorman commented May 10, 2017

When you say we can easily get osm2pgsql 0.86.0 through backports do you mean it's already in xenial-backports or just that we should be able to backport it easily?

No, I'm suggesting you get the most recent release you can. I haven't been following Ubuntu versions, but it looks like it is 0.88.1

@pnorman

This comment has been minimized.

Show comment
Hide comment
@pnorman

pnorman May 23, 2017

Contributor

We've tagged 4.0.0, but haven't done a release announcement yet.

Contributor

pnorman commented May 23, 2017

We've tagged 4.0.0, but haven't done a release announcement yet.

@tomhughes

This comment has been minimized.

Show comment
Hide comment
@tomhughes

tomhughes May 23, 2017

Member

Assuming that the 3.3.0 rollout has stabilised a bit by the end of the week I was planning to start rolling this out maybe next weekend (I hadn't actually realised it wasn't released yet!) so I guess the main thing we will need to know is what options we should be using to osm2pgsql for the reload? Previously I have used this:

osm2pgsql --slim --cache=32768 --flat-nodes=/store/database/nodes --number-processes=6 planet.pbf
Member

tomhughes commented May 23, 2017

Assuming that the 3.3.0 rollout has stabilised a bit by the end of the week I was planning to start rolling this out maybe next weekend (I hadn't actually realised it wasn't released yet!) so I guess the main thing we will need to know is what options we should be using to osm2pgsql for the reload? Previously I have used this:

osm2pgsql --slim --cache=32768 --flat-nodes=/store/database/nodes --number-processes=6 planet.pbf
@pnorman

This comment has been minimized.

Show comment
Hide comment
@pnorman

pnorman May 23, 2017

Contributor

The required options are osm2pgsql -G --hstore --style openstreetmap-carto.style --tag-transform-script openstreetmap-carto.lua -d gis ~/path/to/data.osm.pbf, where the openstreetmap-carto.* files come from the osm-carto repo.

To this you will want to add --slim --cache 36000 --flat-nodes ... --number-processes 8 to be able to import the entire planet and for performance, assuming the machines have >=64GB RAM and >= 12 threads.

It would also be possible to add --exclude-invalid-polygon which would make osm2pgsql stricter with polygons, and closer to the upcoming behaviour of 0.94.0. It would help mapper feedback on invalid data, at the cost of a worse rendering where there is invalid data. There's also a minor performance boost with it. There are merits in excluding them or not, and I'm slightly in favour of it. @lonvia is also in favour of it.

Contributor

pnorman commented May 23, 2017

The required options are osm2pgsql -G --hstore --style openstreetmap-carto.style --tag-transform-script openstreetmap-carto.lua -d gis ~/path/to/data.osm.pbf, where the openstreetmap-carto.* files come from the osm-carto repo.

To this you will want to add --slim --cache 36000 --flat-nodes ... --number-processes 8 to be able to import the entire planet and for performance, assuming the machines have >=64GB RAM and >= 12 threads.

It would also be possible to add --exclude-invalid-polygon which would make osm2pgsql stricter with polygons, and closer to the upcoming behaviour of 0.94.0. It would help mapper feedback on invalid data, at the cost of a worse rendering where there is invalid data. There's also a minor performance boost with it. There are merits in excluding them or not, and I'm slightly in favour of it. @lonvia is also in favour of it.

@imagico

This comment has been minimized.

Show comment
Hide comment
@imagico

imagico May 23, 2017

It would also be possible to add --exclude-invalid-polygon which would make osm2pgsql stricter with polygons, and closer to the upcoming behaviour of 0.94.0.

This question essentially amounts to how strict you want to be regarding polygon validity. Right now this is as lenient as it gets, accepting anything into the database.

Libosmium (and therefore 0.94.0) currently only rejects non-noded self intersections and open rings i think - which only amounts to parts of the invalid geometries in the database. I assume --exclude-invalid-polygon relies on GEOS IsValid() which if i am not mistaken also rejects duplicate segments. That would be a large number of features. See http://area.jochentopf.com/stats/ for current numbers.

imagico commented May 23, 2017

It would also be possible to add --exclude-invalid-polygon which would make osm2pgsql stricter with polygons, and closer to the upcoming behaviour of 0.94.0.

This question essentially amounts to how strict you want to be regarding polygon validity. Right now this is as lenient as it gets, accepting anything into the database.

Libosmium (and therefore 0.94.0) currently only rejects non-noded self intersections and open rings i think - which only amounts to parts of the invalid geometries in the database. I assume --exclude-invalid-polygon relies on GEOS IsValid() which if i am not mistaken also rejects duplicate segments. That would be a large number of features. See http://area.jochentopf.com/stats/ for current numbers.

@tomhughes

This comment has been minimized.

Show comment
Hide comment
@tomhughes

tomhughes Jun 26, 2017

Member

Carto 4.0.0 is now fully deployed.

Member

tomhughes commented Jun 26, 2017

Carto 4.0.0 is now fully deployed.

@tomhughes tomhughes closed this Jun 26, 2017

@sommerluk

This comment has been minimized.

Show comment
Hide comment
@sommerluk

sommerluk Jun 26, 2017

@tomhughes Thanks a lot for your work!

@tomhughes Thanks a lot for your work!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment