-
Notifications
You must be signed in to change notification settings - Fork 20
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
11 times slower than old style :( #68
Comments
I monitored system resources during the rendering and can see the following: Old machine utizilies 100% one cpu core with postgres while rendering, then shortly renderd, done Seems it has so scan the whole database. Just saw there is another indexes.sql than upstream (which doesn't work here), https://github.com/giggls/openstreetmap-carto-de/blob/master/indexes-hstore.sql - i will try that. |
Will probably need additional indexes. Sorry, I can not really help here. Did you apply indexes-hstore.sql? Should even run faster as l10n is no longer done during rendering. |
Applying this file helped. The example meta tile is now rendered in ~60s, which is more suitable but still slower than the old render stack. |
There might well be more potential for query-optimisation. Send patches if you have some. |
I am not fit in postgresql and i hate it. Just did a "make test" on old and new style. It's 10.82s average per tile in old style and 12.63s in new style. I may use that but i expected something better. That is also a lot slower than the times mentioned here: #40 (comment) - maybe you can post what machine/settings/dataset you use to reach these times. |
What do you expect from me as a reaction to such a comment? |
I thought the importlevel l10n was invented to be faster than on rendering level, now i get slower results in a good testing scenario because i have identical machines where one was updated and other not. If this is not interesting for you OK, and if it looks normal and fine to you that the newer is slower than the older OK. I thought you may have at least an advice as you are interested in a good and fast working style in what could possibly went wrong. You are digged deeper in this material than i am. I have no knowledge of postgresql or carto or whatsoever. I am just asking if the results of my installation are normal behaviour or not. |
Well PostgreSQL query-optimization is actually a scope way beyond the purpose of this issue tracker here. Are the PostgreSQL Servers configured in comparable ways regarding important performance stuff (sync settings, memory, ...)? Which database layout, carto-de and osml10n versions are you using on your old machine? And most important: Which indexes do you have there compared to the new machine? Do they differ? Increased style complexity (inherited from upstream carto) might also be an issue I can not do that much against. I can currently only compare the version (unfortunately) still running at https://tile.openstreetmap.de (Debian 10, PostgreSQL 11, carto-de v4.24.0-de1 and old l10n 2.5.8) and my new approach and these do not differ that much as far as the database layout is concerned. The main difference is that there are separate columns for names now while the old layout did generate them on the fly from hstore. I am unsure if we really have regressions regarding the new approach here. I seriously doubt this. If you want to help me to finding bottlenecks run We can then re-run these queries with This is complicated stuff and it is not a good prerequisite that you hate PostgreSQL. |
Another hint: Make sure you turn off JIT in PostgreSQL configuration. This is in the OSM-Carto installation instructions meanwhile (https://github.com/gravitystorm/openstreetmap-carto/blob/master/INSTALL.md) - it is not yet here. This can lead to quite significant performance differences in my experience. |
Now some comparisons (old => new)
Relevant changes i applied to both postgres instances (didn't compare stock settings):
full_page_writes / autovacuum is turned on after import. Of course now JIT disabled in new instance. Old database structure (sorry for German)
New one
It may however not worth digging this down further because it is fast enough now and faster than the old one. Just wanted to reply. I'm fine if you're like to close the issue now. But i highly would recommend mention these two things in
Thanks, |
I already added a hint regarding indexes-hstore.sql to INSTALL-de.md and removed From what I see it looks like you almost certainly did apply Try the following on both machines:
Regarding the JIT issue I did not yet stumble upon this myself as my machine do not yet run the new code. I will need to update this Repository to upstream version 5.6 anyway and change the Hopefully I will be able to migrate https://tile.openstreetmap.de to an ansible based setup in the foreseeable future. |
Oh forget to mention that you do not want to disable |
|
It makes sense to disable it during initial import but not afterwards and during incremental updates. |
Ahh ok then its fine, i do not run incremental updates. |
Old:
New:
|
Quite surprising that this did actually work reasonably fast at all without special indexes. |
I am surprised too ^^ |
Expected behavior
Rendering at the same or better speed level due to optimizations and newer base software than the older german render stack based on https://github.com/giggls/mapnik-german-l10n
Actual behavior
Two machines, identical hardware, same configuration of base software (postgres etc.)
First with Debian 9, Postgres 9.6, Europe OSM Data, Mapnik 3, renderd, mod_tile and the old style
Second with Debian 11, Postgres 13, Europe OSM Data, Mapnik 3.1, renderd, mod_tile and this newer style
Testing with rendering of tile: 11/1099/672.png (renderd: Rendering projected coordinates 11 1096 672 -> 1408887.305353|6731350.458909 1565430.339281|6887893.492837 to a 8 x 8 tile)
First machine with production load renders the corresponding meta tile in 38 seconds
Second machine idling out of production renders the meta tile 427 seconds :(
Any advice?
The text was updated successfully, but these errors were encountered: