Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make render-time use of mapnik::projection faster and more lock-free #1703

Closed
springmeyer opened this issue Jan 28, 2013 · 5 comments · Fixed by #1705
Closed

Make render-time use of mapnik::projection faster and more lock-free #1703

springmeyer opened this issue Jan 28, 2013 · 5 comments · Fixed by #1705

Comments

@springmeyer
Copy link
Member

Because we store srs values for mapnik::map and mapnik::layer as strings, during rendering we dynamically set up mapnik::projection objects. The problem with this design is that the creation of mapnik::projection objects is very expensive because a proj4 projection is initialized inside and proj4 locks potentially multiple times: 1) while creating it's proj_ctx. (even with latest proj4), and 2) when reading the CSV files in PROJ_LIB in order to translate ESPG codes to proj4 literal strings when using projection syntax like +init=epsg:3857.

We aim for a totally lock-free rendering pipeline but currently we face at least layers * 2 # of locks per render because of the first creation of a mapnik::projection object and then due to the copy when it is attached to the mapnik::proj_transform object. This is a non-trivial problem if many maps are being rendered concurrently and especially tragic given that this overhead will occur even if the map is not needing to reproject any data.

potential solutions

  1. Store mapnik::projection objects on layer and map objects. This would avoid needing to allocate (and de-allocate) them at render time

  2. Store const& to projection objects in the proj_transform class to avoid extra copies and allocations

  3. Implement a native is_geographic function that does not require proj4. In cases where no-reprojection will happen for a given render then this could set us up for not needing to use proj4 at all

  4. Implement custom transformations for wgs84 and mercator, so for the extremely common case of data stored in wgs84 and being projected into mercator we could still avoid all overhead of proj initialization and locking.

Also, we should make proj4 able to be fully disabled at compile time if you don't need any fancy projection support.

@springmeyer
Copy link
Member Author

  1. above is a huge, obvious win and doubles results from initial benchmarks. The catch however is that from python we need to make changes like c92c8f1 to avoid crashes.

@springmeyer
Copy link
Member Author

The limitation of 3. above is when the +init=epsg syntax is used we cannot known whether a projection is geographic by its srid along. It needs to be looked up in the proj4 CSV files which is not something I think we should be re-implementing in mapnik.

@springmeyer
Copy link
Member Author

3 and 4 above allow for completely avoiding proj4 initialization for the merc<->longlat transformation case. Benchmarks show a massive different in speed from this work:

11) threaded -> lonlat -> merc coord transformation with proj4 init (literal): 35020 milliseconds
12) threaded -> lonlat -> merc coord transformation with lazy proj4 init (literal): 270 milliseconds

springmeyer pushed a commit that referenced this issue Jan 28, 2013
…2. centralize proj4 strings into constants, 3. tweak projection benchmarking to allocate objects in loop so we can test that specifically - refs #1703
@springmeyer
Copy link
Member Author

2,3,4 now done and available for testing in lazy-proj4 branch: https://github.com/mapnik/mapnik/compare/lazy-proj4

@springmeyer
Copy link
Member Author

for reference, current tests (in master) test purely the tr.forward() speed. My goal here is to reduce the time it takes to create a mapnik::projection so I will be changing the tests to actually create the projection instances on the stack in the loop. But for reference, until I make that change, current benchmarks are:

OS X 10.7 with proj 4.8 installed via homebrew:

$ ./benchmark/run 11 12 13 14
starting benchmark…
11) threaded -> lonlat -> merc coord transformation (epsg): 670 milliseconds
12) threaded -> merc -> lonlat coord transformation (epsg): 170 milliseconds
13) threaded -> lonlat -> merc coord transformation (literal): 670 milliseconds
14) threaded -> merc -> lonlat coord transformation (literal): 540 milliseconds
...benchmark done

Ubuntu 11.10 with proj 4.7

# ./benchmark/run 11 12 13 14
starting benchmark…
11) threaded -> lonlat -> merc coord transformation (epsg): 34410 milliseconds
12) threaded -> merc -> lonlat coord transformation (epsg): 110 milliseconds
13) threaded -> lonlat -> merc coord transformation (literal): 30400 milliseconds
14) threaded -> merc -> lonlat coord transformation (literal): 23550 milliseconds
...benchmark done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant