Deepcopy and Pickle support for Boost Python objects #345

Closed
artemp opened this Issue Oct 11, 2011 · 15 comments

2 participants

@artemp
Mapnik member

Pickling support will be a major advancement for Mapnik as it will allow for:

1) maps (or sub objects) to be serialized in binary form, which will be potentially 1000's of times faster to load than xml and

2) allow for bundling of map objects along with other pickled info (think rendering directives, auxillary styles, serialized data - geojson) and

3) will allow for serialization of Map state as well as the basic styles and layers and

4) will be easy to marshall into pickle-based datastores (think custom django fields or couchdb) and

5) will have potentially lots of other uses

I've added pickling support to a few of the main container objects exposed in the Python bindings. But many more objects need pickling support added.

Copy and Deepcopy are supported in boost via the same support needed for pickling.
http://www.boost.org/doc/libs/1_39_0/libs/python/doc/v2/pickle.html

We need to discuss how to accomplish this (fairly) extensive task to complete full pickle/deepcopy support of Mapnik objects in python, and this ticket will track status (and super-cede other pickle-related tickets).

List of boost python cpp files that need pickle/copy support added or extended:

  • source:trunk/bindings/python/mapnik_polygon_pattern_symbolizer.cpp [1155]
  • source:trunk/bindings/python/mapnik_polygon_symbolizer.cpp [1157]
  • source:trunk/bindings/python/mapnik_rule.cpp [1342] [1344]
  • source:trunk/bindings/python/init.py [1342] NEEDS REVIEW
  • source:trunk/bindings/python/mapnik_symbolizer.cpp [1342] [1344] NEEDS REVIEW
  • source:trunk/bindings/python/mapnik_shield_symbolizer.cpp [1158]
  • source:trunk/bindings/python/mapnik_datasource.cpp
  • source:trunk/bindings/python/mapnik_style.cpp [921] [1149]
  • source:trunk/bindings/python/mapnik_parameters.cpp [1162] NEEDS REVIEW
  • source:trunk/bindings/python/mapnik_projection.cpp [1159]
  • source:trunk/bindings/python/mapnik_line_pattern_symbolizer.cpp [1155]
  • source:trunk/bindings/python/mapnik_text_symbolizer.cpp [1164] [1298] [1314]
  • source:trunk/bindings/python/mapnik_proj_transform.cpp [1163]
  • source:trunk/bindings/python/mapnik_view_transform.cpp [1163]
  • source:trunk/bindings/python/mapnik_line_symbolizer.cpp [1160]
  • source:trunk/bindings/python/mapnik_coord.cpp [1147]
  • source:trunk/bindings/python/mapnik_stroke.cpp [1161] [1164]
  • source:trunk/bindings/python/mapnik_point_symbolizer.cpp [1295]
  • source:trunk/bindings/python/mapnik_raster_symbolizer.cpp [1154]
  • source:trunk/bindings/python/mapnik_query.cpp [1156]
@artemp
Mapnik member

[springmeyer] Note: this ticket condenses/super-cedes #233, #234, #167

@artemp
Mapnik member

[springmeyer] and #205

@artemp
Mapnik member

[springmeyer] Objects that need pickling are:

  • source:/trunk/binding/python/
@artemp
Mapnik member

[springmeyer] mapnik.Coord pickling added in r1147

@artemp
Mapnik member

[springmeyer] problem with layer pickling fixed in r1148

@artemp
Mapnik member

[springmeyer] pickling of styles fixed in r1149

@artemp
Mapnik member

[springmeyer] via brentp: http://code.google.com/p/pickled-object-database/
cc'ed josh re: stylecache

@artemp
Mapnik member

[springmeyer] fix'ed josh's email

@artemp
Mapnik member

[springmeyer] Note: I've begun work on source:trunk/bindings/python/mapnik_rule.cpp, which is the big boost::visitor dispatch spot. But, I've run out of time to get it working. If anyone starts working on it get in touch.

@artemp
Mapnik member

[springmeyer] added pickling support for PointSymbolizer in r1295. We should in the future look into pickling the actual image data...

@artemp
Mapnik member

[springmeyer] slight fix to TextSymbolizer pickling in r1298 - removing the attempt to pickle fonsets as they are not even exposed in python bindings yet (see: #348)

@artemp
Mapnik member

[springmeyer] another fix to TextSymbolizer in r1314

@artemp
Mapnik member

[springmeyer] r1342, r1344 added large patch to finish full draft support for pickling all Map objects

@springmeyer
Mapnik member

over time it has proven to be hard to keep all symbolizer pickle methods up to date, and while python pickling can be quite valuable for key api objects like mapnik.Box2d or mapnik.Query - that store state - the entire pickling of maps has not been a pressing need. What would be extremely useful is the ability to quickly clone map objects from any binding language - not just python. That is now handled in #1390. Also, serialization could be handled by moving to json based maps, but that is a separate issue that will be tied to Carto CSS syntax (see #905).

So, we should remove pickling from symbolizers and any other places where pickling does not have a strong usecase if we support map deepcopy (as per #1390).

@springmeyer
Mapnik member

closing, will handling tracking removal at #1391

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment