Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Efficient serializer for Geometry types #139

Merged
merged 1 commit into from Oct 19, 2017

Conversation

@mbasmanova
Copy link

commented Oct 18, 2017

Introduced efficient Kryo-compatible custom serializer for Geometry types. The new serializer uses a lot fewer bytes per geometry. For a Point object with no user data the custom serializer uses 23 bytes compared with 401 bytes used by default Kryo serializer and 1170 bytes used by default Java serializer. For other types, on average, custom serializer uses 5-6 times fewer bytes then default Kryo serializer and 10+ times fewer bytes then default Java serializer.

To activate custom serializer set the following configuration properties:

conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
conf.set("spark.kryo.registrator", GeoSparkKryoRegistrator.class.getName());

Efficient serializer for Geometry types
Introduced efficient Kryo-compatible custom serializer for Geometry types. The new serializer uses a lot fewer bytes per geometry. For a Point object with no user data the custom serializer uses 23 bytes compared with 401 bytes used by default Kryo serializer and 1170 bytes used by default Java serializer. For other types, on average, custom serializer uses 5-6 times fewer bytes then default Kryo serializer and 10+ times fewer bytes then default Java serializer.

To activate custom serializer one needs to set the following configuration properties:

  conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
  conf.set("spark.kryo.registrator", GeoSparkKryoRegistrator.class.getName());

@mbasmanova mbasmanova force-pushed the mbasmanova:efficient-serde branch from e05ffa5 to 44a46f2 Oct 18, 2017

@jiayuasu jiayuasu merged commit bcae361 into DataSystemsLab:master Oct 19, 2017

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.