chill-akka optional lz4 compression

[luben]

Please take a look at this branch. It implements transparent LZ4 compression of the Kryo serialized messages in the Akka serializer. By default it is disabled but could be enabled with:

akka.chill.compression = on

The lz4 jar is lazily linked on-demand if compression is used so that to keep backward compatible behaviour if it is not used.

[johnynek]

Compression with serialization is a useful idea in many cases, however I think putting this at the level of Akka is the wrong place. Many might want this in spark, storm or scalding and you could not get it with this implementation.

There are at least two ways to do this as far as I can see:

1. Make a way to combine compression with any registered kyro serializer, or all serializers (so, new CompressedSerializer[T](wraps: Serializer[T]) extends Serializer[T]
2. Make an Injection that does this compression and let users turn on using InjectiveSerializer.

What do you think is the best way?

[luben]

I was also thinking that is will be better to push down the serialization down the stack but I don't know how to configure/enable it using the standard akka configs. I see that there is an adapter for typesafe-config (used by akka) as ChillConfig (the AkkaConfig class) but I have not found a description how it could be used and how to manually register serializers.

From configuration POV using separate CompressedSerializer[T] seems cleaner. But I have not thought about the implementation details.

Is there any documentation of who to use ChillConfig? Or even just an example?