Storage Layer vs. Independent Server

[akbarahmed]

This issue is to discuss how we should look at the Dynomite, Redis dependency, and whether we should create separate packaging repos.

The main question I have is whether we should create a hard dependency between Dynomite and Redis in the Netflix/dynomite repo or whether we should create separate packaging repo(s) that define these dependencies.

By way of background, I've been working on systemd for Dynomite recently. A nice feature in systemd is that we can define dependencies. For example, we can specify that Dynomite will start after Redis, and we can define a dependency where the Dynomite system script will start Redis if Redis is not running. The short story is that we can define a strong dependency between Dynomite and Redis. We can of course do the same thing in an init script.

The above got me thinking about the role of Redis in a Dynomite cluster. On each server, we have a Dynomite instance and a Redis instance. The Dynomite instance does not function properly if Redis is unavailable. Thus there is a strong runtime dependency.

Conceptually, I'm starting to think we should look at Redis as Dynomite's storage layer rather than as an independent server. However, I have mixed feelings about this because a strong dependency on Redis will make the UX for Dynomite + Redis better, but would move Dynomite away from a generic Dynamo layer.

A better approach than creating a hard dependency on Redis in the Dynomite repo may be to create a separate "packaging" repo that treats Netflix/dynomite and antirez/redis as an upstream dependencies.

Question

What are your thoughts on creating separate packaging repos (for .deb and .rpm) that treat Netflix/dynomite as an upstream project.

We can then create separate init/systemd scripts in the packaging repos with a strong dependency between Dynomite and Redis. This will keep Netflix/dynomite as a generic Dynamo layer while the packages can provide a simpler user experience for people who want to use Dynomite with Redis.

[ipapapa]

My 2 cents: Dynomite and Redis should be kept separate. Dynomite does not have a runtime dependency on Redis only, but as a generic Dynamo layer, a runtime dependency on any storage engine, for example memcached #227 and in the future ForrestDB #228

Dynomite decides on which data store it is going to use through .yml file https://github.com/Netflix/dynomite/blob/dev/conf/dynomite.yml#L7 (data_store: 0 is Redis, data_store: 1 is Memcached etc.). We plan to undertake the effort in the future to simplify the path to any data store engine so that the data store code will be totally independent (right now only the data store protocol is under src/proto)

Hence, I agree with you that the first step is to treat Dynomite as an upstream project. Then based on configuration, we can create scripts with strong dependencies to the corresponding storage engines (it would actually be nice to support both Redis and Memcached).

[akbarahmed]

@ipapapa I think we're in agreement.