Permalink
Browse files

redis backend to zipkin, CR by Dan Simon

motivation
we didn't want to use the cassandra backend, so we used made an
alternative storage backend in redis.

implementation
redis data structures are generally pretty good, but we needed slightly
different ways of thinking about them for zipkin.  I have slightly
different abstractions around them, which comprise most of this code.
Index and Storage end up mostly being just about manipulating those data
structures.

Author: @mosesn
Fixes #201
URL: #201
  • Loading branch information...
1 parent 2238844 commit fbc29a875533e466e86ce38ccefd7b79dc240511 @mosesn mosesn committed with Franklin Hu Nov 19, 2012
Showing with 1,452 additions and 1 deletion.
  1. +46 −0 doc/redis.md
  2. +20 −1 project/Project.scala
  3. +41 −0 zipkin-redis/src/main/scala/com/twitter/zipkin/config/RedisIndexConfig.scala
  4. +41 −0 zipkin-redis/src/main/scala/com/twitter/zipkin/config/RedisStorageConfig.scala
  5. +35 −0 zipkin-redis/src/main/scala/com/twitter/zipkin/storage/redis/ExpiringValue.scala
  6. +66 −0 zipkin-redis/src/main/scala/com/twitter/zipkin/storage/redis/RedisHash.scala
  7. +194 −0 zipkin-redis/src/main/scala/com/twitter/zipkin/storage/redis/RedisIndex.scala
  8. +85 −0 zipkin-redis/src/main/scala/com/twitter/zipkin/storage/redis/RedisListMap.scala
  9. +48 −0 zipkin-redis/src/main/scala/com/twitter/zipkin/storage/redis/RedisSetMap.scala
  10. +54 −0 zipkin-redis/src/main/scala/com/twitter/zipkin/storage/redis/RedisSortedSetMap.scala
  11. +70 −0 zipkin-redis/src/main/scala/com/twitter/zipkin/storage/redis/RedisStorage.scala
  12. +40 −0 zipkin-redis/src/main/scala/com/twitter/zipkin/storage/redis/TimeRange.scala
  13. +93 −0 zipkin-redis/src/main/scala/com/twitter/zipkin/storage/redis/package.scala
  14. +58 −0 zipkin-redis/src/test/scala/com/twitter/zipkin/storage/redis/RedisConversionsSpec.scala
  15. +77 −0 zipkin-redis/src/test/scala/com/twitter/zipkin/storage/redis/RedisHashSpec.scala
  16. +125 −0 zipkin-redis/src/test/scala/com/twitter/zipkin/storage/redis/RedisIndexSpec.scala
  17. +96 −0 zipkin-redis/src/test/scala/com/twitter/zipkin/storage/redis/RedisListMapSpec.scala
  18. +46 −0 zipkin-redis/src/test/scala/com/twitter/zipkin/storage/redis/RedisSetMapSpec.scala
  19. +85 −0 zipkin-redis/src/test/scala/com/twitter/zipkin/storage/redis/RedisSortedSetMapSpec.scala
  20. +35 −0 zipkin-redis/src/test/scala/com/twitter/zipkin/storage/redis/RedisSpecification.scala
  21. +97 −0 zipkin-redis/src/test/scala/com/twitter/zipkin/storage/redis/RedisStorageSpec.scala
View
@@ -0,0 +1,46 @@
+## Redis
+Redis is an alternative to the default storage backend, which is Cassandra.
+
+### What is redis?
+Redis is essentially a key-value store, where the "value" at the end of the key/value pair is actually a data structure. Redis is in memory by default, although it can be persisted to disk, and now journals by default, which persists it to disk.
+
+### Installing Redis
+Your preferred package manager probably lets you install redis. The piece of redis that you absolutely need is redis-server. redis-cli will probably also come in handy, but redis-server is essential. There is more information on how to download redis [here](http://redis.io/download).
+
+### Using Redis
+You can start redis by running redis-server, which will start an instance of redis-server that listens on port 6379. For further configuration options, you can use a [redis.conf](https://raw.github.com/antirez/redis/2.6/redis.conf) file to configure it.
+
+### Zipkin + Redis
+There are a few configuration changes that must be made before you can use zipkin-redis in your zipkin project.
+
+#### Config Changes
+Go into zipkin/zipkin-collector-service/config/collector-dev.scala, and replace the lines which say:
+```scala
+ def storageConfig = new CassandraStorageConfig {
+ def cassandraConfig = _cassandraConfig
+ }
+
+ def indexConfig = new CassandraIndexConfig {
+ def cassandraConfig = _cassandraConfig
+ }
+```
+
+with
+
+```scala
+ def storageConfig = new RedisStorageConfig {
+ val host = "0.0.0.0"
+ val port = 6379
+ }
+
+ def indexConfig = new RedisIndexConfig {
+ val host = "0.0.0.0"
+ val port = 6379
+ }
+```
+Then do the same in zipkin/zipkin-query-service/config/query-dev.scala. Host and port should be the host and port your redis-server is listening on.
+
+#### SBT Changes
+Open up your project/Project.scala, and in the .dependsOn arguments list for collectorService and queryService, append redis to the end of the arguments list.
+
+Then, run your zipkin instance normally!
View
@@ -37,7 +37,7 @@ object Zipkin extends Build {
base = file(".")
) settings(
crossPaths := false
- ) aggregate(hadoop, hadoopjobrunner, test, thrift, queryCore, queryService, common, scrooge, collectorScribe, web, cassandra, collectorCore, collectorService, kafka)
+ ) aggregate(hadoop, hadoopjobrunner, test, thrift, queryCore, queryService, common, scrooge, collectorScribe, web, cassandra, collectorCore, collectorService, kafka, redis)
lazy val hadoop = Project(
id = "zipkin-hadoop",
@@ -321,6 +321,25 @@ object Zipkin extends Build {
(base / "config" +++ base / "src" / "test" / "resources").get
}
).dependsOn(common, scrooge)
+
+ lazy val redis = Project(
+ id = "zipkin-redis",
+ base = file("zipkin-redis"),
+ settings = defaultSettings ++ SubversionPublisher.newSettings
+ ).settings(
+ parallelExecution in Test := false,
+ libraryDependencies ++= Seq(
+ "com.twitter" % "finagle-redis" % FINAGLE_VERSION,
+ "org.slf4j" % "slf4j-log4j12" % "1.6.4" % "runtime",
+ "com.twitter" % "util-logging" % UTIL_VERSION
+ ) ++ testDependencies,
+
+ /* Add configs to resource path for ConfigSpec */
+ unmanagedResourceDirectories in Test <<= baseDirectory {
+ base =>
+ (base / "config" +++ base / "src" / "test" / "resources").get
+ }
+ ).dependsOn(scrooge)
}
/*
@@ -0,0 +1,41 @@
+/*
+ * Copyright 2012 Tumblr Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.twitter.zipkin.config
+
+import com.twitter.conversions.time.intToTimeableNumber
+import com.twitter.finagle.redis.Client
+import com.twitter.util.Duration
+import com.twitter.zipkin.storage.redis.RedisIndex
+
+/**
+ * RedisIndexConfig has sane defaults, except you must specify your host and port.
+ */
+trait RedisIndexConfig extends IndexConfig {
+ lazy val _client: Client = Client("%s:%d".format(host, port))
+
+ val tracesTimeToLive: Duration = 7.days
+ val port: Int
+ val host: String
+
+ /**
+ * The canonical way to make a new RedisIndex
+ */
+ def apply(): RedisIndex = new RedisIndex {
+ val database = _client
+ val ttl = Some(tracesTimeToLive)
+ }
+}
@@ -0,0 +1,41 @@
+/*
+ * Copyright 2012 Tumblr Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.twitter.zipkin.config
+
+import com.twitter.conversions.time.intToTimeableNumber
+import com.twitter.finagle.redis.Client
+import com.twitter.util.Duration
+import com.twitter.zipkin.storage.redis.RedisStorage
+
+/**
+ * RedisStorageConfig has sane defaults, except you must specify your host and port.
+ */
+trait RedisStorageConfig extends StorageConfig {
+ lazy val _client: Client = Client("%s:%d".format(host, port))
+
+ val port: Int
+ val host: String
+ val tracesTimeToLive: Duration = 7.days
+
+ /**
+ * The canonical way of making a new RedisStorage
+ */
+ def apply(): RedisStorage = new RedisStorage {
+ val database = _client
+ val ttl = Some(tracesTimeToLive)
+ }
+}
@@ -0,0 +1,35 @@
+/*
+ * Copyright 2012 Tumblr Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.twitter.zipkin.storage.redis
+
+import com.twitter.util.Time
+
+/**
+ * ExpiringValue represents a value with a time to live.
+ * @param expiresAt is the time when ExpiringValue will expire.
+ * @param value is the value that will expire.
+ */
+case class ExpiringValue[A](expiresAt: Time, value: A)
+
+object ExpiringValue {
+
+ /**
+ * @param expiresAt is a long value in seconds from the epoch.
+ */
+ def apply[A](expiresAt: Long, value: A): ExpiringValue[A] =
+ ExpiringValue(Time.fromSeconds(expiresAt.toInt), value)
+}
@@ -0,0 +1,66 @@
+/*
+ * Copyright 2012 Tumblr Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.twitter.zipkin.storage.redis
+
+import com.twitter.finagle.redis.Client
+import com.twitter.util.Future
+import org.jboss.netty.buffer.ChannelBuffer
+
+/**
+ * RedisHash is a map.
+ * @param database the redis client to use
+ * @param prefix the namespace of the map
+ */
+class RedisHash(database: Client, name: String) {
+
+ /**
+ * Makes the key refer to the value.
+ */
+ def put(key: ChannelBuffer, value: ChannelBuffer) = {
+ database.hSet(name, key, value)
+ }
+
+ /**
+ * Gets the referred to value.
+ */
+ def get(key: ChannelBuffer): Future[Option[ChannelBuffer]] =
+ database.hGet(name, key)
+
+ /**
+ * Increments the value of the value referred to by key by the specified amount.
+ */
+ def incrBy(key: ChannelBuffer, incrValue: Long): Future[Long] = database.hGet(name, key) map {
+ _ match {
+ case Some(curValue) => {
+ val sum = incrValue + chanBuf2Long(curValue)
+ database.hSet(name, key, sum)
+ sum
+ }
+ case None => {
+ database.hSet(name, key, incrValue)
+ incrValue
+ }
+ }
+ }
+
+ /**
+ * Removes the key/value pairs which have the specified keys.
+ */
+ def remove(keys: Seq[ChannelBuffer]): Future[Long] = {
+ database.hDel(name, keys) map (_.longValue)
+ }
+}
Oops, something went wrong.

0 comments on commit fbc29a8

Please sign in to comment.