Skip to content
This repository has been archived by the owner on Sep 18, 2021. It is now read-only.

Commit

Permalink
KEST-272: proper config inheritance; log queue/alias config on reload
Browse files Browse the repository at this point in the history
RB_ID=87497
  • Loading branch information
Stephan Zuercher committed Sep 21, 2012
1 parent ea9d4da commit bf4ac4d
Show file tree
Hide file tree
Showing 12 changed files with 590 additions and 177 deletions.
10 changes: 10 additions & 0 deletions ChangeLog
Original file line number Diff line number Diff line change
@@ -1,3 +1,13 @@
2.3.4
-----
- modify configuration to allow inheritance by fanouts to masters, masters
to default.

2.3.3
-----
- change kestrel.sh to use old PID file so that daemon-based installs can
be halted

2.3.2
-----
release: 23 Aug 2012
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,7 @@ Configuration

Queue configuration is described in detail in `docs/guide.md` (an operational
guide). Scala docs for the config variables are
[here](http://robey.github.com/kestrel/doc/main/api/net/lag/kestrel/config/KestrelConfig.html).
[here](http://robey.github.com/kestrel/api/main/api/net/lag/kestrel/config/KestrelConfig.html).


Performance
Expand Down
63 changes: 58 additions & 5 deletions docs/guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,9 @@ deleted with the "delete" command.
Configuration
-------------

**NOTE:** Kestrel 2.3.4 introduces inheritance for queue configurations. For more
information, see below.

The config files for kestrel are scala expressions loaded at runtime, usually
from `production.scala`, although you can use `development.scala` by passing
`-Dstage=development` to the java command line.
Expand All @@ -48,7 +51,7 @@ The config file evaluates to a `KestrelConfig` object that's used to configure
the server as a whole, a default queue, and any overrides for specific named
queues. The fields on `KestrelConfig` are documented here with their default
values:
[KestrelConfig.html](http://robey.github.com/kestrel/api/main/api/net/lag/kestrel/config/KestrelConfig.html)
[KestrelConfig](http://robey.github.com/kestrel/api/main/api/net/lag/kestrel/config/KestrelConfig.html)

To confirm the current configuration of each queue, send "dump_config" to
a server (which can be done over telnet).
Expand All @@ -62,11 +65,61 @@ Logging is configured according to `util-logging`. The logging configuration
syntax is described here:
[util-logging](https://github.com/twitter/util/blob/master/util-logging/README.markdown)

Per-queue configuration is documented here:
[QueueBuilder.html](http://robey.github.com/kestrel/api/main/api/net/lag/kestrel/config/QueueBuilder.html)
Per-queue configuration options are documented here:
[QueueBuilder](http://robey.github.com/kestrel/api/main/api/net/lag/kestrel/config/QueueBuilder.html)

Queue alias configuration options are documented here:
[AliasBuilder](http://robey.github.com/kestrel/api/main/api/net/lag/kestrel/config/AliasBuilder.html)

Configuration Changes Starting in Kestrel 2.3.4
-----------------------------------------------

Starting with Kestrel 2.3.4, queue configurations are inherited:

* Any queue with no explict configuration (see `queues` in `KestrelConfig`) uses the default
queue configuration (see `default` in `KestrelConfig`). This behavior is unchanged from
previous versions.
* Any master (e.g. not fanout) queue with a queue configuration overrides the default queue
configuration. For example, if `default.maxMemorySize` is set, all explicitly configured
queues will inherit that setting *unless* explicitly overridden in the queue's configuration.
Older versions of Kestrel *did not* apply values from the default queue configuration to any
explicitly configured queue.
* Any fanout queue (e.g., a queue with a `+` in its name), inherits its master queue's
configuration, unless explicitly overridden (see `queues` in `KestrelConfig`). Older versions
of Kestrel silently ignored explicit fanout queue configurations.

### Example Configuration
-------------------------

Existing configurations should continue to load, but the resulting configuration may
differ. As an example, the following configuration file and table illustrate the differences
between a configuration loaded by Kestrel 2.3.3 and Kestrel 2.3.4 (and later).

new KestrelConfig {
default.maxMemorySize = 8.megabytes

queues = new QueueBuilder() {
name = "q"
maxItems = 500
} :: new QueueBuilder() {
name = "q+fanout"
maxAge = 1.minute
} :: new QueueBuilder() {
name = "x"
maxMemorySize = 16.megabytes
}
}


Queue alias configuration is documented here:
[AliasBuilder.html](http://robey.github.com/kestrel/api/main/api/net/lag/kestrel/config/AliasBuilder.html)
<table>
<tr><th>Queue</th> <th>Setting</th> <th>Kestrel <= 2.3.3</th> <th>Kestrel >= 2.3.4</th> </tr>
<tr><td>q</td> <td>maxMemorySize</td> <td>128.megabytes</td> <td>8.megabytes</td> </tr>
<tr><td>q+fanout</td> <td>maxMemorySize</td> <td>128.megabytes</td> <td>8.megabytes</td> </tr>
<tr><td>x</td> <td>maxMemorySize</td> <td>16.megabytes</td> <td>16.megabytes</td> </tr>
<tr><td>q</td> <td>maxItems</td> <td>500</td> <td>500</td> </tr>
<tr><td>q+fanout</td> <td>maxItems</td> <td>500</td> <td>500</td> </tr>
<tr><td>q+fanout</td> <td>maxAge</td> <td>None</td> <td>Some(1.minute)</td> </tr>
</table>


Full queues
Expand Down
37 changes: 25 additions & 12 deletions src/main/scala/net/lag/kestrel/QueueCollection.scala
Original file line number Diff line number Diff line change
Expand Up @@ -48,24 +48,34 @@ class QueueCollection(queueFolder: String, timer: Timer, journalSyncScheduler: S
private val aliases = new mutable.HashMap[String, AliasedQueue]
@volatile private var shuttingDown = false

@volatile private var queueConfigMap = Map(queueBuilders.map { builder => (builder.name, builder()) }: _*)
@volatile private var queueBuilderMap = Map(queueBuilders.map { builder => (builder.name, builder) }: _*)
@volatile private var aliasConfigMap = Map(aliasBuilders.map { builder => (builder.name, builder()) }: _*)

private def checkNames {
val duplicates = queueConfigMap.keySet & aliasConfigMap.keySet
val duplicates = queueBuilderMap.keySet & aliasConfigMap.keySet
if (!duplicates.isEmpty) {
log.warning("queue name(s) masked by alias(es): %s".format(duplicates.toList.sorted.mkString(", ")))
}
}

private def buildQueue(name: String, realName: String, path: String) = {
if ((realName contains ".") || (realName contains "/") || (realName contains "~")) {
private def getQueueConfig(name: String, masterName: Option[String] = None): QueueConfig = {
masterName match {
case Some(master) =>
val masterConfig = getQueueConfig(master)
queueBuilderMap.get(name).map { _.apply(Some(masterConfig)) }.getOrElse(masterConfig)
case None =>
queueBuilderMap.get(name).map { _.apply(Some(defaultQueueConfig)) }.getOrElse(defaultQueueConfig)
}
}

private def buildQueue(name: String, masterName: Option[String], path: String) = {
if ((name contains ".") || (name contains "/") || (name contains "~")) {
throw new Exception("Queue name contains illegal characters (one of: ~ . /).")
}
val config = queueConfigMap.getOrElse(name, defaultQueueConfig)
log.info("Setting up queue %s: %s", realName, config)
val config = getQueueConfig(name, masterName)
log.info("Setting up queue %s: %s", name, config)
Stats.incr("queue_creates")
new PersistentQueue(realName, path, config, timer, journalSyncScheduler, Some(this.apply))
new PersistentQueue(name, path, config, timer, journalSyncScheduler, Some(this.apply))
}

// preload any queues
Expand All @@ -80,6 +90,7 @@ class QueueCollection(queueFolder: String, timer: Timer, journalSyncScheduler: S
aliases.get(name) match {
case Some(alias) =>
alias.config = config
log.info("Reloaded alias config %s: %s", name, config)
case None =>
log.info("Setting up alias %s: %s", name, config)
val alias = new AliasedQueue(name, config, this)
Expand Down Expand Up @@ -113,10 +124,12 @@ class QueueCollection(queueFolder: String, timer: Timer, journalSyncScheduler: S
newAliasBuilders: List[AliasBuilder]) {
defaultQueueConfig = newDefaultQueueConfig
queueBuilders = newQueueBuilders
queueConfigMap = Map(queueBuilders.map { builder => (builder.name, builder()) }: _*)
queueBuilderMap = Map(queueBuilders.map { builder => (builder.name, builder) }: _*)
queues.foreach { case (name, queue) =>
val configName = if (name contains '+') name.split('+')(0) else name
queue.config = queueConfigMap.get(configName).getOrElse(defaultQueueConfig)
val masterName = if (name contains '+') Some(name.split('+')(0)) else None
val config = getQueueConfig(name, masterName)
queue.config = config
log.info("Reloaded queue config %s: %s", name, config)
}
aliasBuilders = newAliasBuilders
aliasConfigMap = Map(aliasBuilders.map { builder => (builder.name, builder()) }: _*)
Expand All @@ -139,12 +152,12 @@ class QueueCollection(queueFolder: String, timer: Timer, journalSyncScheduler: S
// only happens when creating a queue for the first time.
val q = if (name contains '+') {
val master = name.split('+')(0)
val fanoutQ = buildQueue(master, name, path.getPath)
val fanoutQ = buildQueue(name, Some(master), path.getPath)
fanout_queues.getOrElseUpdate(master, new mutable.HashSet[String]) += name
log.info("Fanout queue %s added to %s", name, master)
fanoutQ
} else {
buildQueue(name, name, path.getPath)
buildQueue(name, None, path.getPath)
}
q.setup
queues(name) = q
Expand Down
44 changes: 44 additions & 0 deletions src/main/scala/net/lag/kestrel/config/AliasConfig.scala
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
/*
* Copyright 2009 Twitter, Inc.
* Copyright 2009 Robey Pointer <robeypointer@gmail.com>
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License. You may obtain
* a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package net.lag.kestrel
package config

case class AliasConfig(
destinationQueues: List[String]
) {
override def toString() = {
("destinationQueues=[%s]").format(destinationQueues.mkString(", "))
}
}

class AliasBuilder {
/**
* Name of the alias being configured.
*/
var name: String = null

/**
* List of queues which receive items added to this alias.
*/
var destinationQueues: List[String] = Nil

def apply() = {
AliasConfig(destinationQueues)
}
}

59 changes: 59 additions & 0 deletions src/main/scala/net/lag/kestrel/config/ConfigValue.scala
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
/*
* Copyright 2012 Twitter, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License. You may obtain
* a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package net.lag.kestrel
package config

/**
* ConfigValue represents a value that may either be a default or a
* configuration-specified value. A ConfigValue may be resolved against
* another value.
*/
sealed abstract class ConfigValue[A](x: A) {
def isSpecified: Boolean
def get = x
def resolve(default: Option[A]): A = {
default match {
case Some(_) if isSpecified => get
case Some(v) => v
case None => get
}
}
}

/*
* SpecifiedValue is a ConfigValue specified in a configuration.
*/
case class SpecifiedValue[A](x: A) extends ConfigValue[A](x) {
def isSpecified = true
}

/*
* Default is a ConfigValue specified as a default.
*/
case class Default[A] private[config] (x: A) extends ConfigValue[A](x) {
def isSpecified = false
}

/**
* The ConfigValue companion object provides implicit methods to convert
* bare objects into SpecifiedValue instances.
*/
object ConfigValue {
implicit def wrap[A](x: A): ConfigValue[A] = SpecifiedValue(x)
implicit def wrapToOption[A](x: A): ConfigValue[Option[A]] = SpecifiedValue(Some(x))
implicit def wrapNone[A](x: None.type): ConfigValue[Option[A]] = SpecifiedValue(None)
}
Loading

0 comments on commit bf4ac4d

Please sign in to comment.