Merge pull request elastic#86 from lusis/master

Updated docs for AMQP also added a "learn you some amqp for great justice"

docs/getting-started-centralized.md

@@ -116,7 +116,7 @@ Here's a good sample config:
       # this output if you don't need it.
       stdout { }
 
-      # Ship events to the amqp fanout queue named 'rawlogs"
+      # Ship events to the amqp fanout exchange named 'rawlogs"
       amqp {
         host => "myamqpserver"
         exchange_type => "fanout"
@@ -146,8 +146,8 @@ parse them to use as the real timestamp value for the event.
         # ship logs to the 'rawlogs' fanout queue.
         type => "all"
         host => "myamqpserver"
-        exchange_type => "fanout"
-        name => "rawlogs"
+        exchange => "rawlogs"
+        name => "rawlogs_consumer"
       }
     }
 

docs/index.html.erb

@@ -18,6 +18,7 @@ layout: content_right
   <ul>
     <li> <a href="tutorials/10-minute-walkthrough"> 10-minute walkthrough</a> - a simple walkthrough to show you how to configure the logstash agent to process events and even old logs. </li>
     <li> <a href="tutorials/metrics-from-logs"> Gathering metrics from logs </a> - take metrics from logs and ship them to graphite, ganglia, and more. </li>
+    <li> <a href="just-enough-amqp-for-logstash">Just enough AMQP for Logstash </a> - Get a quick primer on AMQP and how to use it in Logstash! </li>
   </ul>
 
   <h3> plugin docs </h3>

docs/just-enough-amqp-for-logstash.md

@@ -0,0 +1,156 @@
+---
+title: Just Enough AMQP - logstash
+layout: content\_right
+---
+
+While configuring your AMQP broker is out of scope for logstash, it's important to understand how
+logstash uses AMQP. To do that, we need to understand a little about AMQP.
+
+You should also consider reading [this](http://www.rabbitmq.com/tutorials/amqp-concepts.html) at the RabbitMQ website.
+
+# Exchanges, queues and bindings; OH MY!
+You can get a long way by understanding a few key terms.
+
+## Exchanges
+Exchanges are for message **producers**. In Logstash, we map these to **outputs**.
+Logstash puts messages on exchanges.
+There are many types of exchanges and they are discussed below.
+
+## Queues
+Queues are for message **consumers**. In Logstash, we map these to inputs.
+Logstash reads messages from queues.
+Optionally, queues can consume only a subset of messages. This is done with "routing keys".
+
+## Bindings
+Just having a producer and a consumer is not enough. We must `bind` a queue to an exchange.
+When we bind a queue to an exchange, we can optionally provide a routing key.
+Routing keys are discussed below.
+
+## Broker
+A broker is simply the AMQP server software. There are several brokers but the most common (and arguably popular) is [RabbitMQ](http://www.rabbitmq.com).
+Some others are Apache Qpid (and the commercial version - RedHat MRG)
+
+# Routing Keys
+Simply put, routing keys are somewhat like tags for messages. In practice, they are hierarchical in nature
+with the each level separated by a dot:
+
+- `messages.servers.production`
+- `sports.atlanta.baseball`
+- `company.myorg.mydepartment`
+
+Routing keys are really handy with a tool like logstash where you
+can programatically define the routing key for a given event using the metadata that logstash provides:
+
+- `logs.servers.production.host1`
+- `logs.servers.development.host1.syslog`
+- `logs.servers.application_foo.critical`
+
+From a consumer/queue perspective, routing keys also support two types wildcards - `#` and `*`.
+
+- `*` matches any single word.
+- `#` matches any number of words and behaves like a traditional wildcard.
+
+Using the above examples, if you wanted to bind to an exchange and see messages for just production,
+you would use the routing key `logs.servers.production.*`. If you wanted to see messages for host1, regardless of environment
+you could use `logs.servers.%.host1.#`.
+
+Wildcards can be a bit confusing but a good general rule to follow is to use `*` in places where you need wildcards for a known element.
+Use `#` when you need to match any remaining placeholders. Note that wildcards in routing keys only make sense on the consumer/queue binding,
+not in the publishing/exchange side.
+
+We'll get into some of that neat stuff below. For now, it's enough to understand the general idea behind routing keys.
+
+# Exchange types
+There are three primary types of exchanges that you'll see.
+
+## Direct
+A direct exchange is one that is probably most familiar to people. Message comes in and, assuming there is a queue bound, the message is picked up.
+You can have multiple queues bound to the same direct exchange. The best way to understand this pattern is pool of workers (queues) that read from a
+direct exchange to get units of work. Only one consumer will see a given message in a direct exchange.
+
+You can set routing keys on messages published to a direct exchange. This allows you do have workers that do different tasks read from the same global
+pool of messages yet consume only the ones they know how to handle.
+
+The RabbitMQ concepts guide (linked below) does a good job of describing this visually [here](http://www.rabbitmq.com/img/tutorials/intro/exchange-direct.png)
+
+## Fanout
+Fanouts are another type of exchange. Unlike direct exchanges, every queue bound to a fanout exchange will see the same messages.
+This is best described as a PUB/SUB pattern. This is helpful when you need broadcast messages to multiple interested parties.
+
+Fanout exchanges do NOT support routing keys. All bound queues see all messages.
+
+## Topic
+Topic exchanges are special type of fanout exchange. Fanout exchanges don't support routing keys. Topic exchanges do support them.
+Just like a fanout exchange, all bound queues see all messages with the additional filter of the routing key.
+
+# AMQP in logstash
+As stated earlier, in Logstash, Outputs publish to Exchanges. Inputs read from Queues that are bound to Exchanges.
+Logstash uses the `bunny` AMQP library for interaction with a broker. Logstash endeavors to expose as much of the configuration for both exchanges and queues.
+There are many different tunables that you might be concerned with setting - including things like message durability or persistence of declared queues/exchanges.
+See the relevant input and output documentation for AMQP for a full list of tunables.
+
+# Sample configurations, tips, tricks and gotchas
+There are several examples in the logstash source directory of AMQP usage, however a few general rules might help eliminate any issues.
+
+## Check your bindings
+If logstash is publishing the messages and logstash is consuming the messages, the `exchange` value for the input should match the `name` in the output.
+
+sender agent
+
+```
+input { stdin { type = "test" } }
+output {
+  amqp {
+    name => "test_exchange"
+    host => "my_amqp_server"
+    exchange_type => "fanout"
+  }
+}
+```
+
+receiver agent
+
+```
+input {
+  amqp {
+    name => "test_queue"
+    host => "my_amqp_server"
+    exchange => "test_exchange" # This matches the exchange declared above
+  }
+}
+output { stdout { debug => true }}
+```
+
+## Message durability
+By default, logstash will attempt to ensure that you don't lose any messages. This is reflected in the AMQP default settings as well.
+However there are cases where you might not want this. A good example is where AMQP is not your primary method of shipping.
+
+In the following example, we use AMQP as a sniffing interface. Our primary destination is the embedded ElasticSearch instance. We have
+a secondary AMQP output that we use for duplicating messages. However we disable durability on this interface so that messages
+don't pile up waiting for delivery. We only use AMQP when we want to watch messages in realtime. Additionally, we're going to leverage
+routing keys so that we can optionally filter incoming messages to subsets of hosts. The exercise of getting messages to this logstash
+agent are left up to the user.
+
+```
+input { # some input definition here}
+output {
+  elasticsearch { embedded => true }
+  amqp {
+    name => "logtail"
+    host => "my_amqp_server"
+    exchange_type => "topic" # We use topic here to enable pub/sub with routing keys
+    key => "logs.%{host}"
+    durable => false # If there are no bindings, messages will essentially devnull
+    persistent => false # If the logstash agent disconnects, the declared exchange will go away with it
+  }
+}
+```
+
+Now if you want to stream logs in realtime, you can use the programming language of your choice to bind a queue to the `logtail` exchange.
+If you do not specify a routing key, you will see every message that comes in to logstash. However, you can specify a routing key like
+`logs.apache1` and see only messages from host `apache1`.
+
+Note that any logstash variable is valid in the key definition. This allows you to create really complex routing key hierarchies for advanced filtering.
+
+Note that RabbitMQ has specific rules about durability and persistence matching on both the queue and exchange. You should read the RabbitMQ documentation
+to make sure you don't crash your RabbitMQ server with messages awaiting someone to pick them up.

etc/agent.lgtm.conf

@@ -16,15 +16,23 @@ input {
 }
 
 output {
+  # This will be your durable shipping mechanism
   amqp {
     host => "myamqpserver"
     exchange_type => "fanout"
     name => "rawlogs"
   }
+  # This is an optional non-durable shipping mechanism
+  # With this, you can sniff logs from your own code
   amqp {
     host => "127.0.0.1"
     exchange_type => "topic"
     name => "logsniff"
+    durable => false
+    persistent => false
+    # The following is optional
+    # but allows you to consume based on sender
+    key => "logstash.%{host}"
   }
   stdout { }
 }

etc/examples/indexer.conf

@@ -3,7 +3,7 @@ input {
     host => "127.0.0.1"
     user => "guest"
     pass => "guest"
-    exchange_type => "topic"
+    exchange => "logstash"
     name => "testing"
     type => "all"
   }