Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Browse files

Write a lein task for packaging the deb.

  • Loading branch information...
commit 4e9a58168efc0b1c3d64de8bb28007f0b6ba0dd5 1 parent 9a348bd
@aphyr authored
View
1  .lein-classpath
@@ -0,0 +1 @@
+:tasks
View
28 pkg/README.markdown
@@ -1,28 +0,0 @@
-Riemann
-=======
-
-Riemann is a network event stream processor. It is designed for monitoring,
-analytics, and alerts for events from multiple services. Riemann listens on
-port 5555 for protocol buffer messages containing events and processes them
-through various streams.
-
-You can use Riemann to graph the average rate of requests in your application,
-email responsible parties every time an exception is thrown, and plot the 50th,
-95th, and 99th percentile latencies for your HTTP service. It is a tool to make
-writing comprehensive, site-specific analytics easy.
-
-[Main Riemann Site](http://aphyr.github.com/riemann/)<br />
-
-Plan
-====
-
-I built Riemann with the goal of getting it out the door as quickly as
-possible. There are many slow or kludgy parts, but they should all be readily
-replaceable as I find the time. Top on my list:
-
-- Add a raw Netty UDP listener for accepting events. We lose a lot of time to
-aleph.tcp.
-- Use Korma/HSQL to implement a faster index for query-heavy installations.
-- Think more carefully about time-partitioning functions.
-- Reservoir sampling
-- Event pubsub
View
9 pkg/deb/riemann
@@ -0,0 +1,9 @@
+#!/bin/bash
+if [ -z "$1" ]
+then
+ config="/etc/riemann/riemann.config"
+else
+ config=$1
+fi
+
+java -XX:+UseConcMarkSweepGC -jar "/usr/lib/riemann/riemann.jar" "$config"
View
275 pkg/etc/riemann.config.guide
@@ -1,275 +0,0 @@
-; vim: filetype=clojure
-
-; Hi! This is a Riemann config file. It's a Clojure program that describes how
-; to process events. I'm going to take you on a representative tour of
-; riemann monitoring. When you're done, you can write your own config--as
-; simple or complicated as necessary.
-;
-; If you aren't familiar with Riemann or its events, see
-;
-; https://github.com/aphyr/riemann
-;
-; You may also want to refer to the API docs, especially for streams. This
-; guide should give you a taste for Riemann's abilities, but I won't be
-; exhaustive.
-;
-; http://aphyr.github.com/riemann/riemann.streams.html
-;
-; All right! First, let's set up a log file. If you need to debug your
-; configuration, you can always call (debug <anything>). clojure.tools.logging
-; is included.
-
-(logging/init :file "riemann.log")
-
-; If you haven't seen Clojure before, (fun arg1 arg2 ...) is a function call.
-; :file is a keyword, like Erlang atoms or Ruby symbols. "riemann.log" is a
-; regular double-quoted string.
-
-; Servers accept events from the network. We'll start a TCP and UDP server:
-
-(tcp-server)
-(udp-server)
-
-; By default Riemann listens on port 5555 and every interface. You can change
-; these:
-
-; (tcp-server :host "12.34.56.78" :port 5555)
-
-; The index stores the most recent state for any [host, service] pair. Clients
-; can search the index for various states with a basic query language. The
-; default implementation is a NonBlockingHashMap.
-;
-; (let [name value name2 value2 ...] body) means "inside body, let these names
-; refer to these values."
-
-(let [my-index (index)
-
- ; Let's also define a stream that feeds states into this index. We'll
- ; call it index for short.
-
- index (update-index my-index)]
-
- ; Now we need to *do* something with the events that flow in. Riemann applies
- ; each event to a series of streams.
- (streams
-
- ; A stream is just a function that accepts a map. (prn x) prints x to the
- ; console. Let's print every event that passes in:
-
- prn
-
- ; Let's pay attention to one kind of message in particular.
-
- (where (state "error")
-
- ; The where stream ignores any event that doesn't match. Events that
- ; do match get passed on to its children. Let's log those error
- ; events:
-
- (fn [event] (info event))
-
- ; We just created a new function with fn, taking one argument: event.
- ; That function just logs the event to the Riemann logfile.
- )
-
- ; Where is powerful stuff. You can match using equality:
- (where (host "von braun"))
-
- ; Regular expressions
- (where (description #"an+elids"))
-
- ; The presence of a given tag
- (where (tagged "mutant"))
-
- ; Arbitrary functions on values
- (where (> (* metric 1000) 2.5))
-
- ; Which makes range queries easy
- (where (< 5 metric 10))
-
- ; Boolean operators
- (where (not (or (tagged "www")
- (and (state "ok") (nil? metric)))))
-
- ; And arbitrary functions
- (defn global? [event] (nil? (:host event)))
- (where (global? event))
-
- ; Imagine you wanted to know the time it takes for your app's API requests
- ; to complete. The API emits events like:
- ;
- ; {:service "api req"
- ; :metric: 0.240} ; 240 milliseconds
- ;
- ; So first, we select only the API requests
-
- (where (service "api req")
-
- ; Now, we'll calculate the 50th, 95th, and 99th percentile for all
- ; requests in each 5-second interval.
-
- (percentiles 5 [0.5 0.95 0.99]
-
- ; Percentiles will emit events like
- ; {:service "api req 0.5" :metric 0.12}
- ; We'll add them to the index, so they can show up
- ; on our dashboard.
-
- index)
-
- ; What else can we do with API requests? Let's figure out the total
- ; request rate. (rate interval & children) sums up metrics and
- ; divides by time.
-
- (rate 5)
-
- ; But this isn't quite right--these event metrics are *times*, so
- ; we're actually calculating the number of seconds spent by the API,
- ; each second. So we *set* the metric of every event to 1, *then*
- ; take the rate:
-
- (with :metric 1 (rate 5 index))
-
- ; (with) takes each event and calls (rate) with a *changed*
- ; copy--one where :metric is always 1. Then (rate) adds up all those
- ; 1's over five seconds, and sends that metric to the index.
-
- ; (with) has a counterpart, by the way: (default). It works exactly
- ; the same, but it only alters the event when the value is nil. Both
- ; with and default accept maps as well:
-
- (default {:state "ok" :ttl 60} index)
- )))
-
-; Imagine your web server sends an event every time it hits an exception. Your
-; app is pretty sizeable and maintained by several people, so you attach tags
-; for various parts--the model, view, controller, etc.
-;
-; {:service "web server"
-; :state "exception"
-; :description "my stacktrace"
-; :tags ["view"]}
-;
-; Let's send an email to the right team whenever an exception like this is
-; thrown. This example uses local sendmail:
-
-(def email (mailer {:from "riemann@trioptimum.com"}))
-
-; You can use any options for https://github.com/drewr/postal
-;
-; (mailer {:from "riemann@trioptimum.com"
-; :host "mx1.trioptimum.com"
-; :user "foo"
-; :pass "bar"})
-
-(streams
- (where (and (service "web server")
- (state "exception"))
- (tagged "controller"
- (email "5551234567@txt.att.net"))
- (tagged "view"
- (email "delacroix@trioptimum.com" "bronson@trioptimum.com"))
- (tagged "model"
- (email "staff@vonbraun.mil"))))
-
-; The mailer can accept lists of events, too. To avoid getting slammed with
-; too many emails we can use the rollup function--it will combine multiple
-; events into a single message. To send at most 5 emails every hour:
-
-(def tell-ops (rollup 5 3600 (email "ops@vonbraun.mil")))
-(streams
- (where (state "critical") tell-ops))
-
-; See that? We used def to define a new stream--plugging together primitives to
-; solve a specific problem. You can reuse tell-ops all over your config. If you
-; come up with a stream that lots of people could use, send me a pull request
-; and we'll make it a part of the standard release.
-
-; Rollup preserves all events, but sometimes you just want to drop excess
-; events on the floor. Let's send an email for at most 5 state changes every
-; day.
-(by [:host :service]
- (changed :state
- (throttle 5 (* 3600 24)
- (email "grumpy@devs.com"))))
-
-; You can forward to other monitoring systems too. Let's connect to graphite:
-
-(def graph (graphite {:host "be2.tx"}))
-
-; And graph the rate of web requests on each server. To do that, we'll
-; need to split up the stream into several rates, one per host.
-
-(streams
- (where (service "web req")
- (by :host
- (rate 1 graph))))
-
-; The (by) stream creates a new rate every time it sees a new host. It forwards
-; all events from web1 to one rate, all events from web2 to another, and so on.
-
-; By also comes in handy when you want to use a single where expression to
-; track many distinct things.
-
-(where (service #"^riak (gets|puts)")
- (by [:host :service] index graph))
-
-; Sometimes you'll want to combine the state of several services. For instance,
-; imagine that every server reports its current CPU use. You want to show only
-; the *maximum* cpu state on your dashboard.
-
-(where (service "cpu")
- ; Coalesce tracks the most recent event received for any given host and
- ; service (as long as it hasn't exceeded its TTL). Every time it
- ; receives an event, it forwards a list of all those events.
- (coalesce
- ; Then we bring those states together using the combine stream, and
- ; any function that takes a list of events.
- (combine folds/maximum
- index)))
-
-; You'll also find folds/minimum, mean, median, and sum. See
-; http://aphyr.github.com/riemann/riemann.folds.html
-
-; When you have *many* events, you can use multiple Riemann servers to scale
-; out. You might, for instance, run one Riemann server per datacenter, and
-; forward only state changes in each service to a master server for a birds-eye
-; view.
-
-(let [client (tcp-client :host "aggregator")]
- (by [:host :service]
- (changed :state
- (forward client))))
-
-; When services disappear or fail, their states in the index will get stale.
-; Periodically, Riemann can scan the index and delete states that have exceeded
-; their TTL. You'll receive events with the deleted statee's original :host,
-; :service, and :state "expired".
-;
-; To expire old states every ten seconds:
-
-(periodically-expire 10)
-
-; You can select expired events with where, or the expired stream:
-
-(streams
- (where (state "expired") prn)
- (expired prn))
-
-; This way, Riemann can issue an alert for a service that failed to check in
-; regularly. The default TTL is 60 seconds, but you can submit ttls with each
-; event or assign them using (with) or (default).
-
-(streams
- (default :ttl 10 index)
-
- (by [:host :service]
- (changed :state email "shodan@tau.ceti.five")))
-
-; Any service that fails to check in within every 10 seconds will be removed
-; and an alert sent.
-
-; Now you're ready to write your own streams. Check out the streams API:
-; http://aphyr.github.com/riemann/riemann.streams.html. Feel free to email me
-; as well: aphyr@aphyr.com. Issues and pull requests welcome on github:
-; https://github.com/aphyr/riemann
View
2  pkg/etc/riemann.config → pkg/riemann.config
@@ -12,7 +12,7 @@
index (default :ttl 300 (update-index (index)))]
(streams
- (with {:metric_f 1 :host nil :state "ok" :service "events/sec"}
+ (with {:metric 1 :host nil :state "ok" :service "events/sec"}
(rate 5 index))
(where (service #"^per")
View
1  pkg/bin/riemann → pkg/tar/riemann
@@ -8,5 +8,4 @@ else
config=$1
fi
-
java -XX:+UseConcMarkSweepGC -jar "$top/lib/riemann-0.1.2-standalone.jar" "$config"
View
22 project.clj
@@ -9,8 +9,7 @@
(defproject riemann "0.1.3-SNAPSHOT"
:description
-"A network event stream processor. Intended for analytics, metrics, and
-alerting; and to glue various monitoring systems together."
+"A network event stream processor. Intended for analytics, metrics, and alerting; and to glue various monitoring systems together."
:url "http://github.com/aphyr/riemann"
; :warn-on-reflection true
; :jvm-opts ["-server" "-d64" "-Xms1024m" "-Xmx1024m" "-XX:+UseParNewGC" "-XX:+UseConcMarkSweepGC" "-XX:+CMSParallelRemarkEnabled" "-XX:+AggressiveOpts" "-XX:+UseFastAccessorMethods" "-verbose:gc" "-XX:+PrintGCDetails"]
@@ -18,6 +17,7 @@ alerting; and to glue various monitoring systems together."
:repositories {
"boundary-site" "http://maven.boundary.com/artifactory/repo"
}
+ :maintainer {:email "aphyr@aphyr.com"}
:dependencies [
[org.clojure/clojure "1.4.0"]
[org.clojure/math.numeric-tower "0.0.1"]
@@ -43,9 +43,7 @@ alerting; and to glue various monitoring systems together."
[clj-librato "0.0.2-SNAPSHOT"]
[incanter/incanter-charts "1.3.0"]
]
- :profiles {:dev {:dependencies [[clj-glob "1.0.0"]]}}
- :plugins [[codox "0.6.1"]
- [lein-deb "1.0.0-SNAPSHOT"]]
+ :plugins [[codox "0.6.1"]]
:test-selectors {:default (fn [x] (not (or (:integration x)
(:time x)
(:bench x))))
@@ -61,19 +59,5 @@ alerting; and to glue various monitoring systems together."
:java-source-path "src/riemann/"
:aot [riemann.bin]
:main riemann.bin
- :deb {:maintainer {:name "Kyle Kingsbury"
- :email "aphyr@aphyr.com"}
- ; I wish I could use relative paths here, but lein-deb complains
- ; "No directory specified for tarfileset", and lein macros need them
- ; to be strings. Arrrgh.
- :filesets [{:file "/home/aphyr/riemann/riemann-0.1.3-standalone.jar"
- :fullpath "/usr/lib/riemann/riemann.jar"}
- {:file "/home/aphyr/riemann/riemann.config"
- :fullpath "/etc/riemann/riemann.config"}
- {:file "/home/aphyr/riemann/bin/riemann"
- :fullpath "/usr/bin/riemann"
- :fileMode "755"}]
- :depends ""}
- :deb-skip-jar true
:codox {:output-dir "site/api"}
)
View
3  src/leiningen/package.clj
@@ -5,8 +5,7 @@
[leiningen.deb :only [deb]]
[leiningen.pom :only [pom]]
[clojure.java.io :only [file]]
- [org.satta.glob :only [glob]]
- ))
+ [org.satta.glob :only [glob]]))
(defn package [project]
; Clojars pom
View
97 tasks/leiningen/fatdeb.clj
@@ -0,0 +1,97 @@
+(ns leiningen.fatdeb
+ (:use [clojure.java.shell :only [sh]]
+ [clojure.java.io :only [file delete-file writer copy]]
+ [clojure.string :only [join capitalize]]
+ [leiningen.uberjar :only [uberjar]]))
+
+(defn delete-file-recursively
+ "Delete file f. If it's a directory, recursively delete all its contents.
+ Raise an exception if any deletion fails unless silently is true."
+ [f & [silently]]
+ (System/gc) ; This sometimes helps release files for deletion on windows.
+ (let [f (file f)]
+ (if (.isDirectory f)
+ (doseq [child (.listFiles f)]
+ (delete-file-recursively child silently)))
+ (delete-file f silently)))
+
+(defn deb-dir
+ "Debian package working directory."
+ [project]
+ (file (:root project) "target/deb/riemann"))
+
+(defn cleanup
+ [project]
+ ; Delete working dir.
+ (when (.exists (deb-dir project))
+ (delete-file-recursively (deb-dir project))))
+
+(defn reset
+ [project]
+ (cleanup project)
+ (sh "rm" (str (:root project) "/target/*.deb")))
+
+(defn control
+ "Control file"
+ [project]
+ (join "\n"
+ (map (fn [[k v]] (str (capitalize (name k)) ": " v))
+ {:package (:name project)
+ :version (:version project)
+ :section "base"
+ :priority "optional"
+ :architecture "all"
+ :depends "bash"
+ :maintainer (:email (:maintainer project))
+ :description (:description project)})))
+
+(defn write
+ "Write string to file, plus newline"
+ [file string]
+ (with-open [w (writer file)]
+ (.write w (str string "\n"))))
+
+(defn make-deb-dir
+ "Creates the debian package structure in a new directory."
+ [project]
+ (let [dir (deb-dir project)]
+ (.mkdirs dir)
+
+ ; Meta
+ (.mkdirs (file dir "DEBIAN"))
+ (write (file dir "DEBIAN" "control") (control project))
+ (write (file dir "DEBIAN" "conffiles")
+ (join "\n" ["/etc/riemann/riemann.config"]))
+
+ ; Jar
+ (.mkdirs (file dir "usr" "lib" "riemann"))
+ (copy (file (:root project) "target"
+ (str "riemann-" (:version project) "-standalone.jar"))
+ (file dir "usr" "lib" "riemann" "riemann.jar"))
+
+ ; Binary
+ (.mkdirs (file dir "usr" "bin"))
+ (copy (file (:root project) "pkg" "deb" "riemann")
+ (file dir "usr" "bin" "riemann"))
+ (.setExecutable (file dir "usr" "bin" "riemann") true false)
+
+ ; Config
+ (.mkdirs (file dir "etc" "riemann"))
+ (copy (file (:root project) "pkg" "riemann.config")
+ (file dir "etc" "riemann" "riemann.config"))
+
+ dir))
+
+(defn dpkg
+ "Convert given package directory to a .deb."
+ [project deb-dir]
+ (print (:err (sh "dpkg" "--build"
+ (str deb-dir)
+ (str (file (:root project) "target"))))))
+
+(defn fatdeb [project]
+ (doto project
+ (reset)
+ (uberjar)
+ (dpkg (make-deb-dir project))
+ (cleanup)))
Please sign in to comment.
Something went wrong with that request. Please try again.