Permalink
Browse files

update readme and code to reflect web extraction

  • Loading branch information...
mmcgrana committed May 5, 2012
1 parent ad573cc commit 0b55543cdf537231091ad069128e0bffc513ad43
Showing with 9 additions and 46 deletions.
  1. +8 −27 Readme.md
  2. BIN doc/architecture.png
  3. +0 −9 src/pulse/conf.clj
  4. +1 −10 src/pulse/web.clj
View
@@ -1,28 +1,24 @@
# Pulse
-Real-time Heroku operations dashboard, powered by the Heroku event stream.
+Real-time, streaming, distributed event -> metric computation.
## Overview
-Pulse provides web-accessible, real-time metrics for the [Heroku platform](http://www.heroku.com/) kernel. Heroku engineers use Pulse internally to maintain ambient awareness of Heroku's distributed infrastructure and to quickly diagnose platform issues.
+Pulse provides near-real-time, streaming, distributed event-to-metric compuation for the [Heroku platform](http://www.heroku.com/) kernel. Heroku engineers use Pulse to maintain ambient awareness of Heroku's distributed infrastructure and to quickly diagnose platform issues.
-Pulse works by consuming, processing, and performing statistical calculations against the platform's unified [event log stream](http://adam.heroku.com/past/2011/4/1/logs_are_streams_not_files/). All Heroku components log extensively and in an easily machine-parsable format; by processing these logs, we can build various platform metrics, analytics, monitoring, auditing, and debugging tools orthogonally to the critical-path components themselves. Pulse is one such tool:
-
-![pulse architecture](http://s3.amazonaws.com/pulse-doc/architecture.png)
+Pulse works by consuming, processing, and performing statistical calculations against the platform's unified [event log stream](http://adam.heroku.com/past/2011/4/1/logs_are_streams_not_files/). All Heroku components log extensively and in an easily machine-parsable format; by processing these logs, we can build various platform metrics, analytics, monitoring, auditing, and debugging tools orthogonally to the critical-path components themselves. Pulse is one such tool.
The log messages that ultimately power Pulse originate from hosts across the Heroku platform. These messages are sent to a load-balanced pool of log forwarders. The log forwarders in turn distribute messages to an internal [Splunk](http://www.splunk.com/) cluster for archival and long-term analytics, and also host "Aorta" servers which provide Pulse access to the log stream.
-Pulse itself is a distributed application implemented with three [process types](http://devcenter.heroku.com/articles/process-model): `receiver`, `merger`, and `web`. Processes of the `receiver` type are responsible for physically consuming a load-balanced fraction of the platform event stream, parsing each text-encoded event into a data structure, and streaming that data through the initial phase of the Pulse statistical pipeline. The `receiver` processes periodically broadcast "stat fragments" corresponding to partial roll-ups of the various stats that Pulse tracks, for consumption by the `merger` process. To ensure that the production Pulse deployment can process the entire Heroku log stream at peak platform load, this process type is horizontally scaled.
-
-The `merger` is a singleton processes responsible for providing a unified statistical view for the Pulse deployment. It continuously combines stat fragments as they arrive from the `receiver` processes and periodically emits the resulting "stat snapshots" for consumption by front-end clients.
+Pulse itself is a distributed application implemented with two [process types](http://devcenter.heroku.com/articles/process-model): `receiver` and `merger`. Processes of the `receiver` type are responsible for physically consuming a load-balanced fraction of the platform event stream, parsing each text-encoded event into a data structure, and streaming that data through the initial phase of the Pulse statistical pipeline. The `receiver` processes periodically broadcast "stat fragments" corresponding to partial roll-ups of the various stats that Pulse tracks, for consumption by the `merger` processes. To ensure that the production Pulse deployment can process the entire Heroku log stream at peak platform load, this process type is highly horizontally scaled.
-The key consumers of the stats snapshot stream are processes of the `web` type. These processes consume and buffer a short history of these stat snapshots, and dump their buffer in response to web requests. Web clients use this data to display corresponding sparkline graphs.
+The `merger` is a smaller set of sharded singleton processes responsible for providing a unified statistical view for the Pulse deployment. These processes continuously combine stat fragments as they arrive from the `receiver` processes and periodically emits the resulting "stat snapshots" as JSON into the given `METRICS_URLS`.
Pulse is written in Clojure and deployed to Heroku itself using the platform's native Clojure support.
-## Local deploy
+## Local Deploy
Ensure that Aorta is running at `AORTA_URL`, then:
@@ -35,7 +31,7 @@ $ foreman start
```
-## Heroku platform deploy
+## Heroku Platform Deploy
Ensure that Aortas are running and dyno-reachable at `AORTA_URLS`, and that the username in the `AORTA_URLS` are scoped to this particular deployment of Pulse. Then:
@@ -47,30 +43,15 @@ $ heroku addons:add redistogo:large -r $DEPLOY
$ heroku config:add BUILDPACK_URL=https://github.com/heroku/heroku-buildpack-clojure.git -r $DEPLOY
$ heroku config:add DEPLOY=$DEPLOY -r $DEPLOY
$ heroku config:add CLOUD=heroku.com -r $DEPLOY
-$ heroku config:add SESSION_SECRET=$(openssl rand -hex 16) -r $DEPLOY
$ heroku config:add FORCE_HTTPS=true -r $DEPLOY
$ heroku config:add GRAPHITE_PERIOD=21600 -r $DEPLOY
$ heroku config:add PUBLISH_THREADS=2 -r $DEPLOY
-$ heroku config:add API_URL=... -r $DEPLOY
$ heroku config:add GRAPHITE_URL=... -r $DEPLOY
-$ heroku config:add PROXY_URL=... -r $DEPLOY
-$ heroku config:add PROXY_SECRET=... -r $DEPLOY
$ heroku config:add SCALES_URL=... -r $DEPLOY
-$ heroku config:add CANONICAL_HOST=... -r $DEPLOY
$ heroku config:add REDIS_URL=... -r $DEPLOY
$ heroku config:add AORTA_URLS=... -r $DEPLOY
$ heroku config:add METRICS_URLS=... -r $DEPLOY
$ git push $DEPLOY master
-$ heroku scale receiver=60 merger0=1 merger1=1 merger2=1 merger3=1 merger4=1 web=5 emitter=1 -r $DEPLOY
+$ heroku scale receiver=60 merger0=1 merger1=1 merger2=1 merger3=1 merger4=1 emitter=1 -r $DEPLOY
```
-
-
-## See also
-
-* [Nimrod](https://github.com/sbtourist/nimrod): Non-invasive, log-based metrics server in Clojure
-* [Graylog2](http://graylog2.org/): Log management and searching
-* [Splunk](http://www.splunk.com/): Log visibility software
-* [Loggly](http://www.loggly.com/): Logging as a service
-* [Papertrail](https://papertrailapp.com): Hosted cloud log management
-* [Leech web](https://github.com/heroku/leech-web) and [Leech drain](https://github.com/heroku/leech-web): Log tailing and filtering
View
Binary file not shown.
View
@@ -11,12 +11,6 @@
(defn port [] (Integer/parseInt (env! "PORT")))
(defn redis-url [] (env! "REDIS_URL"))
(defn aorta-urls [] (str/split (env! "AORTA_URLS") #","))
-(defn session-secret [] (env! "SESSION_SECRET"))
-(defn proxy-url [] (env! "PROXY_URL"))
-(defn proxy-secret [] (env! "PROXY_SECRET"))
-(defn force-https? [] (boolean (env "FORCE_HTTPS")))
-(defn scales-url [] (env! "SCALES_URL"))
-(defn api-url [] (env! "API_URL"))
(defn metrics-urls [] (str/split (env! "METRICS_URLS") #","))
(defn deploy [] (env! "DEPLOY"))
(defn cloud [] (env! "CLOUD"))
@@ -25,6 +19,3 @@
(defn graphite-period [] (env! "GRAPHITE_PERIOD"))
(defn publish-threads [] (Integer/parseInt (env! "PUBLISH_THREADS")))
(defn merger-count [] 5)
-
-(defn api-password []
- (second (str/split (:auth (util/url-parse (api-url))) #":")))
View
@@ -12,21 +12,13 @@
(fn [req]
(if (= (:server-name req) canonical-host)
(handler req)
- (redirect (format "%s://%s/"
- (if (conf/force-https?) "https" "http")
- canonical-host)))))
+ (redirect (format "https://%s/" canonical-host)))))
(defn core-app [{:keys [uri] :as req}]
{:status 404
:headers {"Content-Type" "text/plain"}
:body "Pulse no longer provides a web interface."})
-(defn wrap-force-https [handler]
- (fn [{:keys [headers server-name uri] :as req}]
- (if (and (conf/force-https?) (not= (get headers "x-forwarded-proto") "https"))
- (redirect (format "https://%s%s" server-name uri))
- (handler req))))
-
(defn wrap-logging [handler]
(fn [{:keys [request-method uri] :as req}]
(let [method (name request-method)
@@ -47,7 +39,6 @@
(defn app []
(-> core-app
(wrap-canonical-host (conf/canonical-host))
- (wrap-force-https)
(wrap-logging)))
(defn -main []

0 comments on commit 0b55543

Please sign in to comment.