Statsd logger #3

wants to merge 9 commits into


None yet

2 participants

mrflip commented Jun 24, 2012

A statsd agent and plugin, and a middleware to log all response times and durations to statsd

Philip (flip... added some commits Jun 23, 2012
Philip (flip) Kromer Made the support files (gemspec, Gemfile, etc) look like goliath's) 4280923
Philip (flip) Kromer Middlewares to do many useful forms of diagnosis and fault injection:
* force_delay       does a 'sleep for x +- y' seconds before responding
* force_drop        drops the connection, before or after its middleware descendents run
* force_fault       raises the requested type of error (eg _fault=404 will bail out of the rpocessing chain, returning a 404 Not Found response
* force_response    forcibly replaces the status, headers or body with what's given in the URL
* handle_exceptions gives you response methods, same that the middleware provides, for your app
* diagnostics recycles info about the request into the output
Philip (flip) Kromer A statsd agent and plugin, and a middleware to log all response times…
… and durations ot statsd
Philip (flip) Kromer middleware to load white-listed params into the env; another to load …
…static values into the env
Philip (flip) Kromer reworked injection middlewares to be configured by env, not params (b…
…ack to how it ics api) so that it's actually useful outside test rig
Philip (flip) Kromer minor tweaks to utility middlewares while landing the last pieces 98766c2
@igrigorik igrigorik commented on the diff Jul 5, 2012
+ #
+ # @example wait one second then drop the connection with no response (note: `_drop_after`, not `_drop`)
+ # curl 'http://localhost:9000/?_drop_after=true&_delay=1'
+ #
+ class ForceDrop
+ include Goliath::Rack::AsyncMiddleware
+ def call(env)
+ return super unless env[:force_drop].to_s == 'true'
+ "Forcing dropped connection"
+ env.stream_close
+ [0, {}, {}]
+ end
+ def post_process(env, status, headers, body)
igrigorik Jul 5, 2012 Owner

If I'm interepreting this correctly.. this would still execute the action for N seconds, but if N > defined value, then connection is closed once the response is ready? Shouldn't we return on defined value instead with hard cutoff?

mrflip Jul 6, 2012 Member

I can't tell exactly but I think my "clarifying" example (of delay+drop) has given the false impression this class implements a delay itself.

The ForceDrop middleware drops connection either:

  1. immediately in the trip down the middleware chain (if :force_drop declared) -- the request does not get to anything past it
  2. Immediately in the trip up the middleware chain (if `:force_drop_after) -- in this case, the app and all middlewares below this one have finished.

This middleware doesn't itself implement a delay (in the example test rig, the ForceDelay middleware does). I wanted an example of where :force_drop_after made sense -- here it's simulating an endpoint that would sit for N seconds doing nothing and then hang up cold. An example where it's injected into an app might be to simulate a successful write to the DB but a failure in follow-on handling / client acknowledgement.

Does that make more sense? Or have I answered a totally different question than yours?

mrflip commented Jul 6, 2012

By the way -- the ForceDelay makes it wait at least N seconds, the ForceTimeout makes it respond in at most N seconds. Right now to get it to return in as-much-as-possible exactly N seconds you would stack a ForceTimeout of N seconds in front of a ForceDelay of N+1 seconds.


Ok, perhaps a broader question then is: what's the point of "ForceDrop"? I understand ForceDelay (although seems like a limited use case), ForceTimeout for guaranteeing a cutoff also makes sense..

Philip (flip... added some commits Jul 6, 2012
Philip (flip) Kromer force timeout calls its response chain rather than raising an exception.
 it now formulates the response in the timeout condition directly, and calls its middleware predecessor with that error reposnse. This was always the intended behavior, but a too-helpful rescue block was making it skip the remaining middleware chain.
Philip (flip) Kromer more clearer the documentation was made to be e328f50
mrflip commented Jul 6, 2012

ForceDrop is more like ForceFault: they are both ways to inject a non-standard response. ForceFault raises an exception that is handled; your client will get a valide response with that error. ForceDrop slams the connection shut before any useful response is sent back.

The use case would be to layer them onto a special deploy of a backend service in order to see how the other apps that depend on it react. In our case, we're bringing online a CI environment that deploys the full stack to virtualbox VMs and runs integration tests against its components' outward contract. The stack has dozens of moving parts, many of them mediated by HTTP. Modules like these let us test resiliency of the stack by injecting faults: if it's our app, by conditionally include these middleware; if it's a DB or other app, by inserting a transparent proxy.

Of the injection middlewares, only Timeout (broadly) and Delay (in limited cases) might be directly useful in the actual production app itself. Force-injecting a Fault, Drop, body or headers is mainly something you would want to do in a integration-test context.

mrflip commented Jul 6, 2012

by the way, if some of these seem too idiosyncratic I'm glad to rework the pull request to exclude them. I had been waiting to see what other people would push in to goliath/contrib so I would know what kind of stuff went here, but in the absence of that I figured "meh, I'll chuck all the possibly-interesting stuff up there and trust that Ilya or dj2's swift sword will show the line".

Philip (flip) Kromer Rakefile does yard & tests too. Dependency list more minimalist.
The rakefile is copied from goliath; it now has tasks for running tests, making yardoc, etc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment