Permalink
Switch branches/tags
Commits on Jan 25, 2014
Commits on Jan 24, 2014
  1. Add fuzzing to the max-ttl values for closing active drains.

    This should let us spread out the distribution of reconnecting active
    drains, which is currently very clumpy. Because the randomization
    happens every time the close_timers triggers (~20x per hour) few
    drains will exceed the mean value, which means we need a large fuzzing
    range.
    
    Simulation of reconnect times: http://p.hagelb.org/drain-sim.clj.html
    technomancy committed Jan 24, 2014
Commits on Jan 22, 2014
  1. Merge branch 'close-active-drains'

    # Please enter a commit message to explain why this merge is necessary,
    # especially if it merges an updated upstream into a topic branch.
    #
    # Lines starting with '#' will be ignored, and an empty message aborts
    # the commit.
    technomancy committed Jan 22, 2014
Commits on Jan 21, 2014
  1. Remove unused line.

    technomancy committed Jan 21, 2014
Commits on Jan 20, 2014
Commits on Jan 18, 2014
  1. Simply restart close timer if it triggers in sending.

    This should be more graceful, but it means that sometimes we'll have
    active drains that last longer than the max TTL by a several
    increments of the idle timer. I don't think this is a big problem
    since the max TTL is a "soft maximum" intended to improve the load
    balancing properties rather than something that needs strict adherence.
    technomancy committed Jan 18, 2014
  2. Take care of idle_timeout messages from old revisions.

    This will let us seamlessly upgrade without needing to cancel a bunch
    of timers and restart them with the new message name.
    technomancy committed Jan 18, 2014
Commits on Jan 17, 2014
Commits on Jan 16, 2014
  1. Rename idle_timer -> close_timer.

    This timer is now responsible for closing for multiple reasons, not just idling.
    technomancy committed Jan 16, 2014
Commits on Jan 14, 2014
  1. Don't update last_good_time in close_if_idle.

    Turns out last_good_time actually doesn't get reset in between
    disconnect/reconnect cycles; we need to preserve it.
    
    See af34407#commitcomment-5061990
    technomancy committed Jan 14, 2014
  2. Close even active drains after they've been around too long.

    This will avoid bad behaviour around load balancers where more
    reliable nodes tend to accumulate too many connections over time as
    the other ones drop out.
    
    We do this by piggybacking on the existing timer we're using for idle
    timeouts, so we should probably rename that timer.
    
    The original design also called for randomizing the total timeout, but
    this implementation just piggybacks on the fuzzing already present in
    the idle timer. This may not be sufficient fuzzing; there's a check
    every fuzzed five minutes to see whether it's gone over the 5 hour
    limit, but it would probably better if the 5 hour limit itself were
    fuzzed.
    technomancy committed Jan 14, 2014
Commits on Jan 13, 2014
  1. Re-use existing timestamp.

    technomancy committed Jan 13, 2014
  2. We'll need this for the upcoming change to add an upper bound on drai…

    …n lifetime.
    
    Since we're doing a cluster replacement anyway for closing idle
    drains, we might as well get this record change in while we're at it
    so the next upgrade can be a live one.
    technomancy committed Jan 13, 2014
Commits on Jan 10, 2014
  1. Merge branch 'close-idle-http-drains'

    # Please enter a commit message to explain why this merge is necessary,
    # especially if it merges an updated upstream into a topic branch.
    #
    # Lines starting with '#' will be ignored, and an empty message aborts
    # the commit.
    technomancy committed Jan 10, 2014