Missing client keys

[willscripted]

My two PRs skirt the root of the issue so I feel like I should log this issue So far I can't find a simple test case to reproduce it though. Here's the problem:

Background

We're using sensu to run checks and report events for ~ 600 servers. We have a handler to check keepalive events for servers that no longer exist. When appropriate, the handler makes a call to the API to delete the client.

Problem - Missing redis keys

Client keys are disappearing in redis. Eg. client:8eef4f7a-45f7-11e2-b603-12313d23e203 is nil when it should contain client data. This causes all sorts of problems down the pipe - anything that iterates through the 'clients' set and parses the JSON value expected for the keys indicated.

Symptoms

• Broken /clients resource in the API and all reliant services [https://github.com/ [API] "GET /clients" should not throw 500 if a client key is invalid #430]
• Remnant data
  history, events, etc. with that client's key that should have been, but has not been, deleted by the API call [https://github.com/[API] make DELETE "/clients/#{id}" idempotent #431]

Those PRs are just fixing symptoms. Have not found a good solution for ensuring the db's integrity.
I believe the API and the server are the only ones to touch redis - is there anywhere else i should be looking?

[portertech]

Is this an issue with concurrency? eg. concurrent DELETE /client/8eef4f7a-45f7-11e2-b603-12313d23e203 and one call is getting nil and attempting to parse it as JSON? Yes, the server/s & API/s are the only components that talk to Redis.

[willscripted]

Unless a failure in one causes the processing in the 2nd to abort, I don't see how concurrency could create this behavior. I suppose messing up the redis connection might do that?

[willscripted]

Found it - flood of keepalive messages off the queue caused em-redis callbacks to execute much, much later. Eg. a client would be added to the clients set 90 minutes after the client key was set. If the client was deleted in that 90 minutes, the key would exist in the set but the client:#{id} would have been removed. This extra key would cause everything else to barf

This was fixed by cf0b006 or whenever ack'ing was first added. Then keepalives wouldn't be crowding in memory, waiting to get processed

[portertech]

@will-ob 0.9.9 includes many improvements, I recommend upgrading.

[willscripted]

I expect/hope we will in the next two week =]
thanks