Actually remove users from RabbitMQ after expiration

[Hexcles]

At the moment, it seems like we never really delete expired users from RabbitMQ, but only in the Azure table. We should do this around here: https://github.com/taskcluster/taskcluster-pulse/blob/master/src/data.js#L49

Meanwhile, it is worthwhile to experiment what would happen to an established connection when the user is deleted.

[djmitche]

Mostly done in #74, but @Hexcles raises a good point about what happens when a password changes.

[djmitche]

StatusCodeError: 400 - "{\"error\":\"bad_request\",\"reason\":\"405 RESOURCE_LOCKED - cannot obtain exclusive access to locked queue 'queue/taskcluster-garbage-foo/exclusive/DJMury05Q5evTjK2GsGuHQ' in vhost '/'\"}"

I suspect we will need to enumerate connections and delete them.

[djmitche]

In #76 we discovered a race condition in rabbtimq that would allow connections to persist forever. The solution is, unfortunately, really complex:

We manage rmq users in their own table, tracking whether they are currently active or being deleted. When deactivating a user, we delete the user and then write a deactivation time to the table, and terminate any active connections, queues, exchanges. A scheduled task then consults this table and, if at least 10 minutes have passed, again terminates any active connections, queues, and exchanges, then deletes the row from the table.

Then rotation (including claimNamespace({forceRotation: true})) entails creating a brand new user that does not already exist in the users table and scheduling the current user for deactivation at the next rotation interval. We can just append -N to the namespace name for N=[1, ..] until we find an unused username.

Jonas mentioned in response that when rotating users we definitely do not want to delete queues and exchanges, so most likely namespace expiration (where we do want to delete such things) would need to wait until all users have been deleted.

[djmitche]

Per https://bugzilla.mozilla.org/show_bug.cgi?id=1335954 we have a different solution, basically leaving all the disconnecting and deleting to the monitor.

[djmitche]

So, I think this is done.