Consul election causing app instance failures

[saliceti]

This is related to an issue we already raised: #177.

The cells maintain a presence via a lock in consul. If they loose the presence, the app instances on the cell are killed and redeployed to other cells.

The cell may loose the lock when there is no consul leader. It may happen when the consul leader VM is updated. Normally the election happens in a few seconds.
The workaround to solve the issue above was to add a 5s timeout to the cell to cover the election case. However, we have observed that sometime the election takes 5-6 seconds, which is enough to trigger the whole killing process again.

We could increase the timeout again to cover that. But maybe we should think of a more permanent solution. For example, if the instances on the missing cell were killed after being redeployed to other cells, we wouldn't see those errors.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/136610329

The labels on this github issue will be updated when the story is started.

[emalm]

Thanks for the report, @saliceti. In order for us to understand the full extent of the behavior you describe, can you provide logs from the BBS and the cell reps that illustrate the exact timing of these events during the consul unavailability and subsequent recovery? If the cell presences are gone, the BBS by definition can't actively kill instances (as it can't find their hosts), but if replacement instances are being scheduled and claimed by cells as they re-establish their cell presences, that could produce significant instance churn and potentially downtime as the system reconciles itself. The re-scheduling process itself may also result in the routes for these instances being unregistered, as networking information will be removed from the BBS's ActualLRP records, even if the instances themselves remain running.

Thanks,
Eric

[emalm]

Closing due to inactivity. If you see this behavior again and can provide adequate logs, we'll look into it. We're also considering other refinements to handling suspect cell registrations, but nothing definite scheduled in the Diego tracker yet.

Thanks,
Eric

[saliceti]

Sorry I missed your message, and now I don't have access to these logs anymore. The important bits were the same as in #177
If the error happens again, I will gather more logs.