Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Electing a 0.8.x leader during an upgrade can cause a panic in older servers #2889
Haven't seen this in the wild, but this code could possibly cause older Consul servers to panic if a 0.8.x server gains leadership during an upgrade:
The panic would occur when an older server gets a Raft log entry for the autopilot config, which it won't understand. To avoid this until fixed, make sure to upgrade the followers before updating the current leader.
This is probably pretty rare since most folks upgrade the leader last to avoid unnecessary elections, but the consequences are high enough to make it worth avoiding. We could have the autopilot loop skip out if not all servers are at least at the right version and have it create the config, so that way it's created quickly once all the servers are upgraded, even if there's not a leader transition.
@slackpad, I am not sure how rare it is for people in the AWS / CloudFormation world who use immutable AMIs and autoscaling group update policies. If I use CloudFormation to push out a new AMI that has the new version of Consul then the order in which the existing EC2 instances are replaced is non-deterministic.