fallback a manual or timeout-based terminate to stop if terminate fails #33

Merged
merged 2 commits into from Feb 8, 2013

Conversation

Projects
None yet
3 participants
Contributor

zzzeek commented Feb 7, 2013

re-send of pull req #27

I'm observing that in some cases, Amazon EC2 will force-provision one of my instances that has termination protection applied to it as one of Jenkins new nodes. This leads to the undesirable state that the automated "termination" of this instance, as well as a manual attempt to terminate this instance, produces the exception "AWS Error Code: OperationNotPermitted, AWS Error Message: The instance 'i-5d694121' may not be terminated. Modify its 'disableApiTermination' instance attribute and try again."

As any failure like this regarding EC2 essentially means monetary charges to the owner's amazon account, it is critical that in the event of a failure like this, the instance is at least stopped; else automated builds start up instances which stay running and charging money indefinitely until someone manually intervenes.

This pull request reworks the terminate() method to fall back to a new stop() method in the event of any AmazonClientException. The implementation for stop() which was factored out of the idleTimeout() method, which now has a simple conditional of calling terminate() or stop() based on the value of stopOnTerminate. As is the case before, I'm running this on jenkins.sqlalchemy.org and if I encounter any issues with it, I'll update the pullreq over here.

zzzeek added some commits Nov 17, 2012

@zzzeek zzzeek - rework terminate() and idleTimeout() so that the Amazon instruction…
…s are in indepdendent

methods stop() and terminateInstance()
- terminate() falls back to stop() if the termination command fails; this to ensure that the EC2
instance is stopped in case the instance has termination protection on it, as it is observed
that EC2 will provision an existing stopped/termination-protected instance in some cases
8628f0b
@zzzeek zzzeek Merge branch 'terminate_fallback'
Conflicts:
	src/main/java/hudson/plugins/ec2/EC2Slave.java
9101f7e

Jenkins » ec2-plugin #46 SUCCESS
This pull request looks good
(what's this?)

Member

francisu commented Feb 8, 2013

Thank you for your contribution.

@francisu francisu added a commit that referenced this pull request Feb 8, 2013

@francisu francisu Merge pull request #33 from zzzeek/master
fallback a manual or timeout-based terminate to stop if terminate fails
f0d1e98

@francisu francisu merged commit f0d1e98 into jenkinsci:master Feb 8, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment