reboot(wait) operation crashes

[bitprophet]

Description

Trying to use the reboot() operation. The reboot occurs, but then the script errors out. No waiting or continuing.

The reboot(180) call is the very first thing in my function, so I don't suspect anything else is causing the issue.

My environment:

Mac OS X, Python 2.7, Fabric 0.9.3

output from my last try:

[encrypt1] Executing task 'reboot_encrypted_server'
[encrypt1] sudo: reboot
Traceback (most recent call last):
File "/usr/local/Cellar/python/2.7/lib/python2.7/site-packages/Fabric-0.9.3-py2.7.egg/fabric/main.py", line 540, in main
commands[name](*args, **kwargs)
File "/Users/jgerry/pdx/svn/trunk/jkg/fab/fabfile.py", line 71, in reboot_encrypted_server
reboot(180) 
File "/usr/local/Cellar/python/2.7/lib/python2.7/site-packages/Fabric-0.9.3-py2.7.egg/fabric/operations.py", line 662, in reboot
del connections[env.host_string]
KeyError: 'encrypt1'
Disconnecting from encrypt1... done.

---

Originally submitted by Jason Gerry (jgerry) on 2010-12-02 at 02:48pm EST

---

Closed as Done on 2011-03-23 at 03:49pm EDT

[bitprophet]

Jeff Forcier (bitprophet) posted:

---

Updated metadata, cleaned up description so the output is in a code block.

---

on 2010-12-08 at 01:17pm EST

[bitprophet]

Jeff Forcier (bitprophet) posted:

---

Nothing springs to mind for a cause of your problem, but hopefully it's something simple. If it's an innocent order-of-operations problem where your connections dict is being cleared out beforehand, then we can just update the del logic so it's more graceful. However, it's possible something else is going on w/r/t env.host_string or the connections key values, which I'd want to clear up in case it's more serious.

For a first step, if you could throw some debug print statements in there to introspect the key/values of fabric.state.connections immediately prior to calling reboot, that might help shed some light on things. I.e. is that dict empty, or what?

Otherwise, if you're able to provide your fabfile or a snippet thereof, I can try to replicate. Don't have easy access to Python 2.7 on OS X but I've been meaning to work out a multi-Python setup for a while, may be a good excuse. Are you on OS X 10.5 or 10.6?

---

on 2010-12-08 at 01:22pm EST

[bitprophet]

Jason Gerry (jgerry) posted:

---

Happy to help out any way I can. I'm actually on vacation until Monday, I'm trying not to work (too much at least) but when I get back I'll poke around at it some more and get the debugging info.

I've been doing my development on Python 2.7 / OS X 10.6, but I'm moving this project to a CentOS 5.5 environment for actual use first thing next week. I'm curious as to whether I'll be able to reproduce the issue in a Linux environment.

---

on 2010-12-08 at 01:32pm EST

[bitprophet]

Jeff Forcier (bitprophet) posted:

---

Cool, no rush obviously :) I'm going to be busy myself so I may not even have time to review anything until next week.

Definitely interested in whether it is reproducible on other platforms or not, that's always helpful.

---

on 2010-12-08 at 06:30pm EST

[bitprophet]

Jason Gerry (jgerry) posted:

---

Migrated to my CentOS 5.5 env today, the error persists. I'll try to get some more debug info to you this week when I have time.

---

on 2010-12-14 at 11:06am EST

[bitprophet]

Jason Gerry (jgerry) posted:

---

From my fabfile, I've commented out the reboot() function and added some debug:

def reboot_encrypted_server(acct,region,instance_id):
    import fabric.state
    print 'Host string: ' + env.host_string
    #for foo in dir(fabric.state.connections):
    print fabric.state.connections.items()
    #reboot(180) 

Output:

$ fab -H encrypt1 reboot_encrypted_server:acct=pdx_ops,instance_id=i-xxxxxxxx
[encrypt1] Executing task 'reboot_encrypted_server'
Host string: encrypt1
[]
Done.

So yes, it looks like that dict is empty.

---

on 2010-12-17 at 02:32pm EST

[bitprophet]

Jeff Forcier (bitprophet) posted:

---

Definitely odd, but hopefully not indicative of something bigger. Will try to slap a patch in soon. Should be as simple as wrapping the del in an if.

---

on 2010-12-29 at 02:57pm EST

[bitprophet]

Jeff Forcier (bitprophet) posted:

---

Applied in changeset commit:710d4ae9790deb48ae0067d352fca69695a29504.

---

on 2011-02-27 at 12:05am EST

[bitprophet]

Miquel Torres (tobami) posted:

---

I can confirm this bug on Fabric 1.0.0. I see the fix is for the 0.9.x branch, have you patched 1.0.x as well?

---

on 2011-03-23 at 06:09am EDT

[bitprophet]

Jeff Forcier (bitprophet) posted:

---

Should have, yes. From now on, higher release numbers will include lower releases' bugfixes unless said bugfixes no longer apply for some reason. In this case, reboot hasn't changed any from 0.9 to 1.0 so that fix should have been merged or cherry-picked into the 1.0 release branch (and master of course).

---

on 2011-03-23 at 03:49pm EDT