Issue with buffering input with sudo

[bitprophet]

Description

After updating to 1.0.0 I am having trouble with a task that uses sudo. The command looks something like this:

def set_permissions():
    require('path', 'project_name', 'version', provided_by=environment_list)
    with cd(env.path):
        sudo('chown -R :mygroup releases/next' % env)

Once run I get something like this:

$ fab dev set_permissions
[XX.XX.XX.XX] Executing task 'set_permissions'
[XX.XX.XX.XX] sudo: chown -R :mygroup releases/next
[XX.XX.XX.XX] out: sudo password: 
[XX.XX.XX.XX] out: 
[XX.XX.XX.XX] out: Sorry, try again.
[XX.XX.XX.XX] out: sudo password: 

If I change the sudo command to sudo('chown -R :mygroup releases/next' % env, pty=False) And then type "123456789012345678901234567890" for a password I get this output:

$ fab dev set_permissions
[XX.XX.XX.XX] Executing task 'set_permissions'
[XX.XX.XX.XX] sudo: chown -R :mygroup releases/next
[XX.XX.XX.XX] out: sudo password: 342356057

My understanding is that this should echo every key I press back to me locally and it looks like it is losing many of them. I am on OS X connecting to a Rackspace ubuntu linux server and have:

Fabric==1.0.0

paramiko==1.7.6

---

Originally submitted by Paul Oswald (poswald) on 2011-03-17 at 09:12pm EDT

Relations

• Duplicated by sudo password input sometimes doesn't work #359: sudo password input sometimes doesn't work
• Related to Heavy CPU usage due to new interactivity behavior #312: Heavy CPU usage due to new interactivity behavior
• Related to Intermittent output freezing #352: Intermittent output freezing

---

Closed as Done on 2011-06-24 at 08:35pm EDT

[bitprophet]

Jeff Forcier (bitprophet) posted:

---

Thanks for the report, you or somebody else reported a similar problem (but without any specifics) earlier so it is good to have details. Few more questions:

• What version of Python are you on?
• Is this completely reproducible (i.e. it always drops the first N characters of your password) or is it somewhat random as to how much it drops? Is it always the beginning of the string or does it also sometimes drop other parts (end, middle)?
• Does this only happen with the sudo password prompt (as opposed to other output, the login prompt if any, etc)?
• Does your password have any special characters in it (i.e. anything not A-Z/0-9)?

---

on 2011-03-17 at 10:09pm EDT

[bitprophet]

Paul Oswald (poswald) posted:

---

Python:

$ python --version
Python 2.6.1

Seems random. It will accept my password every now and then. Sudo doesn't normally echo your chars back so I don't know really what is dropping. That's why I tried with pty=False. Consecutive runs will echo different characters. When typing "1234567890" on four consecutive runs with pty=False (various speeds of typing, ctrl c between each run) I got: 6, 340, 38, and 167 echoed back to me. Each character seems to have about a 1 in 5 chance of being echoed. The speed at which I type doesn't seem to matter.

I don't have any other code that prompts me so I'm not really sure about that point. I can look into testing it. What should I try?

It also fails for passwords without special chars in it. That doesn't seem to be the problem.

---

on 2011-03-17 at 10:26pm EDT

[bitprophet]

Jeff Forcier (bitprophet) posted:

---

Paul Oswald wrote:

Python 2.6.1

Thanks.

Seems random. It will accept my password every now and then. Sudo doesn't normally echo your chars back so I don't know really what is dropping. That's why I tried with pty=False. Consecutive runs will echo different characters. When typing "1234567890" on four consecutive runs with pty=False (various speeds of typing, ctrl c between each run) I got: 6, 340, 38, and 167 echoed back to me. Each character seems to have about a 1 in 5 chance of being echoed. The speed at which I type doesn't seem to matter.

OK, so no real patterns, just intermittent loss of data. Bugger.

I don't have any other code that prompts me so I'm not really sure about that point. I can look into testing it. What should I try?

Just stuff like psql/mysql, package installations which require confirmation, or even a remote Python script that uses raw_input. My main concern here is to figure out if it's localized to the sudo password stuff or if it's for everything.

It also fails for passwords without special chars in it. That doesn't seem to be the problem.

That was just a long shot to narrow things down if there had been some strong pattern above, doesn't matter at this point :)

---

I will try to see if I can replicate this sometime soon. If you're feeling brave, it should be possible to debug this by inspecting fabric/io.py and either logging to file or printing to screen, and see what's out of sync; but no pressure.

---

on 2011-03-18 at 04:26pm EDT

[bitprophet]

Paul Oswald (poswald) posted:

---

Just starting to look into this now but this might be interesting to you... I put the following in io.py Line 96:

    password = fabric.network.prompt_for_password(
                        prompt=" ", no_colon=True, stream=pipe
                    )
    print "pass '%s'" % password

and run with pty=False I get this output when I type '1234567890'

[XX.XX.XX.XX] sudo: chown -R :hats releases/next
[XX.XX.XX.XX] out: sudo password: 1257
pass '346890'

[XX.XX.XX.XX] out: Sorry, try again.

So... that's strange right?

---

on 2011-03-18 at 11:13pm EDT

[bitprophet]

Paul Oswald (poswald) posted:

---

Is it possible that the r, w, x = select([sys.stdin], [], [], 0.0) line of input_loop which is trying to access stdin is getting into a race condition with the get_pass call in network.py which also pulls from stdin? That would explain the random nature of it.

---

on 2011-03-19 at 05:09am EDT

[bitprophet]

Paul Oswald (poswald) posted:

---

I was looking at the issue in #312 which I am also suffering from and I saw the patch adds a sleep into this function. I decided that it might have a significant effect here as well and I tried it out. This gives us further evidence this is a race condition: adding the sleep call to line 131 of io.py from the patch in ticket #312 appears to fix it.

This means you probably cannot reliably reproduce this issue from the main branch even though technically the bug is still there. That patch just lowers the likelihood of input_loop winning the race.

---

on 2011-03-20 at 02:16am EDT

[bitprophet]

Jeff Forcier (bitprophet) posted:

---

Paul -- interesting, but good to know that it may stem from that same problem.

I also have a private report from another user with the exact same problem (as far as I can tell without asking him to debug as you did). When I merge/incorporate that sleep/loop change I'll ask him to test it too before a release goes out.

---

on 2011-03-21 at 04:28pm EDT

[bitprophet]

Jeff Forcier (bitprophet) posted:

---

I cannot reproduce this problem on my end, even after trying a few different target systems and flipping pty between false and true.

However, I was able to put in a patch for #312, so if that root cause was also the problem here, it may be solved now.

Paul, can you nab the 1.0 branch from the Git repo and give it a try to see if it's still fixed it for you? (I changed the sleep value from the original patch.)

---

on 2011-03-22 at 05:12pm EDT

[bitprophet]

Paul Oswald (poswald) posted:

---

Yes the latest from bitprophet/fabric master with that patch seems to work for me. Again, I don't think this actually fixes the underlying problem, it just makes it much less likely to happen.

---

on 2011-03-23 at 07:57am EDT

[bitprophet]

Jeff Forcier (bitprophet) posted:

---

Totally hear you on that point, but I already spend an inordinate amount of time chasing stuff like this. At this point I need to just wait until (if) it becomes a noticeable issue even after this change. I also expect these core aspects to get more attention/changes in the future, too, which may impact it.

---

on 2011-03-23 at 04:09pm EDT

[bitprophet]

Jeff Forcier (bitprophet) posted:

---

I'm going to close this for now since it seems to have gone away for you, please comment if the problem rears its head again at any point.

---

on 2011-03-23 at 04:09pm EDT

[bitprophet]

Paul Oswald (poswald) posted:

---

I've been using this daily and over time I can tell you that it still fails pretty often for me. Maybe every 10-20 times. The main difference is that you get three chances to get it right and it has never failed three times in a row. It is a bit annoying though.

---

on 2011-03-31 at 04:43am EDT

[bitprophet]

Jeff Forcier (bitprophet) posted:

---

Reopened. I was unable to replicate earlier so I doubt I'll be able to replicate now, but I can at least try to double check the code to see if there's some obvious cause of stdin not getting picked up accurately (such as your theory above re: select and getpass fighting over stdin).

---

on 2011-03-31 at 08:18am EDT

[bitprophet]

Paul Oswald (poswald) posted:

---

I ran this code on the latest version:

def sudotest():
  sudo('uname -a' % env, pty=False)

…and then tried typing the digits '123456789' very carefully five times in a row. Here's what I get:

$ fab dev sudotest
[XX.XX.XX.XX] Executing task 'sudotest'
[XX.XX.XX.XX] sudo: uname -a
[XX.XX.XX.XX] out: sudo password: 84

If I hold down a key it prints a character every few seconds or so randomly. My keyboard repeat rate is set fairly high in OS X.

Again, it's not a critical issue for me anymore but it might indicate deeper issues. Hope that helps and let me know if you need any other testing done.

---

on 2011-03-31 at 08:35am EDT

[bitprophet]

**** (teemu) posted:

---

Another victim here. With Fabric 1.0.0 on Macbook Pro with Snow Leopard 10.6.6, sudo password input fails over 80% of attempts, which makes deployment with Fabric impractical unless I manually sudo on server first and use sudo's passwordless time window to do the deployment. Everything worked perfectly with older Fabric (0.8.x, IIRC)

I can dig deeper, as this is critical for me, just tell me what to check. (library versions, what lines of code to debug etc)

---

on 2011-04-01 at 03:12am EDT

[bitprophet]

Ryan Showalter (ryanshow) posted:

---

I'm also experiencing this issue.

The first sudo attempt almost always fails, and the subsequent requests have about a 50% success rate.

Using Python 2.6 and Fabric 1.0

This was not an issue with Fabric 0.9

---

on 2011-04-12 at 08:58pm EDT

[bitprophet]

Jeff Forcier (bitprophet) posted:

---

Teemu, Ryan -- did you guys see the 1.0.1 release? If you could upgrade to it and verify Paul's report that 1.0.1 helps a lot but does not completely eradicate the problem, that would be useful.

I am still planning on investigating this at a deeper level, starting with Paul's theory re: getpass vs our input loop.

---

on 2011-04-19 at 11:36am EDT

[bitprophet]

Rocky Burt (rockyburt) posted:

---

I'd like to mention that this bug is affecting me as well. On my Fabric 1.0.1 install ... about 3 out of 4 sudo tries ends up failing due to this issue while running in a terminal. Oddly enough, if I run it from a shell inside emacs, it never fails.

---

on 2011-04-26 at 07:22am EDT

[bitprophet]

Rocky Burt (rockyburt) posted:

---

It's now gotten to a point that I can never put my sudo password in correctly in a terminal and must always do it through Emacs.

---

on 2011-04-27 at 08:40am EDT

[bitprophet]

Rocky Burt (rockyburt) posted:

---

One more update... my versions are:
Python 2.7.1 (Ubuntu Natty 64bit)
Fabric 1.0.1
Paramiko 1.7.6

---

on 2011-04-27 at 08:42am EDT

[bitprophet]

Edoardo Batini (eolo999) posted:

---

Same problem here:

python 2.7.1
paramiko 1.7.6
fabric 1.0.1

---

on 2011-05-02 at 09:57am EDT

[bitprophet]

Richard Anderson (AntiMS) posted:

---

Same problem here as well.

Python 2.7.1, Paramiko 1.7.6, Fabric 1.0.1

My password attempts probably fail around 99% of the time at this point. Yes, I literally have to put my password in an average of 100 times to get a "hit".

If it makes a difference I'm running the fab command itself on a remote box to which I am connected using regular old OpenSSH. I'm running fab with localhost as the remote box. (That is, I'm doing a "fab -H 127.0.0.1 some fabric commands".)

---

on 2011-05-17 at 12:52pm EDT

[bitprophet]

Orphée Lafond-Lummis (orftz) posted:

---

Same thing here. Python 2.7.1, Paramiko 1.7.6, Fabric 1.0.1

Ran Fabric via ssh both locally and remotely, it's the same pain in both cases. However, when Fabric is given the arg --show=debug, everything seem to be alright.

---

on 2011-05-17 at 04:01pm EDT

[bitprophet]

Max Arnold (LwarX) posted:

---

Can someone try this patch ? It fixed (probably) related issue #352 for me.

---

on 2011-05-18 at 01:13am EDT

[bitprophet]

Orphée Lafond-Lummis (orftz) posted:

---

Installed latest master branch (commit 3b4d8dd) from Github, and had the same problem as before. I also installed latest 1.1 branch (commit c3e79b6) and had the same problem.

With your patch on top of both branches, I had no problem while testing sudo locally and remotely. I tried 10-20 times, so while it apparently fixes the bug, I cannot assure it does. Plus, I'm not familiar with the Fabric codebase so I don't know if your patch breaks something that I'm not aware of.

Anyways, personally I'll continue to use your patch Max Arnold. Thank you very much for it. Have a good day.

---

on 2011-05-18 at 12:33pm EDT

[bitprophet]

Andi Albrecht (andialbrecht) posted:

---

Same issue here: Python 2.7.1, Fabric 1.1a, Paramiko 1.7.6

Patch by Max Arnold solved it for me! Thanks :)

---

on 2011-06-10 at 07:25am EDT

[bitprophet]

Jeff Forcier (bitprophet) posted:

---

Applied in changeset commit:efd9fef820dd6f2a593c172eac7e500c5e8f0602.

---

on 2011-06-23 at 11:09pm EDT

[bitprophet]

Jeff Forcier (bitprophet) posted:

---

1.0.2 just went out the door, so if y'all could grab that and let me know if it does NOT appear to solve the problem, that'd be appreciated. Took Max's patch verbatim, but I was never able to reproduce the symptoms myself so I'm going on faith.

---

on 2011-06-24 at 08:35pm EDT