Remote command errors with operation not permitted

[jamiees2]

Hi.

When using the remote-shell-cmd runner as follows:

st2 run core.remote hosts="localhost" username="" password="" cmd="ls"

, it always fails with the following error:

status: failed
result:
{
    "localhost": {
        "failed": true,
        "traceback": "  File "/usr/lib/python2.7/site-packages/st2common/models/system/action.py", line 364, in _run
    output = run(self.command, combine_stderr=False, pty=False, quiet=True)
  File "/usr/lib64/python2.7/site-packages/fabric/network.py", line 649, in host_prompting_wrapper
    return func(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/fabric/operations.py", line 1056, in run
    shell_escape=shell_escape)
  File "/usr/lib64/python2.7/site-packages/fabric/operations.py", line 925, in _run_command
    stderr=stderr, timeout=timeout)
  File "/usr/lib64/python2.7/site-packages/fabric/operations.py", line 809, in _execute
    worker.raise_if_needed()
  File "/usr/lib64/python2.7/site-packages/fabric/thread_handling.py", line 12, in wrapper
    callable(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/fabric/io.py", line 231, in input_loop
    r, w, x = select([sys.stdin], [], [], 0.0)
  File "/usr/lib/python2.7/site-packages/eventlet/green/select.py", line 79, in select
    listeners.append(hub.add(hub.READ, k, on_read, on_error, lambda x: None))
  File "/usr/lib/python2.7/site-packages/eventlet/hubs/epolls.py", line 53, in add
    self.register(fileno, new=True)
  File "/usr/lib/python2.7/site-packages/eventlet/hubs/poll.py", line 45, in register
    self.poll.register(fileno, mask)
",
        "succeeded": false,
        "error": "[Errno 1] Operation not permitted"
    }
}

Any ideas?

[lakshmi-kannan]

@jamiees2: What version of st2 are you using? Maybe you are affected by #1762.

It's weird. In some hosts, setting parallel to True breaks (this is understandable because of the interaction between multiprocessing and eventlets). So we set it to False and some users have complained about weird errors. Depending on your st2 version, it might be either True or False. We could switch it to the opposite and check.

BTW username and password need not be provided. It looks like they are both empty.

Also, we are replacing the current SSH runner. Fabric is really buggy especially in an eventlet environment. Progress is being made as we speak. lakshmi-kannan#2.

[jamiees2]

$ st2 --version
st2 ('0.11.1.7',)

I just tried adding the parameter parallel: "False" and parallel: "True", neither worked. I'll try updating and see what happens then.

Oh, and the username and password were provided, just removed for posting here :)

That's awesome aswell, is there any ETA for when the new ssh runner will land? I'm working on getting windows+ssh working in stackstorm using freeSSHd. I have the runner ready, but just need to debug this error to test it.

[jamiees2]

Great! Updating stackstorm resolved the issue. Thanks!

[lakshmi-kannan]

@jamiees2 Just want to call out that the upgrade fixing the issue is more coincidental than anything else. We don't understand the root cause very well. The real fix would be to swap fabric with the new runner which we got a better grip for. Regarding ETA, it should be available in the next release. I can let you know when it's ready for testing and give you instructions on how to try it out i.e. if you are interested and want to help us in testing :).

[jamiees2]

@lakshmi-kannan Great! Sure, happy to help :)

I'm currently working on this runner: https://gist.github.com/jamiees2/6e744897eae5107b63de which works as of now, although some cleanup is needed ;).

Also, perhaps this is relevant? http://stackoverflow.com/questions/25224875/python-eventlet-fabric-gives-ioerror-errno-1-operation-not-permitted-when-run I found this, but wasn't in the mood for replacing eventlet.

[lakshmi-kannan]

Yeah, we identified a bunch of bugs with fabric + eventlet. We use a fork of fabric. Anyways, long story short, fabric doesn't work well for us.

About your windows runner, I am little confused why you can't use the default ssh runner. Is there some cross platform issue that's stopping you? I took a cursory look at your runner and didn't see much difference from existing fabric runner for *nix.

[jamiees2]

Well, to be honest, there was just this line here: https://gist.github.com/jamiees2/6e744897eae5107b63de#file-windows-py-L82

Initially there was more, but I realized that wasn't neccessary.

The script runner has to be somewhat different because of entirelly different commands though :)