WinError87 when executing local on Connection

[KantiCodes]

Hi all, I`m new to Invoke and even newer to Fabric!

I have some functionality in Invoke and now I`m trying to make a similar demo using fabric

Following code results in WinError87

[WinError 87] Parameter is incorrect

I know it's probably caused by me misunderstanding something - but I`d really appreciate a hint :)

Setup:

Python 3.8.5
Fabric 2.6.0

[davidjmemmett]

Hi KantiCodes,

Just to confirm, you have an SSH server running on localhost on port 8050, and the username you're wanting to connect to is 45502?

Which SSH server are you using? If you open an SSH connection using the standard openssh client (ssh 45502@localhost -p8050) and run cd .., does this work as expected?

Cheers,
David

[KantiCodes]

Hi @davidjmemmett

Thanks so much for taking time to look at my problem :)

No, I do not have an SSH running - my understanding was that until I only run c.local() the connection is not established and it's just as if I was using invoke.

Sorry about that.

To add a little background to what I want to do.

I created invoke module and wrapped it into my own pip installing library: http://docs.pyinvoke.org/en/stable/concepts/library.html#reusing-invoke-s-cli-module-as-a-distinct-binary

Here is a simplified example:

tasker/main.py

tasker/tasks.py

I can install and access my wrapped "module"

Now my goal:

• For some of the tasks I need to communicate with web server, therefore I would like to use Fabric and keep my current(Invoke's) functionality and add new tasks that.

As far as I understand @bitprophet designed fabric to be Invoke+, therefore I would like to keep my Invoke solution and add being able to connect over SSH to web servers.

To wrap up:
I want to create a fabric module performing tasks both locally and externally(over ssh) and wrap that into a pip package that I can install :D...

I am all new to this topic - let me know if my intentions are clear

[davidjmemmett]

Gotcha - would you be able to provide the full stack trace please? I'd like to see what's happening further up the stack.

[KantiCodes]

Hey. there you go.

[davidjmemmett]

From what I can see, this appears to be an invoke issue - cc'ing @jefftriplett

I don't have a Microsoft Windows machine at hand to test this with I'm afraid.

[jefftriplett]

Sadly, I don't have a Windows machine to test either.

I'm a little confused by using local through with a remote connection though. I do see it's documented in the docs by what looks like a pass-through to invoke.

I would try using invoke.run() locally to see if you get the same error. The locals sort-of lead me to this from https://docs.fabfile.org/en/2.5/api/connection.html#fabric.connection.Connection.local to http://docs.pyinvoke.org/en/latest/api/__init__.html#invoke.run

[KantiCodes]

Thanks for your answer @jefftriplett.

The reason why I would like to use fabric.local instead of the invoke.run is following:

• As I described my end goal above the tool I`m building is going to use both Invoke and Fabric and it is my understanding that I need to build a Fabric tool that uses invoke (either by sub classing or explicitly) and not the other way around.

Nonetheless I have seen some code examples where people were doing following:
root/tasks.py

• Note that the task is from invoke

root/main.py

So now I am a little bit confused which one to choose. I do understand though that using fabric.local() is equivalent to invoke.run() I just thought that's the way to go.

[jefftriplett]

@KantiCodes sorry for the confusion. I was mostly curious if one worked vs. the other to help us figure out if the issue was really Invoke or Fabric since neither of us has a Windows machine to test it on.

[KantiCodes]

No probs.

To answer your question: Invoke works properly.

[KantiCodes]

Btw, could you elaborate on the screen shoot I sent?

What do u think about using Fabric functionality inside invoke@task?

[davidjmemmett]

I'm convinced that the issue you're having is related to the Microsoft Windows executor, rather than anything odd happening between invoke and fabric invoking invoke. I'll spin up a Microsoft Windows test VM and have a play. For now, here's my findings:

tasks.py:

from invoke import task
from fabric import Connection

@task
def test(c):
    c.run('echo test from invoke')

    fab_c = Connection('localhost')
    fab_c.local('echo test from invoke via fabric')

fabfile.py is a symbolic link to tasks.py, meaning they're identical:

$ stat fabfile.py 
  File: fabfile.py -> tasks.py
  Size: 8         	Blocks: 0          IO Block: 4096   symbolic link
Device: 802h/2050d	Inode: 1192219     Links: 1
Access: (0777/lrwxrwxrwx)  Uid: ( 1000/   david)   Gid: ( 1000/   david)
Access: 2021-02-22 20:41:32.409633886 +0000
Modify: 2021-02-22 20:41:30.489574349 +0000
Change: 2021-02-22 20:41:30.489574349 +0000
 Birth: -

invoke output:

$ inv test
test from invoke
test from invoke via fabric

fabric output:

$ fab test
test from invoke
test from invoke via fabric

[davidjmemmett]

Hi @KantiCodes,

I'm afraid I'm unable to reproduce this.

The steps I took:

1. Create a Microsoft Windows 10 VM using the free images provided.
2. Installed Python 3.6.6 using the Windows x86-64 executable installer (checked the box to add python to the $PATH.
3. Launched Microsoft PowerShell
4. Upgraded pip:
   > python -m pip install --upgrade pip
5. Installed fabric
> python -m pip install fabric
6. Created a tasks.py identical to the one above:
   > notepad.exe tasks.py
7. Copied tasks.py into fabfile.py (as Microsoft Windows still doesn't properly support symbolic links apparently):
   > cp tasks.py fabfile.py

PS C:\Users\IEUser> inv test
test from invoke
test from fabric
PS C:\Users\IEUser> fab test
test from invoke
test from fabric

Even after replacing the echo calls with cd .., it had no effect.

Can you please provide a small, standalone fabfile.py to reproduce this issue please? If possible, using a clean VM environment to test this would be preferable.

Cheers,
David

[davidjmemmett]

I do apologise, you're using Python 3.8.5; I'll grab that and test!

[davidjmemmett]

Aha, this is definitely related to the Python version. I've managed to reproduce this issue using Python 3.8.5.

[davidjmemmett]

I've also managed to reproduce this issue using Python 3.8.8 (latest 3.8).

[davidjmemmett]

Even in the latest Python 3.9.2 released a couple of days ago.

[davidjmemmett]

I've narrowed the issue itself down to env being dropped when running under fabric. Dumping the arguments to invoke.runners.Local:start shows that env is fully populated when running under invoke, but an empty dictionary when running under fabric.

[davidjmemmett]

It looks like this is a known issue. If I set replace_env to False, it works as expected, however this will need to be set specifically for local calls only and never for remove calls. I wonder why it worked with Python 3.6.6 though... weird.

I'll continue to investigate.

[davidjmemmett]

Hi @KantiCodes,

As a temporary test (please do not rely on this long-term), you could install fabric from the commit I pushed to see if this fixes the issue for you:

python -m pip install git+https://github.com/davidjmemmett/fabric.git@f8cf9ad30efb773f055bf7c174ef98ba225405fc

Cheers,
David

[davidjmemmett]

Disclaimer: given the nature of this commit, there are potential security implications (whereby local environment variables have the potential to be transmitted unknowingly): i.e. it's not yet been reviewed.

[KantiCodes]

Hi @davidjmemmett,

Considering what you wrote, for now I will just stick with Invoke instead of sub-classing.

[bitprophet]

Thanks for triaging this David! I may be reading too fast but I don't see if we've found what changed in Python 3.8 to break this - the ultimate change you identified needs to happen regardless, but it's weird if Python 3.6 and 3.7 don't trigger the issue.

@KantiCodes as David and Jeff noted, using "straight Invoke" is fine here, especially if it works around the issue for now. Eventually I do want "just use Fabric because you intend to do both local and remote things" to be a smooth experience - it's not there yet unfortunately. Mixing the two in the same code shouldn't be all that much more annoying for now, hopefully!

I'm talking with David in his PR and I expect to merge it sometime, so let's close this to roll it into there. Thanks for the report!

[fripSide]

I also meet the same problem. In Python 3.8.5 with Fabric 2.6.9, I failed to execute local commend in CMD with either ctx.run or conn.local.
`
from fabric import task, Connection

@task
def test(ctx):
conn = Connection("xxx")
ctx.run("cd D:/")
# conn.local("cd D:/")
`

The error message is,
File "d:\dev\python\python3.8.5\lib\site-packages\invoke\runners.py", line 1271, in start
self.process = Popen(
File "d:\dev\python\python3.8.5\lib\subprocess.py", line 854, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "d:\dev\python\python3.8.5\lib\subprocess.py", line 1308, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
OSError: [WinError 87]

I solve this problem by manually merge the two commits.
davidjmemmett@0ce88d2
https://github.com/davidjmemmett/fabric/blob/0ce88d26ee69f22fda1ebcd4bd579e1652131fd6/fabric/runners.py

I hope that you can release a new version to fix this problem as many people use Python 3.8/3.9 now.

[wookiesh]

Hello, I'm experiencing the same issue here with pythonpython 3.9.6 and fabric 2.6 (up to date according to poetry ?) but the issue is closed.

Any pointer on how to address it ?

[davidjmemmett]

There hasn't been a release for a while - @bitprophet

[swasher]

I have this issue too (Windows10, Python 3.9.5, fabric 2.6.0). But if I understood the thread correctly, the problem appear only with local connection. I have with remote too, BUT only in tasks:

from fabric import Connection
c = Connection('sandglass', user='swasher')
c.run('echo test from fabric')

Output:

 λ fab
test from fabric
Usage: fab [--cor....

If I try put this into task, I got error:

from fabric import Connection, task

c = Connection('sandglass', user='swasher')

@task
def proba(c):
    c.run('echo test from fabric')

Output:

(sandglass) λ fab proba
Traceback (most recent call last):
  File "c:\users\alexeyd\appdata\local\programs\python\python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\users\alexeyd\appdata\local\programs\python\python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\alexeyd\.virtualenvs\sandglass-rdO3Dwhs\Scripts\fab.exe\__main__.py", line 7, in <module>
  File "c:\users\alexeyd\.virtualenvs\sandglass-rdo3dwhs\lib\site-packages\invoke\program.py", line 384, in run
    self.execute()
  File "c:\users\alexeyd\.virtualenvs\sandglass-rdo3dwhs\lib\site-packages\invoke\program.py", line 566, in execute
    executor.execute(*self.tasks)
  File "c:\users\alexeyd\.virtualenvs\sandglass-rdo3dwhs\lib\site-packages\invoke\executor.py", line 129, in execute
    result = call.task(*args, **call.kwargs)
  File "c:\users\alexeyd\.virtualenvs\sandglass-rdo3dwhs\lib\site-packages\invoke\tasks.py", line 127, in __call__
    result = self.body(*args, **kwargs)
  File "C:\Users\alexeyd\PycharmProjects\sandglass\fabfile.py", line 8, in proba
    c.run('echo test from invoke')
  File "c:\users\alexeyd\.virtualenvs\sandglass-rdo3dwhs\lib\site-packages\invoke\context.py", line 95, in run
    return self._run(runner, command, **kwargs)
  File "c:\users\alexeyd\.virtualenvs\sandglass-rdo3dwhs\lib\site-packages\invoke\context.py", line 102, in _run
    return runner.run(command, **kwargs)
  File "c:\users\alexeyd\.virtualenvs\sandglass-rdo3dwhs\lib\site-packages\invoke\runners.py", line 380, in run
    return self._run_body(command, **kwargs)
  File "c:\users\alexeyd\.virtualenvs\sandglass-rdo3dwhs\lib\site-packages\invoke\runners.py", line 431, in _run_body
    self.start(command, self.opts["shell"], self.env)
  File "c:\users\alexeyd\.virtualenvs\sandglass-rdo3dwhs\lib\site-packages\invoke\runners.py", line 1291, in start
    self.process = Popen(
  File "c:\users\alexeyd\appdata\local\programs\python\python39\lib\subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "c:\users\alexeyd\appdata\local\programs\python\python39\lib\subprocess.py", line 1420, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
OSError: [WinError 87] Параметр задан неверно

Russian output, OSError: [WinError 87] Параметр задан неверно -> The parameter is incorrect