Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error while executing "python setup.py" in CLI : FileNotFoundError: [WinError 2] The system cannot find the file specified #1

Closed
ghost opened this issue Oct 3, 2022 · 8 comments

Comments

@ghost
Copy link

ghost commented Oct 3, 2022

When did the bug happened :

  • When I run the command "python setup.py" in the command prompt, I got this traceback :
Installing Pytorch (this may take a few minutes):
Traceback (most recent call last):
  File "C:\Users\undisclosed\anaconda3\envs\dream-factory\dream-factory\setup.py", line 152, in <module>
    install_pytorch(verbose)
  File "C:\Users\undisclosed\anaconda3\envs\dream-factory\dream-factory\setup.py", line 55, in install_pytorch
    exec(cmd, verbose)
  File "C:\Users\undisclosed\anaconda3\envs\dream-factory\dream-factory\setup.py", line 27, in exec
    subprocess.run(command.split(' '), stdout=subprocess.DEVNULL)
  File "C:\Users\undisclosed\anaconda3\envs\dream-factory\lib\subprocess.py", line 505, in run
    with Popen(*popenargs, **kwargs) as process:
  File "C:\Users\undisclosed\anaconda3\envs\dream-factory\lib\subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "C:\Users\undisclosed\anaconda3\envs\dream-factory\lib\subprocess.py", line 1420, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified

What OS was affected :

  • Windows 10 Pro (64-bit) version 21H2

The solution :

  • I changed those lines in "setup.py" :
- Line 25        subprocess.run(command.split(' '))
+ Line 25        subprocess.run(command.split(' '), shell=True)

- Line 27        subprocess.run(command.split(' '), stdout=subprocess.DEVNULL)
+ Line 27        subprocess.run(command.split(' '), stdout=subprocess.DEVNULL, shell=True)

Where did I find the solution :

Final Note :

  • When the solution was applied, the rest of the setup went well and the program work very well !
  • I already made 30 new AI-art images. Thank you very much ! 😁😎🤩
rbbrdckybk added a commit that referenced this issue Oct 3, 2022
@rbbrdckybk
Copy link
Owner

Thanks for the feedback; didn't run into that issue on any of my test installs but just applied a fix that should prevent it in the future!

@ghost
Copy link
Author

ghost commented Oct 3, 2022

>> Testing the latest version of the code with a fresh Anaconda environment

Same result :

Installing Pytorch (this may take a few minutes):
Traceback (most recent call last):
  File "C:\Users\undisclosed\anaconda3\envs\dream-factory-test-env\dream-factory\setup.py", line 155, in <module>
    install_pytorch(verbose)
  File "C:\Users\undisclosed\anaconda3\envs\dream-factory-test-env\dream-factory\setup.py", line 55, in install_pytorch
    exec(cmd, verbose)
  File "C:\Users\undisclosed\anaconda3\envs\dream-factory-test-env\dream-factory\setup.py", line 27, in exec
    subprocess.run(shlex.split(command), stdout=subprocess.DEVNULL)
  File "C:\Users\undisclosed\anaconda3\envs\dream-factory-test-env\lib\subprocess.py", line 505, in run
    with Popen(*popenargs, **kwargs) as process:
  File "C:\Users\undisclosed\anaconda3\envs\dream-factory-test-env\lib\subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "C:\Users\undisclosed\anaconda3\envs\dream-factory-test-env\lib\subprocess.py", line 1426, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified

Test : Failed ❌

>> I put a print statement just before the exception was raised so I can see what was the final inputs before the crash :

(dream-factory-test-env) C:\Users\undisclosed\anaconda3\envs\dream-factory-test-env\dream-factory>python setup.py --force
True

Cloning all required repositories:
   fetching CompVis/taming-transformers...

executable:     None,
args:           git clone https://github.com/CompVis/taming-transformers,
creationflags:  0,
env:            None,
cwd:            None,
startupinfo:    <subprocess.STARTUPINFO object at 0x000001D3E676B610>

fatal: destination path 'taming-transformers' already exists and is not an empty directory.
   fetching openai/CLIP...

executable:     None,
args:           git clone https://github.com/openai/CLIP,
creationflags:  0,
env:            None,
cwd:            None,
startupinfo:    <subprocess.STARTUPINFO object at 0x000001D3E676B1F0>

fatal: destination path 'CLIP' already exists and is not an empty directory.
   fetching rbbrdckybk/stable-diffusion...

executable:     None,
args:           git clone https://github.com/rbbrdckybk/stable-diffusion,
creationflags:  0,
env:            None,
cwd:            None,
startupinfo:    <subprocess.STARTUPINFO object at 0x000001D3E671BEE0>

fatal: destination path 'stable-diffusion' already exists and is not an empty directory.
   fetching xinntao/Real-ESRGAN...

executable:     None,
args:           git clone https://github.com/xinntao/Real-ESRGAN,
creationflags:  0,
env:            None,
cwd:            None,
startupinfo:    <subprocess.STARTUPINFO object at 0x000001D3E671BF70>

fatal: destination path 'Real-ESRGAN' already exists and is not an empty directory.

Installing Pytorch (this may take a few minutes):

executable:     None,
args:           conda install -y pandas pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch,
creationflags:  0,
env:            None,
cwd:            None,
startupinfo:    <subprocess.STARTUPINFO object at 0x000001D3E671BEE0>

Traceback (most recent call last):
  File "C:\Users\undisclosed\anaconda3\envs\dream-factory-test-env\dream-factory\setup.py", line 155, in <module>
    install_pytorch(verbose)
  File "C:\Users\undisclosed\anaconda3\envs\dream-factory-test-env\dream-factory\setup.py", line 55, in install_pytorch
    exec(cmd, verbose)
  File "C:\Users\undisclosed\anaconda3\envs\dream-factory-test-env\dream-factory\setup.py", line 27, in exec
    subprocess.run(shlex.split(command), stdout=subprocess.DEVNULL)
  File "C:\Users\undisclosed\anaconda3\envs\dream-factory-test-env\lib\subprocess.py", line 505, in run
    with Popen(*popenargs, **kwargs) as process:
  File "C:\Users\undisclosed\anaconda3\envs\dream-factory-test-env\lib\subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "C:\Users\undisclosed\anaconda3\envs\dream-factory-test-env\lib\subprocess.py", line 1426, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified

>> My new guess :

Anaconda does not recommend adding itself to system PATH anymore and does not offer the option to do it automatically so you have to do it manually. If you have a new installation and you follow the installation recommendations as I did, conda won't be accessible outside an Anaconda shell :

Anaconda Installation Window

So when calling "subprocess.run()" without the "shell=True" argument... a new command prompt is spawned, and calling "conda install -y pandas pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch" won't work because, "conda" can't be found in system PATH.

It works with "shell=True" because it uses the "current" open shell which for me... is the Anaconda shell with the correct "env" activated

>> Testing my new hypothesis :

  • added "C:\Users\undisclosed\anaconda3" and "C:\Users\undisclosed\anaconda3\Scripts" to system PATH
  • create a fresh Anaconda environment
  • run setup.py

result :

(dream-factory-test-env_2) C:\Users\undisclosed\anaconda3\envs\dream-factory-test-env_2\dream-factory>python setup.py

Cloning all required repositories:
   fetching CompVis/taming-transformers...
Cloning into 'taming-transformers'...
remote: Enumerating objects: 1335, done.
Receiving objects:  96% (1282/1335), 407.96 MiB | 3.21 MiB/sd 1335 eceiving objects:  95% (1269/1335), 407.96 MiB | 3.21 MiB/s
Receiving objects: 100% (1335/1335), 409.77 MiB | 2.30 MiB/s, done.
Resolving deltas: 100% (278/278), done.
   fetching openai/CLIP...
Cloning into 'CLIP'...
remote: Enumerating objects: 236, done.
Receiving objects: 100% (236/236), 8.92 MiB | 7.09 MiB/s, done.6 eceiving objects:  99% (235/236), 7.01 MiB | 7.00 MiB/s

Resolving deltas: 100% (122/122), done.
   fetching rbbrdckybk/stable-diffusion...
Cloning into 'stable-diffusion'...
remote: Enumerating objects: 477, done.
Receiving objects:  93% (444/477), 42.32 MiB | 3.16 MiB/ssed 477Receiving objects:  91% (435/477), 42.32 MiB | 3.16 MiB/s
Receiving objects: 100% (477/477), 42.51 MiB | 2.91 MiB/s, done.
Resolving deltas: 100% (200/200), done.
   fetching xinntao/Real-ESRGAN...
Cloning into 'Real-ESRGAN'...
remote: Enumerating objects: 755, done.
remote: Counting objects: 100% (20/20), done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 755 (delta 5), reused 14 (delta 3), pack-reused 735
Receiving objects: 100% (755/755), 5.38 MiB | 7.01 MiB/s, done.
Resolving deltas: 100% (406/406), done.

Installing Pytorch (this may take a few minutes):

Installing all required dependencies:
   installing basicsr...
   installing cherrypy...
   installing datetime...
   installing diffusers...
   installing einops...
   installing facexlib...
   installing ftfy...
   installing gfpgan...
   installing imageio...
   installing imageio-ffmpeg...
   installing kornia...
   installing numpy...
   installing omegaconf...
   installing opencv-python...
   installing pillow...
   installing pytorch-lightning...
   installing torch-fidelity...
   installing transformers...

Setting up Real-ESRGAN:
   downloading pre-trained models...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 63.9M  100 63.9M    0     0  7389k      0  0:00:08  0:00:08 --:--:-- 7751k
   running ESRGAN setup script...

Creating config.txt containing default settings...


All done - don't forget to place your model.ckpt file in this directory! :
stable-diffusion\models\ldm\stable-diffusion-v1


(dream-factory-test-env_2) C:\Users\undisclosed\anaconda3\envs\dream-factory-test-env_2\dream-factory>

Test : Passed ✔

>> Suggestion :

Considering that adding the "shell=True" argument to "subprocess.run()" is considered a security flaw because it is vulnerable to shell injection.

My only recommendation would be to add, in the "Requirements section", that Anaconda must be in the system PATH before running "setup.py" otherwise... all new Anaconda installations like mine are doomed to fail.

@rbbrdckybk
Copy link
Owner

Wow, that's for the very detailed write-up!

I can't reproduce this on my end, but I can add a command-line switch to enable shell=True. I'll check for FileNotFoundError and display some text explaining that it can be enabled if the user wants. Will re-open this until I have a minute to make the change.

@rbbrdckybk rbbrdckybk reopened this Oct 5, 2022
@masonc15
Copy link

masonc15 commented Oct 7, 2022

Just wanted to add a note that I also encountered this issue on Windows. Thanks for the solution!

rbbrdckybk added a commit that referenced this issue Oct 15, 2022
@rbbrdckybk
Copy link
Owner

Finally got around to handling this (hopefully - my installations work without the shell option so I can't do a perfect test). Added a FileNotFoundError check and prompt the user to re-run setup with a new --shell option, which should allow the installation to work in environments that require shell=true.

@ghost
Copy link
Author

ghost commented Oct 16, 2022

>> Testing the latest version of the code with a fresh Anaconda environment, in a VM, with the defaults Anaconda installation options :

Testing that the error is still present in the default install :

(dream-factory) C:\Users\undisclosed\dream-factory>python setup.py

Cloning all required repositories:
   fetching CompVis/taming-transformers...
Cloning into 'taming-transformers'...
remote: Enumerating objects: 1335, done.
remote: Total 1335 (delta 0), reused 0 (delta 0), pack-reused 1335 receiving objects:  95% (1269/1335), 408.75 MiB | 5.08 MiB/s
Receiving objects: 100% (1335/1335), 409.77 MiB | 4.09 MiB/s, done.
Resolving deltas: 100% (278/278), done.
   fetching openai/CLIP...
Cloning into 'CLIP'...
remote: Enumerating objects: 236, done.
remote: Total 236 (delta 0), reused 0 (delta 0), pack-reused 236R
Receiving objects: 100% (236/236), 8.92 MiB | 7.10 MiB/s, done.
Resolving deltas: 100% (122/122), done.
   fetching rbbrdckybk/stable-diffusion...
Cloning into 'stable-diffusion'...
remote: Enumerating objects: 492, done.
remote: Counting objects: 100% (11/11), done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 492 (delta 4), reused 10 (delta 4), pack-reused 481
Receiving objects: 100% (492/492), 42.53 MiB | 3.62 MiB/s, done.
Resolving deltas: 100% (207/207), done.
   fetching xinntao/Real-ESRGAN...
Cloning into 'Real-ESRGAN'...
remote: Enumerating objects: 755, done.
remote: Counting objects: 100% (20/20), done.
remote: Compressing objects: 100% (13/13), done.
remote: Total 755 (delta 5), reused 15 (delta 4), pack-reused 735
Receiving objects: 100% (755/755), 5.37 MiB | 5.00 MiB/s, done.
Resolving deltas: 100% (406/406), done.

Installing Pytorch (this may take a few minutes):

Unable to find executable - try re-running setup with the --shell option, e.g.:
   python setup.py --shell --force

The error is still present 🚧

>> Testing the new "--shell" flag :

(dream-factory) C:\Users\undisclosed\dream-factory>python setup.py --shell --force

Cloning all required repositories:
   fetching CompVis/taming-transformers...
fatal: destination path 'taming-transformers' already exists and is not an empty directory.
   fetching openai/CLIP...
fatal: destination path 'CLIP' already exists and is not an empty directory.
   fetching rbbrdckybk/stable-diffusion...
fatal: destination path 'stable-diffusion' already exists and is not an empty directory.
   fetching xinntao/Real-ESRGAN...
fatal: destination path 'Real-ESRGAN' already exists and is not an empty directory.

Installing Pytorch (this may take a few minutes):

Unable to find executable - try re-running setup with the --shell option, e.g.:
   python setup.py --shell --force

Test : Failed ❌

The solution :

The "shell" parameter is not used when calling "install_pytorch()" function, in "setup.py".

  • I changed this line in "setup.py" :
- Line 55        exec(cmd, verbose)
+ Line 55        exec(cmd, verbose, shell)

>> Testing the new solution :

(dream-factory) C:\Users\undisclosed\dream-factory>python setup.py --shell --force

Cloning all required repositories:
   fetching CompVis/taming-transformers...
fatal: destination path 'taming-transformers' already exists and is not an empty directory.
   fetching openai/CLIP...
fatal: destination path 'CLIP' already exists and is not an empty directory.
   fetching rbbrdckybk/stable-diffusion...
fatal: destination path 'stable-diffusion' already exists and is not an empty directory.
   fetching xinntao/Real-ESRGAN...
fatal: destination path 'Real-ESRGAN' already exists and is not an empty directory.

Installing Pytorch (this may take a few minutes):

Installing all required dependencies:
   installing basicsr...
   installing cherrypy...
   installing datetime...
   installing diffusers...
   installing einops...
   installing facexlib...
   installing ftfy...
   installing gfpgan...
   installing imageio...
   installing imageio-ffmpeg...
   installing kornia...
   installing numpy...
   installing omegaconf...
   installing opencv-python...
   installing pillow...
   installing pytorch-lightning...
   installing torch-fidelity...
   installing transformers...

Setting up Real-ESRGAN:
   downloading pre-trained models...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 63.9M  100 63.9M    0     0  7195k      0  0:00:09  0:00:09 --:--:-- 7724k
   running ESRGAN setup script...

Creating config.txt containing default settings...


All done - don't forget to place your model.ckpt file in this directory! :
stable-diffusion\models\ldm\stable-diffusion-v1

Test : Passed ✔

To-do:

  • Update line #55 in "setup.py", according to the solution section.

Thank you ! 😄

@rbbrdckybk
Copy link
Owner

That's what I get for committing without testing! 😄 Should be fixed now - appreciate the help!

@ghost
Copy link
Author

ghost commented Oct 18, 2022

>> Final test of the latest version of the code, in a VM. Defaults Anaconda installation options. [Closing this issue] :

Testing that the error is still present in the default install :

(dream-factory-2) C:\Users\undisclosedos\dream-factory-2\dream-factory>python setup.py

Cloning all required repositories:
   fetching CompVis/taming-transformers...
Cloning into 'taming-transformers'...
remote: Enumerating objects: 1335, done.
remote: Total 1335 (delta 0), reused 0 (delta 0), pack-reused 1335
Receiving objects: 100% (1335/1335), 409.77 MiB | 6.37 MiB/s, done.
Resolving deltas: 100% (280/280), done.
   fetching openai/CLIP...
Cloning into 'CLIP'...
remote: Enumerating objects: 236, done.
remote: Total 236 (delta 0), reused 0 (delta 0), pack-reused 236 eceiving objects: 100% (236/236), 7.19 MiB | 7.12 MiB/sReceiving objects: 100% (236/236), 8.92 MiB | 7.04 MiB/s, done.

Resolving deltas: 100% (120/120), done.
   fetching rbbrdckybk/stable-diffusion...
Cloning into 'stable-diffusion'...
remote: Enumerating objects: 492, done.
remote: Counting objects: 100% (11/11), done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 492 (delta 4), reused 10 (delta 4), pack-reused 481
Receiving objects: 100% (492/492), 42.53 MiB | 4.59 MiB/s, done.
Resolving deltas: 100% (207/207), done.
   fetching xinntao/Real-ESRGAN...
Cloning into 'Real-ESRGAN'...
remote: Enumerating objects: 755, done.
remote: Counting objects: 100% (20/20), done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 755 (delta 5), reused 14 (delta 3), pack-reused 735
Receiving objects: 100% (755/755), 5.38 MiB | 7.02 MiB/s, done.
Resolving deltas: 100% (406/406), done.

Installing Pytorch (this may take a few minutes):

Unable to find executable - try re-running setup with the --shell option, e.g.:
   python setup.py --shell --force

The error is still present 🚧

>> Testing "--shell" flag :

(dream-factory-2) C:\Users\Yoshi_Kakarinos\dream-factory-2\dream-factory>python setup.py --shell --force

Cloning all required repositories:
   fetching CompVis/taming-transformers...
fatal: destination path 'taming-transformers' already exists and is not an empty directory.
   fetching openai/CLIP...
fatal: destination path 'CLIP' already exists and is not an empty directory.
   fetching rbbrdckybk/stable-diffusion...
fatal: destination path 'stable-diffusion' already exists and is not an empty directory.
   fetching xinntao/Real-ESRGAN...
fatal: destination path 'Real-ESRGAN' already exists and is not an empty directory.

Installing Pytorch (this may take a few minutes):

Installing all required dependencies:
   installing basicsr...
   installing cherrypy...
   installing datetime...
   installing diffusers...
   installing einops...
   installing facexlib...
   installing ftfy...
   installing gfpgan...
   installing imageio...
   installing imageio-ffmpeg...
   installing kornia...
   installing numpy...
   installing omegaconf...
   installing opencv-python...
   installing pillow...
   installing pytorch-lightning...
   installing torch-fidelity...
   installing transformers...

Setting up Real-ESRGAN:
   downloading pre-trained models...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 63.9M  100 63.9M    0     0  7322k      0  0:00:08  0:00:08 --:--:-- 7761k
   running ESRGAN setup script...

Creating config.txt containing default settings...


All done - don't forget to place your model.ckpt file in this directory! :
stable-diffusion\models\ldm\stable-diffusion-v1

Test : Passed ✔

Final note :

Great job @rbbrdckybk !! 💪🍺🏆
Wishing you all the best for this project and have fun coding ! 😎🐱‍👤

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants