Install grumpy-tools before building grumpy-runtime

[abitrolly]

pip2 install grumpy-runtime fails with the ERROR
#113 (comment)
because it relies on grumpy-tools.

grumpy-tools packages is specified in setup.py as a
dependency, and it doesn't work, because for the build process to
succeed, the dependency should already exist at the time setup.py
is executed.

The official workaround is pyproject.toml from PEP-518
pypa/pip#2381

[abitrolly]

This still doesn't fix the error. pip claims that it installs build dependencies.

$ pipenv clean
$ pipenv run python setup.py sdist
...
copying pyproject.toml -> grumpy-runtime-0.3.0
...
$  pipenv run pip install dist/grumpy-runtime-0.3.0.tar.gz 
Processing ./dist/grumpy-runtime-0.3.0.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting grumpy-tools>=0.3.0 (from grumpy-runtime==0.3.0)
...

But then it still fails with error.

...
Building wheels for collected packages: grumpy-runtime
  Building wheel for grumpy-runtime (PEP 517) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/bin/python /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmpRJ3HDO                                                                                                                   
       cwd: /tmp/pip-req-build-DJvU2j                                                                                                                                   
  Complete output (757 lines):                                                                                                                                          
  running bdist_wheel                                                                                                                                                   
  running build                                                                                                                                                         
...
      from grumpy_tools import cli                                                                                                                                      
  ImportError: No module named grumpy_tools                                                                                                                             
  Traceback (most recent call last):                                                                                                                                    
    File "/tmp/pip-req-build-DJvU2j/build/bin/pydeps", line 5, in <module>                                                                                              
      from grumpy_tools import cli                                                                                                                                      
  ImportError: No module named grumpy_tools                                                                                                                             
  Re-executing[2]: make run --debug=bjm -r                                                                                                                              
  GNU Make 4.2.1                                                                                                                                                        
  Built for x86_64-redhat-linux-gnu                                            
...
      Successfully remade target file 'build/src/grumpy/type.go'.                                                                                                       
    Must remake target 'build/pkg/linux_amd64/grumpy.a'.                                                                                                                
    Successfully remade target file 'build/pkg/linux_amd64/grumpy.a'.                                                                                                   
     File 'build/pkg/linux_amd64/__python__/os.a' does not exist.                                                                                                       
       File 'build/src/__python__/os/module.go' does not exist.                                                                                                         
      Must remake target 'build/src/__python__/os/module.go'.                                                                                                           
  Traceback (most recent call last):                                                                                                                                    
    File "/tmp/pip-req-build-DJvU2j/build/bin/grumpc", line 5, in <module>                                                                                              
      from grumpy_tools import cli                                                                                                                                      
  ImportError: No module named grumpy_tools                                                                                                                             
  make: *** [build/stdlib.mk:2181: build/src/__python__/os/module.go] Error 1                                                                                           
  Traceback (most recent call last):                                                                                                                                    
    File "/home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py", line 207, in <module>                  
      main()                                                                                                                                                            
...
    File "/usr/lib64/python2.7/subprocess.py", line 190, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['echo "print \'Make Runtime Success\'" | make run --debug=bjm -r']' returned non-zero exit status 2
  ----------------------------------------
  ERROR: Failed building wheel for grumpy-runtime
  Running setup.py clean for grumpy-runtime
Failed to build grumpy-runtime
ERROR: Could not build wheels for grumpy-runtime which use PEP 517 and cannot be installed directly

[alanjds]

Thanks for your contribution.

The tests are not passing with Go=stable. Would you mind do also fix this before the merge?

[abitrolly]

Looks like Python build environment that is set for setup.py is not passed by subprocess to make. Or perhaps make strips the environment. Could also be the problem with shebangs pypa/setuptools#847

[abitrolly]

sys.path in setup.py

  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib/python2.7/site-packages/pip/_vendor/pep517
  /tmp/pip-build-env-OE7ua4/site
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib/python27.zip
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7/plat-linux2
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7/lib-tk
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7/lib-old
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7/lib-dynload
  /usr/lib64/python2.7
  /usr/lib/python2.7
  /tmp/pip-build-env-OE7ua4/overlay/lib/python2.7/site-packages
  /tmp/pip-build-env-OE7ua4/overlay/lib64/python2.7/site-packages
  /tmp/pip-build-env-OE7ua4/normal/lib/python2.7/site-packages
  /tmp/pip-build-env-OE7ua4/normal/lib64/python2.7/site-packages

sys.path in subprocess.check_call with shell=True.

  /tmp/pip-req-build-nK1nBr/build/bin
  /tmp/pip-req-build-nK1nBr/build/lib/python2.7/site-packages
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib/python27.zip
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7/plat-linux2
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7/lib-tk
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7/lib-old
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7/lib-dynload
  /usr/lib64/python2.7
  /usr/lib/python2.7
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib/python2.7/site-packages

/tmp/pip-build-env-OE7ua4 with installed grumpy-tools from build-system requires is gone, and /tmp/pip-req-build-nK1nBr without them is present.

[abitrolly]

Reported the issue upstream in setuptools. Because build environment is not a proper virtualenv or venv, and setup.py is executed inprocess, the sys.path is not passed to subprocess and it picks up parent pip environment.

The workaround for now is to serialize and pass sys.path to subprocess explicitly as PYTHONPATH environment variable. It will probably mix with virtualenv from pip and if/when upstream fixes that, the workaround should be removed.

[abitrolly]

Passing sys.path as PYTHONPATH fails.

  Reading makefiles...                                                             
  Traceback (most recent call last):                                               
    File "/home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7/site.py", line 791, in <module>                                                      
      main()                                                                       
    File "/home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7/site.py", line 781, in main                                                          
      execsitecustomize()                                                          
    File "/home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7/site.py", line 553, in execsitecustomize                                             
      import sitecustomize                                                         
    File "/tmp/pip-build-env-FWZ6LX/site/sitecustomize.py", line 22, in <module>   
      assert not path in sys.path                              

[abitrolly]

Trying to forcefully inject just a single build path /tmp/pip-build-env-tqb_4I/normal/lib64/python2.7/site-packages doesn't work. It passed as PYTHONPATH to make, but not available when pydeps is run.

Maybe parent vritualenv is killing it.

[abitrolly]

@alanjds ready for review. Install works ok.

$ git clone -b fix-install git@github.com:abitrolly/grumpy.git
$ cd grumpy/grumpy-runtime-src
$ pipenv --python 2.7
$ pipenv run pip install .

[alanjds]

Thanks @abitrolly. I bet this was a pain to tackle.

[abitrolly]

Took quite a bit a time, indeed.