distutils.errors.DistutilsOptionError: must supply either home or prefix/exec-prefix -- not both

[rouge8]

With shiv 0.0.14 from PyPI:

$ shiv aws -c aws -p $(which python3.6) -o blergh
 shiv! 🔪
Collecting aws
Collecting fabric>=1.6 (from aws)
Collecting boto (from aws)
  Using cached https://files.pythonhosted.org/packages/bd/b7/a88a67002b1185ed9a8e8a6ef15266728c2361fcb4f1d02ea331e4c7741d/boto-2.48.0-py2.py3-none-any.whl
Collecting prettytable>=0.7 (from aws)
Collecting paramiko<3.0,>=1.10 (from fabric>=1.6->aws)
  Using cached https://files.pythonhosted.org/packages/3e/db/cb7b6656e0e7387637ce850689084dc0b94b44df31cc52e5fc5c2c4fd2c1/paramiko-2.4.1-py2.py3-none-any.whl
Collecting pyasn1>=0.1.7 (from paramiko<3.0,>=1.10->fabric>=1.6->aws)
  Using cached https://files.pythonhosted.org/packages/ba/fe/02e3e2ee243966b143657fb8bd6bc97595841163b6d8c26820944acaec4d/pyasn1-0.4.2-py2.py3-none-any.whl
Collecting pynacl>=1.0.1 (from paramiko<3.0,>=1.10->fabric>=1.6->aws)
  Using cached https://files.pythonhosted.org/packages/74/8e/a6c0d340972d9e2f1a405aaa3f2460950b4c0337f92db0291a4355974529/PyNaCl-1.2.1-cp36-cp36m-macosx_10_6_intel.whl
Collecting bcrypt>=3.1.3 (from paramiko<3.0,>=1.10->fabric>=1.6->aws)
  Using cached https://files.pythonhosted.org/packages/7e/59/d48fd712941da1a5d6490964a37bb3de2e526965b6766273f6a7049ee590/bcrypt-3.1.4-cp36-cp36m-macosx_10_6_intel.whl
Collecting cryptography>=1.5 (from paramiko<3.0,>=1.10->fabric>=1.6->aws)
  Using cached https://files.pythonhosted.org/packages/40/87/acdcf84ce6d25a7db1c113f4b9b614fd8d707b7ab56fbf17cf18cd26a627/cryptography-2.2.2-cp34-abi3-macosx_10_6_intel.whl
Collecting cffi>=1.4.1 (from pynacl>=1.0.1->paramiko<3.0,>=1.10->fabric>=1.6->aws)
  Using cached https://files.pythonhosted.org/packages/8e/be/40b1bc2c3221acdefeb9dab6773d43cda7543ed0d8c8df8768f05af2d01e/cffi-1.11.5-cp36-cp36m-macosx_10_6_intel.whl
Collecting six (from pynacl>=1.0.1->paramiko<3.0,>=1.10->fabric>=1.6->aws)
  Using cached https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Collecting idna>=2.1 (from cryptography>=1.5->paramiko<3.0,>=1.10->fabric>=1.6->aws)
  Using cached https://files.pythonhosted.org/packages/27/cc/6dd9a3869f15c2edfab863b992838277279ce92663d334df9ecf5106f5c6/idna-2.6-py2.py3-none-any.whl
Collecting asn1crypto>=0.21.0 (from cryptography>=1.5->paramiko<3.0,>=1.10->fabric>=1.6->aws)
  Using cached https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl
Collecting pycparser (from cffi>=1.4.1->pynacl>=1.0.1->paramiko<3.0,>=1.10->fabric>=1.6->aws)
Installing collected packages: pyasn1, pycparser, cffi, six, pynacl, bcrypt, idna, asn1crypto, cryptography, paramiko, fabric, boto, prettytable, aws
Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/basecommand.py", line 228, in main
    status = self.run(options, args)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 335, in run
    use_user_site=options.use_user_site,
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/req/__init__.py", line 49, in install_given_reqs
    **kwargs
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 748, in install
    use_user_site=use_user_site, pycompile=pycompile,
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 961, in move_wheel_files
    warn_script_location=warn_script_location,
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/wheel.py", line 216, in move_wheel_files
    prefix=prefix,
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/locations.py", line 165, in distutils_scheme
    i.finalize_options()
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/command/install.py", line 248, in finalize_options
    "must supply either home or prefix/exec-prefix -- not both")
distutils.errors.DistutilsOptionError: must supply either home or prefix/exec-prefix -- not both

Pip install failed!

Here's the packages installed systemwide alongside shiv:

click==6.7
importlib-resources==0.5
pip==10.0.1
setuptools==39.1.0
shiv==0.0.14
wheel==0.31.0

OS X, Python 3.6.5 from Homebrew.

[lorencarvalho]

Hi @rouge8,

Looks to be an known issue with Homebrew'd python: https://github.com/Homebrew/brew/blob/master/docs/Homebrew-and-Python.md#note-on-pip-install---user

Note that shiv is not using the --user command, but it does use --target which suffers the same fate.

You can see that plain old pip install exhibits the problem:

lcarvalh-mn1 ~ ❱❱❱ /usr/local/Cellar/python/3.6.3/bin/pip install --target temp_dir aws --no-deps
Collecting aws
  Using cached https://files.pythonhosted.org/packages/90/56/d088e68699e4f00881d957584e26854a88d0009864ef614c6f567807fc2f/aws-0.2.5.tar.gz
Skipping bdist_wheel for aws, due to binaries being disabled for it.
Installing collected packages: aws
  Running setup.py install for aws ... done
Exception:
Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pip/commands/install.py", line 350, in run
    isolated=options.isolated_mode,
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pip/commands/install.py", line 436, in get_lib_location_guesses
    scheme = distutils_scheme('', *args, **kwargs)
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pip/locations.py", line 153, in distutils_scheme
    i.finalize_options()
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/setuptools/command/install.py", line 38, in finalize_options
    orig.install.finalize_options(self)
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/command/install.py", line 248, in finalize_options
    "must supply either home or prefix/exec-prefix -- not both")
distutils.errors.DistutilsOptionError: must supply either home or prefix/exec-prefix -- not both

I tried one of the workarounds I found while googling around and it seemed to fix the problem:

lcarvalh-mn1 ~ ❱❱❱ cat << EOF > ~/.pydistutils.cfg
> [install]
> prefix=
> EOF
lcarvalh-mn1 ~ ❱❱❱ /usr/local/Cellar/python/3.6.3/bin/pip3 install --target temp_dir aws --no-deps
Collecting aws
Installing collected packages: aws
Successfully installed aws-0.2.5

However, you might still run into issues because that package (aws) appears to only support Python2 😞

lcarvalh-mn1 ~ ❱❱❱ shiv aws -c aws -p /usr/local/Cellar/python/3.6.3/bin/python3 -o blergh -q
lcarvalh-mn1 ~ ❱❱❱ ./blergh
Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "./blergh/__main__.py", line 3, in <module>
  File "./blergh/_bootstrap/__init__.py", line 102, in bootstrap
  File "./blergh/_bootstrap/__init__.py", line 21, in import_string
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 674, in exec_module
  File "<frozen importlib._bootstrap_external>", line 781, in get_code
  File "<frozen importlib._bootstrap_external>", line 741, in source_to_code
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/aws-0.2.5-py3.6.egg/aws/main.py", line 23
    print '%(name)s: %(endpoint)s' % {
                                 ^
SyntaxError: invalid syntax

[rouge8]

However, you might still run into issues because that package (aws) appears to only support Python2 😞

Ha, I meant to do awscli, but ran into that issue with a few packages. I'm guessing anything where a wheel is unavailable is affected.

I wonder if there's a workaround that can go into shiv because this is probably pretty common... I'll see if I can make anything work without needing a ~/.pydistutils.cf 😕

[lorencarvalho]

At first I tried using --install-option="--prefix=" because any args not processed by shiv itself get delegated to pip, so in theory that should've worked! But it didn't seem to make a difference 😞. The other downside to --install-option is that is basically disables the use of wheels which makes installation super slow.

I'm definitely open to an elegant solution to this, but from perusing how this file is handled by distutils and how it's included (by homebrew folks) I'm not hopeful. I'll keep poking at it.

[rouge8]

I think the fix is actually pretty easy -- distutils will also read from a setup.cfg in the current directory, so you can create that and run the pip install from that directory!

This is a minimal diff that seems to work:

diff --git a/src/shiv/cli.py b/src/shiv/cli.py
index d95075d..dbae8e1 100644
--- a/src/shiv/cli.py
+++ b/src/shiv/cli.py
@@ -1,4 +1,5 @@
 import importlib_resources  # type: ignore
+import os
 import shutil
 import sys
 import uuid
@@ -125,10 +126,13 @@ def main(
     interpreter = validate_interpreter(python)
 
     with TemporaryDirectory() as working_path:
+        with Path(working_path, "setup.cfg").open('w') as f:
+            f.write('[install]\nprefix=')
         site_packages = Path(working_path, "site-packages")
         site_packages.mkdir(parents=True, exist_ok=True)
 
         # install deps into staged site-packages
+        os.chdir(working_path)
         pip.install(
             python or sys.executable,
             ["--target", site_packages.as_posix()] + list(pip_args),

[rouge8]

Turned that into a PR: #18

[lorencarvalho]

Reopening per conversation in #22 & #18

[rouge8]

I think the reason #18 didn't actually work is pip changes to the directory of the package its installing when it's building from source.

For whoever tries this next: make sure to set PIP_NO_CACHE_DIR=off in your environment when verifying that a fix actually works 😭

[rouge8]

It sounds like what we want is for pypa/pip#4557 to be merged, but in the meantime could probably follow the same approach as Azure/azure-cli#4466