Build works with PYTHON="python" but fails with explicit PYTHON="/path/to/python"

[marius311]

As per the title, this works

$ which python
/global/homes/m/marius/.conda/envs/local/bin/python
$ PYTHON="python" julia -e 'using Pkg; Pkg.build("PyCall")'
  Building Conda ─→ `~/.julia/packages/Conda/kLXeC/deps/build.log`
  Building PyCall → `~/.julia/packages/PyCall/ttONZ/deps/build.log`
$ cat ~/.julia/packages/PyCall/ttONZ/deps/build.log
[ Info: PyCall is using python (Python 3.6.9) at /global/homes/m/marius/.conda/envs/local/bin/python, libpython = /global/u1/m/marius/.conda/envs/local/lib/libpython3.6m.so.1.0

but putting the path explicitly fails:

$ PYTHON=$(which python) julia -e 'using Pkg; Pkg.build("PyCall")'
  Building Conda ─→ `~/.julia/packages/Conda/kLXeC/deps/build.log`
  Building PyCall → `~/.julia/packages/PyCall/ttONZ/deps/build.log`
┌ Error: Error building `PyCall`: 

with the log attached below. Any ideas what might be happening? I'm unable to reproduce on other systems. This is Julia v1.2, Conda v1.3.0, PyCall v1.91.2.

PREFIX=/global/homes/m/marius/.conda/envs/local
Unpacking payload ...
Collecting package metadata (current_repodata.json): ...working... done                                   
Solving environment: ...working... failed
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed

PackagesNotFoundError: The following packages are not available from current channels:

  - libffi==3.2.1=hd88cf55_4 -> libgcc-ng=7.2.0
  - libffi==3.2.1=hd88cf55_4 -> libstdcxx-ng=7.2.0
  - idna==2.8=py37_0 -> python[version='>=3.7,<3.8.0a0'] -> libffi[version='>=3.2.1,<4.0a0'] -> libgcc-ng=7.2.0
  - idna==2.8=py37_0 -> python[version='>=3.7,<3.8.0a0'] -> libffi[version='>=3.2.1,<4.0a0'] -> libstdcxx-ng=7.2.0
  - python==3.7.3=h0371630_0 -> libffi[version='>=3.2.1,<4.0a0'] -> libgcc-ng=7.2.0
  - python==3.7.3=h0371630_0 -> libffi[version='>=3.2.1,<4.0a0'] -> libstdcxx-ng=7.2.0
  - pyopenssl==19.0.0=py37_0 -> cryptography[version='>=2.2.1'] -> asn1crypto[version='>=0.21.0'] -> python[version='>=3.7,<3.8.0a0'] -> libffi[version='>=3.2.1,<4.0a0'] -> libgcc-ng=7.2.0
  - pyopenssl==19.0.0=py37_0 -> cryptography[version='>=2.2.1'] -> asn1crypto[version='>=0.21.0'] -> python[version='>=3.7,<3.8.0a0'] -> libffi[version='>=3.2.1,<4.0a0'] -> libstdcxx-ng=7.2.0
  - pyopenssl==19.0.0=py37_0 -> cryptography[version='>=2.2.1'] -> cffi[version='>=1.7'] -> pycparser
  - pyopenssl==19.0.0=py37_0 -> cryptography[version='>=2.2.1'] -> idna[version='>=2.1']
  - pyopenssl==19.0.0=py37_0 -> cryptography[version='>=2.2.1'] -> six[version='>=1.4.1']
  - setuptools==41.0.1=py37_0 -> certifi[version='>=2016.09'] -> python[version='>=3.6,<3.7.0a0'] -> libffi[version='>=3.2.1,<4.0a0'] -> libgcc-ng=7.2.0
  - setuptools==41.0.1=py37_0 -> certifi[version='>=2016.09'] -> python[version='>=3.6,<3.7.0a0'] -> libffi[version='>=3.2.1,<4.0a0'] -> libstdcxx-ng=7.2.0
  - setuptools==41.0.1=py37_0 -> certifi[version='>=2016.09'] -> python[version='>=3.6,<3.7.0a0'] -> pip -> wheel
  - tqdm==4.32.1=py_0 -> python -> libffi[version='>=3.2.1,<4.0a0'] -> libgcc-ng=7.2.0
  - tqdm==4.32.1=py_0 -> python -> libffi[version='>=3.2.1,<4.0a0'] -> libstdcxx-ng=7.2.0
  - tqdm==4.32.1=py_0 -> python -> pip -> setuptools -> certifi[version='>=2016.09']
  - tqdm==4.32.1=py_0 -> python -> pip -> wheel
  - conda==4.7.10=py37_0 -> conda-package-handling[version='>=1.3.0'] -> python[version='>=3.7,<3.8.0a0'] -> libffi[version='>=3.2.1,<4.0a0'] -> libgcc-ng=7.2.0
  - conda==4.7.10=py37_0 -> conda-package-handling[version='>=1.3.0'] -> python[version='>=3.7,<3.8.0a0'] -> libffi[version='>=3.2.1,<4.0a0'] -> libstdcxx-ng=7.2.0
  - conda==4.7.10=py37_0 -> conda-package-handling[version='>=1.3.0'] -> python-libarchive-c
  - conda==4.7.10=py37_0 -> conda-package-handling[version='>=1.3.0'] -> six
  - conda==4.7.10=py37_0 -> conda-package-handling[version='>=1.3.0'] -> tqdm
  - conda==4.7.10=py37_0 -> pycosat[version='>=0.6.3']
  - conda==4.7.10=py37_0 -> pyopenssl[version='>=16.2.0'] -> cryptography[version='>=2.2.1'] -> asn1crypto[version='>=0.21.0']
  - conda==4.7.10=py37_0 -> pyopenssl[version='>=16.2.0'] -> cryptography[version='>=2.2.1'] -> cffi[version='>=1.7'] -> pycparser
  - conda==4.7.10=py37_0 -> pyopenssl[version='>=16.2.0'] -> cryptography[version='>=2.2.1'] -> idna[version='>=2.1']
  - conda==4.7.10=py37_0 -> requests[version='>=2.18.4,<3'] -> certifi[version='>=2017.4.17']
  - conda==4.7.10=py37_0 -> requests[version='>=2.18.4,<3'] -> chardet[version='>=3.0.2,<3.1.0']
  - conda==4.7.10=py37_0 -> requests[version='>=2.18.4,<3'] -> urllib3[version='>=1.21.1,<1.25'] -> pysocks[version='>=1.5.6,<2.0,!=1.5.7']
  - conda==4.7.10=py37_0 -> ruamel_yaml[version='>=0.11.14,<0.16']
  - pip==19.1.1=py37_0 -> python[version='>=3.7,<3.8.0a0'] -> libffi[version='>=3.2.1,<4.0a0'] -> libgcc-ng=7.2.0
  - pip==19.1.1=py37_0 -> python[version='>=3.7,<3.8.0a0'] -> libffi[version='>=3.2.1,<4.0a0'] -> libstdcxx-ng=7.2.0
  - pip==19.1.1=py37_0 -> setuptools -> certifi[version='>=2016.09']
  - pip==19.1.1=py37_0 -> wheel
  - cryptography==2.7=py37h1ba5d50_0 -> asn1crypto[version='>=0.21.0'] -> python[version='>=3.7,<3.8.0a0'] -> libffi[version='>=3.2.1,<4.0a0'] -> libgcc-ng=7.2.0
  - cryptography==2.7=py37h1ba5d50_0 -> asn1crypto[version='>=0.21.0'] -> python[version='>=3.7,<3.8.0a0'] -> libffi[version='>=3.2.1,<4.0a0'] -> libstdcxx-ng=7.2.0
  - cryptography==2.7=py37h1ba5d50_0 -> cffi[version='>=1.7'] -> pycparser
  - cryptography==2.7=py37h1ba5d50_0 -> idna[version='>=2.1']
  - cryptography==2.7=py37h1ba5d50_0 -> six[version='>=1.4.1']
  - requests==2.22.0=py37_0 -> certifi[version='>=2017.4.17'] -> python[version='>=3.6,<3.7.0a0'] -> libffi[version='>=3.2.1,<4.0a0'] -> libgcc-ng=7.2.0
  - requests==2.22.0=py37_0 -> certifi[version='>=2017.4.17'] -> python[version='>=3.6,<3.7.0a0'] -> libffi[version='>=3.2.1,<4.0a0'] -> libstdcxx-ng=7.2.0
  - requests==2.22.0=py37_0 -> certifi[version='>=2017.4.17'] -> python[version='>=3.6,<3.7.0a0'] -> pip -> setuptools
  - requests==2.22.0=py37_0 -> certifi[version='>=2017.4.17'] -> python[version='>=3.6,<3.7.0a0'] -> pip -> wheel
  - requests==2.22.0=py37_0 -> chardet[version='>=3.0.2,<3.1.0']
  - requests==2.22.0=py37_0 -> idna[version='>=2.5,<2.9']
  - requests==2.22.0=py37_0 -> urllib3[version='>=1.21.1,<1.25'] -> cryptography[version='>=1.3.4'] -> asn1crypto[version='>=0.21.0']
  - requests==2.22.0=py37_0 -> urllib3[version='>=1.21.1,<1.25'] -> cryptography[version='>=1.3.4'] -> cffi[version='>=1.7'] -> pycparser
  - requests==2.22.0=py37_0 -> urllib3[version='>=1.21.1,<1.25'] -> cryptography[version='>=1.3.4'] -> six[version='>=1.4.1']
  - requests==2.22.0=py37_0 -> urllib3[version='>=1.21.1,<1.25'] -> pyopenssl[version='>=0.14']
  - requests==2.22.0=py37_0 -> urllib3[version='>=1.21.1,<1.25'] -> pysocks[version='>=1.5.6,<2.0,!=1.5.7']
  - ruamel_yaml==0.15.46=py37h14c3975_0 -> python[version='>=3.7,<3.8.0a0'] -> libffi[version='>=3.2.1,<4.0a0'] -> libstdcxx-ng=7.2.0
  - chardet==3.0.4=py37_1 -> python[version='>=3.7,<3.8.0a0'] -> libffi[version='>=3.2.1,<4.0a0'] -> libgcc-ng=7.2.0
  - chardet==3.0.4=py37_1 -> python[version='>=3.7,<3.8.0a0'] -> libffi[version='>=3.2.1,<4.0a0'] -> libstdcxx-ng=7.2.0
  - pycosat==0.6.3=py37h14c3975_0 -> python[version='>=3.7,<3.8.0a0'] -> libffi[version='>=3.2.1,<4.0a0'] -> libstdcxx-ng=7.2.0
  - conda-package-handling==1.3.11=py37_0 -> python[version='>=3.7,<3.8.0a0'] -> libffi[version='>=3.2.1,<4.0a0'] -> libgcc-ng=7.2.0
  - conda-package-handling==1.3.11=py37_0 -> python[version='>=3.7,<3.8.0a0'] -> libffi[version='>=3.2.1,<4.0a0'] -> libstdcxx-ng=7.2.0
  - conda-package-handling==1.3.11=py37_0 -> python-libarchive-c
  - conda-package-handling==1.3.11=py37_0 -> six
  - conda-package-handling==1.3.11=py37_0 -> tqdm
  - python-libarchive-c==2.8=py37_11 -> python[version='>=3.7,<3.8.0a0'] -> libffi[version='>=3.2.1,<4.0a0'] -> libgcc-ng=7.2.0
  - python-libarchive-c==2.8=py37_11 -> python[version='>=3.7,<3.8.0a0'] -> libffi[version='>=3.2.1,<4.0a0'] -> libstdcxx-ng=7.2.0
  - six==1.12.0=py37_0 -> python[version='>=3.7,<3.8.0a0'] -> libffi[version='>=3.2.1,<4.0a0'] -> libgcc-ng=7.2.0
  - six==1.12.0=py37_0 -> python[version='>=3.7,<3.8.0a0'] -> libffi[version='>=3.2.1,<4.0a0'] -> libstdcxx-ng=7.2.0
  - pysocks==1.7.0=py37_0 -> python[version='>=3.7,<3.8.0a0'] -> libffi[version='>=3.2.1,<4.0a0'] -> libgcc-ng=7.2.0
  - pysocks==1.7.0=py37_0 -> python[version='>=3.7,<3.8.0a0'] -> libffi[version='>=3.2.1,<4.0a0'] -> libstdcxx-ng=7.2.0
  - asn1crypto==0.24.0=py37_0 -> python[version='>=3.7,<3.8.0a0'] -> libffi[version='>=3.2.1,<4.0a0'] -> libgcc-ng=7.2.0
  - asn1crypto==0.24.0=py37_0 -> python[version='>=3.7,<3.8.0a0'] -> libffi[version='>=3.2.1,<4.0a0'] -> libstdcxx-ng=7.2.0
  - pycparser==2.19=py37_0 -> python[version='>=3.7,<3.8.0a0'] -> libffi[version='>=3.2.1,<4.0a0'] -> libgcc-ng=7.2.0
  - pycparser==2.19=py37_0 -> python[version='>=3.7,<3.8.0a0'] -> libffi[version='>=3.2.1,<4.0a0'] -> libstdcxx-ng=7.2.0
  - certifi==2019.6.16=py37_0 -> python[version='>=3.7,<3.8.0a0'] -> libffi[version='>=3.2.1,<4.0a0'] -> libgcc-ng=7.2.0
  - certifi==2019.6.16=py37_0 -> python[version='>=3.7,<3.8.0a0'] -> libffi[version='>=3.2.1,<4.0a0'] -> libstdcxx-ng=7.2.0
  - wheel==0.33.4=py37_0 -> python[version='>=3.7,<3.8.0a0'] -> libffi[version='>=3.2.1,<4.0a0'] -> libgcc-ng=7.2.0
  - wheel==0.33.4=py37_0 -> python[version='>=3.7,<3.8.0a0'] -> libffi[version='>=3.2.1,<4.0a0'] -> libstdcxx-ng=7.2.0
  - wheel==0.33.4=py37_0 -> setuptools -> certifi[version='>=2016.09']
  - urllib3==1.24.2=py37_0 -> certifi -> python[version='>=3.6,<3.7.0a0'] -> libffi[version='>=3.2.1,<4.0a0'] -> libgcc-ng=7.2.0
  - urllib3==1.24.2=py37_0 -> certifi -> python[version='>=3.6,<3.7.0a0'] -> libffi[version='>=3.2.1,<4.0a0'] -> libstdcxx-ng=7.2.0
  - urllib3==1.24.2=py37_0 -> certifi -> python[version='>=3.6,<3.7.0a0'] -> pip -> setuptools
  - urllib3==1.24.2=py37_0 -> certifi -> python[version='>=3.6,<3.7.0a0'] -> pip -> wheel
  - urllib3==1.24.2=py37_0 -> cryptography[version='>=1.3.4'] -> asn1crypto[version='>=0.21.0']
  - urllib3==1.24.2=py37_0 -> cryptography[version='>=1.3.4'] -> cffi[version='>=1.7'] -> pycparser
  - urllib3==1.24.2=py37_0 -> cryptography[version='>=1.3.4'] -> idna[version='>=2.1']
  - urllib3==1.24.2=py37_0 -> cryptography[version='>=1.3.4'] -> six[version='>=1.4.1']
  - urllib3==1.24.2=py37_0 -> pyopenssl[version='>=0.14']
  - urllib3==1.24.2=py37_0 -> pysocks[version='>=1.5.6,<2.0,!=1.5.7']
  - cffi==1.12.3=py37h2e261b9_0 -> libffi[version='>=3.2.1,<4.0a0'] -> libgcc-ng=7.2.0
  - cffi==1.12.3=py37h2e261b9_0 -> libffi[version='>=3.2.1,<4.0a0'] -> libstdcxx-ng=7.2.0
  - cffi==1.12.3=py37h2e261b9_0 -> pycparser -> python[version='>=3.6,<3.7.0a0'] -> pip -> setuptools -> certifi[version='>=2016.09']
  - cffi==1.12.3=py37h2e261b9_0 -> pycparser -> python[version='>=3.6,<3.7.0a0'] -> pip -> wheel

Current channels:

  - https://repo.anaconda.com/pkgs/main/linux-64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/linux-64
  - https://repo.anaconda.com/pkgs/r/noarch
  - https://repo.anaconda.com/pkgs/pro/linux-64
  - https://repo.anaconda.com/pkgs/pro/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.


[ Info: Downloading miniconda installer ...
[ Info: Installing miniconda ...
ERROR: LoadError: failed process: Process(`/global/homes/m/marius/.conda/envs/local/installer.sh -b -f -p /global/homes/m/marius/.conda/envs/local`, ProcessExited(1)) [1]

Stacktrace:
 [1] pipeline_error at ./process.jl:813 [inlined]
 [2] #run#536(::Bool, ::typeof(run), ::Cmd) at ./process.jl:728
 [3] run at ./process.jl:726 [inlined]
 [4] _install_conda(::String, ::Bool) at /global/homes/m/marius/.julia/packages/Conda/kLXeC/src/Conda.jl:165
 [5] _install_conda(::String) at /global/homes/m/marius/.julia/packages/Conda/kLXeC/src/Conda.jl:152
 [6] runconda(::Cmd, ::String) at /global/homes/m/marius/.julia/packages/Conda/kLXeC/src/Conda.jl:111
 [7] #add#1(::String, ::typeof(Conda.add), ::String, ::String) at /global/homes/m/marius/.julia/packages/Conda/kLXeC/src/Conda.jl:184
 [8] add at /global/homes/m/marius/.julia/packages/Conda/kLXeC/src/Conda.jl:183 [inlined] (repeats 2 times)
 [9] top-level scope at /global/homes/m/marius/.julia/packages/PyCall/ttONZ/deps/build.jl:84
 [10] include at ./boot.jl:328 [inlined]
 [11] include_relative(::Module, ::String) at ./loading.jl:1105
 [12] include(::Module, ::String) at ./Base.jl:31
 [13] include(::String) at ./client.jl:431
 [14] top-level scope at none:5
in expression starting at /global/homes/m/marius/.julia/packages/PyCall/ttONZ/deps/build.jl:43

[tkf]

The stacktrace says that the failure happens at Conda.add("numpy") in

PyCall.jl/deps/build.jl

Lines 82 to 85 in cf01bf0

	use_conda = dirname(python) == abspath(Conda.PYTHONDIR)
	if use_conda
	Conda.add("numpy")
	end

Maybe related to these issues?
JuliaPy/Conda.jl#146
JuliaPy/Conda.jl#145

[marius311]

Thanks, yea I think its JuliaPy/Conda.jl#146 because on this system I'm using a non-root environment, and indeed if I run the code snippet in the first comment there, its "still trying to install miniconda". I'll close this in favor of that. Let me know if you know of any workarounds until that might be fixed.

[marius311]

Or maybe I should leave this open? I suppose unrelated to that, its a little weird on the PyCall side for there to be any difference if you specify python explicitly or not, if the paths resolve to the same thing in the end? I'll leave it up to you guys.

[tkf]

Yeah, I think use_conda should never be true if python is coming from ENV["PYTHON"] in principle.

But I don't feel like fixing it ATM personally (fixing Conda.jl has higher priority) so I don't mind if it is closed again.