Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Meson build fails due to qhull link issue. #27159

Closed
Jacob-Stevens-Haas opened this issue Oct 20, 2023 · 11 comments · Fixed by #27505
Closed

[Bug]: Meson build fails due to qhull link issue. #27159

Jacob-Stevens-Haas opened this issue Oct 20, 2023 · 11 comments · Fixed by #27505
Labels
Milestone

Comments

@Jacob-Stevens-Haas
Copy link
Contributor

Jacob-Stevens-Haas commented Oct 20, 2023

Bug summary

Building matplotlib fails due to "Connection reset by peer" when meson is building qhull. Perhaps relevant: qhull/qhull#132

Code for reproduction

python -m pip install -ve .

Actual outcome

[Errno 104] Connection reset by peer
~/github/matplotlib$ python -m pip install -ve .
Using pip 22.0.2 from /home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/pip (python 3.10)
Obtaining file:///home/xenophon/github/matplotlib
  Running command pip subprocess to install build dependencies
  Collecting meson-python>=0.13.1
    Using cached meson_python-0.14.0-py3-none-any.whl (76 kB)
  Collecting numpy>=1.25
    Using cached numpy-1.26.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.2 MB)
  Collecting pybind11>=2.6
    Using cached pybind11-2.11.1-py3-none-any.whl (227 kB)
  Collecting setuptools_scm>=7
    Using cached setuptools_scm-8.0.4-py3-none-any.whl (42 kB)
  Collecting pyproject-metadata>=0.7.1
    Using cached pyproject_metadata-0.7.1-py3-none-any.whl (7.4 kB)
  Collecting tomli>=1.0.0
    Using cached tomli-2.0.1-py3-none-any.whl (12 kB)
  Collecting meson>=0.63.3
    Using cached meson-1.2.2-py3-none-any.whl (964 kB)
  Collecting packaging>=20
    Using cached packaging-23.2-py3-none-any.whl (53 kB)
  Collecting typing-extensions
    Using cached typing_extensions-4.8.0-py3-none-any.whl (31 kB)
  Collecting setuptools
    Using cached setuptools-68.2.2-py3-none-any.whl (807 kB)
  Installing collected packages: typing-extensions, tomli, setuptools, pybind11, packaging, numpy, meson, setuptools_scm, pyproject-metadata, meson-python
  Successfully installed meson-1.2.2 meson-python-0.14.0 numpy-1.26.1 packaging-23.2 pybind11-2.11.1 pyproject-metadata-0.7.1 setuptools-68.2.2 setuptools_scm-8.0.4 tomli-2.0.1 typing-extensions-4.8.0
  Installing build dependencies ... done
  Running command Checking if build backend supports build_editable
  Checking if build backend supports build_editable ... done
  Running command Getting requirements to build editable
  Getting requirements to build editable ... done
  Running command pip subprocess to install backend dependencies
  Collecting ninja>=1.8.2
    Using cached ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl (307 kB)
  Collecting patchelf>=0.11.0
    Using cached patchelf-0.17.2.1-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.musllinux_1_1_x86_64.whl (425 kB)
  Installing collected packages: patchelf, ninja
  Successfully installed ninja-1.11.1.1 patchelf-0.17.2.1
  Installing backend dependencies ... done
  Running command Preparing editable metadata (pyproject.toml)
  + meson setup /home/xenophon/github/matplotlib /home/xenophon/github/matplotlib/build/cp310 -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/home/xenophon/github/matplotlib/build/cp310/meson-python-native-file.ini
  The Meson build system
  Version: 1.2.2
  Source dir: /home/xenophon/github/matplotlib
  Build dir: /home/xenophon/github/matplotlib/build/cp310
  Build type: native build
  Project name: matplotlib
  Project version: 3.9.0.dev0
  C compiler for the host machine: cc (gcc 11.4.0 "cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
  C linker for the host machine: cc ld.bfd 2.38
  C++ compiler for the host machine: c++ (gcc 11.4.0 "c++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
  C++ linker for the host machine: c++ ld.bfd 2.38
  Host machine cpu family: x86_64
  Host machine cpu: x86_64
  Program python found: YES (/home/xenophon/github/matplotlib/env/bin/python)
  Did not find pkg-config by name 'pkg-config'
  Found Pkg-config: NO
  Run-time dependency python found: YES 3.10
  pybind11-config found: YES (/tmp/pip-build-env-8wxg444r/overlay/bin/pybind11-config) 2.11.1
  Run-time dependency pybind11 found: YES 2.11.1

  Executing subproject freetype-2.6.1

  freetype-2.6.1| Project name: freetype2
  freetype-2.6.1| Project version: 2.6.1
  freetype-2.6.1| C compiler for the host machine: cc (gcc 11.4.0 "cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
  freetype-2.6.1| C linker for the host machine: cc ld.bfd 2.38
  freetype-2.6.1| Has header "unistd.h" : YES
  freetype-2.6.1| Has header "fcntl.h" : YES
  freetype-2.6.1| Has header "stdint.h" : YES
  freetype-2.6.1| Configuring ftconfig.h using configuration
  freetype-2.6.1| Configuring ftoption.h using configuration
  freetype-2.6.1| Build targets in project: 3
  freetype-2.6.1| Subproject freetype-2.6.1 finished.

  Downloading qhull source from http://www.qhull.org/download/qhull-2020-src-8.0.2.tgz
  WARNING: failed to download with error: [Errno 104] Connection reset by peer. Trying after a delay...
  WARNING: failed to download with error: [Errno 104] Connection reset by peer. Trying after a delay...
  WARNING: failed to download with error: [Errno 104] Connection reset by peer. Trying after a delay...
  WARNING: failed to download with error: [Errno 104] Connection reset by peer. Trying after a delay...
  WARNING: failed to download with error: [Errno 104] Connection reset by peer. Trying after a delay...
  Traceback (most recent call last):
    File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/mesonmain.py", line 194, in run
      return options.run_func(options)
    File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/msetup.py", line 358, in run
      app.generate()
    File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/msetup.py", line 183, in generate
      return self._generate(env, capture, vslite_ctx)
    File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/msetup.py", line 228, in _generate
      intr.run()
    File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreter/interpreter.py", line 3002, in run
      super().run()
    File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 164, in run
      self.evaluate_codeblock(self.ast, start=1)
    File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 190, in evaluate_codeblock
      raise e
    File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 182, in evaluate_codeblock
      self.evaluate_statement(cur)
    File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 196, in evaluate_statement
      return self.function_call(cur)
    File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 521, in function_call
      res = func(node, func_args, kwargs)
    File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/decorators.py", line 277, in wrapper
      return f(*nargs, **wrapped_kwargs)
    File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/decorators.py", line 596, in wrapper
      return f(*wrapped_args, **wrapped_kwargs)
    File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreter/interpreter.py", line 2414, in func_subdir
      self.evaluate_codeblock(codeblock)
    File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 190, in evaluate_codeblock
      raise e
    File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 182, in evaluate_codeblock
      self.evaluate_statement(cur)
    File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 206, in evaluate_statement
      return self.evaluate_if(cur)
    File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 306, in evaluate_if
      self.evaluate_codeblock(node.elseblock)
    File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 190, in evaluate_codeblock
      raise e
    File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 182, in evaluate_codeblock
      self.evaluate_statement(cur)
    File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 198, in evaluate_statement
      self.assignment(cur)
    File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 635, in assignment
      value = self.evaluate_statement(node.value)
    File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 196, in evaluate_statement
      return self.function_call(cur)
    File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 521, in function_call
      res = func(node, func_args, kwargs)
    File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/decorators.py", line 277, in wrapper
      return f(*nargs, **wrapped_kwargs)
    File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreterbase/decorators.py", line 596, in wrapper
      return f(*wrapped_args, **wrapped_kwargs)
    File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreter/interpreter.py", line 879, in func_subproject
      return self.do_subproject(args[0], 'meson', kw)
    File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/interpreter/interpreter.py", line 924, in do_subproject
      subdir = r.resolve(subp_name, method)
    File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/wrap/wrap.py", line 466, in resolve
      self.get_file()
    File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/wrap/wrap.py", line 543, in get_file
      path = self.get_file_internal('source')
    File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/wrap/wrap.py", line 763, in get_file_internal
      self.download(what, cache_path)
    File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/wrap/wrap.py", line 738, in download
      dhash, tmpfile = self.get_data_with_backoff(srcurl)
    File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/wrap/wrap.py", line 731, in get_data_with_backoff
      return self.get_data(urlstring)
    File "/tmp/pip-build-env-8wxg444r/overlay/lib/python3.10/site-packages/mesonbuild/wrap/wrap.py", line 702, in get_data
      block = resp.read(blocksize)
    File "/usr/lib/python3.10/http/client.py", line 466, in read
      s = self.fp.read(amt)
    File "/usr/lib/python3.10/socket.py", line 705, in readinto
      return self._sock.recv_into(b)
  ConnectionResetError: [Errno 104] Connection reset by peer

  ERROR: Unhandled python OSError. This is probably not a Meson bug, but an issue with your build environment.
  error: subprocess-exited-with-error
  
  × Preparing editable metadata (pyproject.toml) did not run successfully.
  │ exit code: 104
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /home/xenophon/github/matplotlib/env/bin/python /home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_editable /tmp/tmp811wdin1
  cwd: /home/xenophon/github/matplotlib
  Preparing editable metadata (pyproject.toml) ... error
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Expected outcome

builds successfully

Additional information

It could be an http vs https issue: the link (and all of qhull's page) is http. Chaning the link/file/hash on qhull.wrap to the equivalent tarball on qhull's github page at least advances the error to

  Downloading qhull source from https://github.com/qhull/qhull/archive/refs/tags/v8.0.2.tar.gz
  Downloading file of unknown size.

  Executing subproject qhull

  qhull| Project name: qhull
  qhull| Project version: 8.0.2
  qhull| C compiler for the host machine: cc (gcc 11.4.0 "cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
  qhull| C linker for the host machine: cc ld.bfd 2.38

  ../../subprojects/qhull-2020.2/meson.build:5:12: ERROR: Include dir src does not exist.

I also tried installing libqhull8.0 and using system libraries, but got the same issue

Operating system

Linux 6.2.0-34-generic; Ubuntu 22.04.1

Matplotlib Version

master from today: dfdb37a

Matplotlib Backend

N/A

Python version

3.10.12

Jupyter version

N/A

Installation

git checkout

@rcomer
Copy link
Member

rcomer commented Oct 20, 2023

Hi @Jacob-Stevens-Haas, since the move to meson, we need a different command to install. This is shown in the development version of the guide:
https://matplotlib.org/devdocs/devel/development_setup.html#install-matplotlib-in-editable-mode

@story645 story645 added the Build label Oct 20, 2023
@ksunden
Copy link
Member

ksunden commented Oct 20, 2023

Your initial error also looks like a (likely temporary) networking error. Perhaps the qhull download servers were offline for a bit.

@Jacob-Stevens-Haas
Copy link
Contributor Author

Jacob-Stevens-Haas commented Oct 23, 2023

Thanks both!

Correcting the Command

since the move to meson, we need a different command to install. This is shown in the development version of the guide

When using the new command from the development guide, I get the ERROR: Include dir src does not exist (The same error as previously when I pointed to the github.com tarball instead of qhull.org)

Error when building with: `python -m pip install --verbose --no-build-isolation --editable .[dev]`
Using pip 22.0.2 from /home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/pip (python 3.10)                                                                                                   
Obtaining file:///home/xenophon/github/matplotlib                                                                                                                                                           
  Running command Checking if build backend supports build_editable                                                                                                                                         
  Checking if build backend supports build_editable ... done                                                                                                                                                
  Running command Preparing editable metadata (pyproject.toml)                                                                                                                                              
  + meson setup /home/xenophon/github/matplotlib /home/xenophon/github/matplotlib/build/cp310 -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/home/xenophon/github/matplotlib/build/cp
310/meson-python-native-file.ini                                                                                                                                                                            
  The Meson build system                                                                                                                                                                                    
  Version: 1.2.2                                                                                                                                                                                            
  Source dir: /home/xenophon/github/matplotlib                                                                                                                                                              
  Build dir: /home/xenophon/github/matplotlib/build/cp310                                                                                                                                                   
  Build type: native build                                                                                                                                                                                  
  Project name: matplotlib                                                                                                                                                                                  
  Project version: 3.9.0.dev0                                                                                                                                                                               
  C compiler for the host machine: cc (gcc 11.4.0 "cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")                                                                                                               
  C linker for the host machine: cc ld.bfd 2.38                                                                                                                                                             
  C++ compiler for the host machine: c++ (gcc 11.4.0 "c++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")                                                                                                           
  C++ linker for the host machine: c++ ld.bfd 2.38                                                                                                                                                          
  Host machine cpu family: x86_64                                                                                                                                                                           
  Host machine cpu: x86_64                                                                                                                                                                                  
  Program python found: YES (/home/xenophon/github/matplotlib/env/bin/python)                                                                                                                               
  Did not find pkg-config by name 'pkg-config'
  Found Pkg-config: NO
  Run-time dependency python found: YES 3.10
  pybind11-config found: YES (/home/xenophon/github/matplotlib/env/bin/pybind11-config) 2.11.1
  Run-time dependency pybind11 found: YES 2.11.1

  Executing subproject freetype-2.6.1

  freetype-2.6.1| Project name: freetype2
  freetype-2.6.1| Project version: 2.6.1
  freetype-2.6.1| C compiler for the host machine: cc (gcc 11.4.0 "cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
  freetype-2.6.1| C linker for the host machine: cc ld.bfd 2.38
  freetype-2.6.1| Has header "unistd.h" : YES
  freetype-2.6.1| Has header "fcntl.h" : YES
  freetype-2.6.1| Has header "stdint.h" : YES                                                                                                                                                               
  freetype-2.6.1| Configuring ftconfig.h using configuration                                                                                                                                                
  freetype-2.6.1| Configuring ftoption.h using configuration                                                                                                                                                
  freetype-2.6.1| Build targets in project: 3                                                                                                                                                               
  freetype-2.6.1| Subproject freetype-2.6.1 finished.                                                                                                                                                       
                                                                                                                                                                                                            
  WARNING: Subproject qhull's revision may be out of date; its wrap file has changed since it was first configured                                                                                          
                                                                                                                                                                                                            
  Executing subproject qhull                                                                                                                                                                                
                                                                                                                                                                                                            
  qhull| Project name: qhull                                                                                                                                                                                
  qhull| Project version: 8.0.2                                                                                                                                                                             
  qhull| C compiler for the host machine: cc (gcc 11.4.0 "cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")                                                                                                        
  qhull| C linker for the host machine: cc ld.bfd 2.38                                                                                                                                                      
                                                                                                                                                                                                            
  ../../subprojects/qhull-2020.2/meson.build:5:12: ERROR: Include dir src does not exist.                                                                                                                   
                                                                                                                                                                                                            
  A full log can be found at /home/xenophon/github/matplotlib/build/cp310/meson-logs/meson-log.txt                                                                                                          
  error: subprocess-exited-with-error                                                                                                                                                                       
                                                                                                                                                                                                            
  × Preparing editable metadata (pyproject.toml) did not run successfully.                                                                                                                                  
  │ exit code: 1                                                                                                                                                                                            
  ╰─> See above for output.                                                                                                                                                                                 
                                                                                                                                                                                                            
  note: This error originates from a subprocess, and is likely not a problem with pip.                                                                                                                      
  full command: /home/xenophon/github/matplotlib/env/bin/python /home/xenophon/github/matplotlib/env/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_ed
itable /tmp/tmp3o0u5ac4
  cwd: /home/xenophon/github/matplotlib     
  Preparing editable metadata (pyproject.toml) ... error                                      
error: metadata-generation-failed               

× Encountered error while generating package metadata.
╰─> See above for output.
                                                   
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.  

The same error occurs even if I replace the wrap file url/filename/hash with the one from github.

Using system libraries

Using the correct command with system Qhull works. Thanks! Details: upgraded from pip 22.2 to pip 23.3, apt install libqhull-dev libqhull8.0

Still not able to download qhull manually

Perhaps the qhull download servers were offline for a bit.

They're not offline, as wget receives a partial download (see above linked issue), but they do seem to be having trouble (several days later). For sanity's sake, is anyone else able to download http://www.qhull.org/download/qhull-2020-src-8.0.2.tgz in any way? Even so, is it perhaps better to use github's link, which uses HTTPS, versus qhull's?

@tacaswell
Copy link
Member

if you do git clean -xfd in your Matplotlib checkout and then try again does it work? My guess is that there may be a partial set of files which is enough to skip re-downloading / unpacking but not enough to work. When in doubt wipe it clean and try again 🔨 💣 .

@ksunden
Copy link
Member

ksunden commented Oct 23, 2023

I'm not too concerned about https, because we do checksum the download (sha256) so having ssl during download doesn't provide much additional safety, actually.

I am able to download the file (though admittedly firefox gives me a warning about https when I click the link from github) (I was also able to do git clean -xfd and reinstall today).

@Jacob-Stevens-Haas
Copy link
Contributor Author

Oh, that's a good point about the checksum. Today it did download (with the mentioned security risk). It might have just been an issue over the weekend.

@tacaswell

This comment was marked as outdated.

@QuLogic
Copy link
Member

QuLogic commented Oct 24, 2023

Meson does the downloading now, and that directory is not used any more. Tarballs are cached in subprojects/packagecache.

@tacaswell
Copy link
Member

oh, good to know!

Hid my incorrect comment to not confuse people in the future.

@Jacob-Stevens-Haas
Copy link
Contributor Author

Jacob-Stevens-Haas commented Dec 6, 2023

While I had been able to get around the issue in the past by using system libraries, I came back to work on a a new laptop at a new workplace and ran into the same issue of being unable to download from qhull servers. A desk neighbor mentioned that they too had had the same problem building matplotlib to make a PR in the past (#27158).

I had assumed I was the only developer trying to work on matplotlib that ran into the qhull issue. But randomly meeting someone else, in-person, only a month later makes me think that a lot of would-be developers on academic/corporate networks could be getting stopped or burdened by this issue.

My colleague believed it was because IT's firewall felt something qhull, perhaps because it was HTTP and not HTTPS. OTOH, the authors of qhull mentioned that the qhull server didn't support range requests, in case that had been the issue.

I have a branch moving the build to qhull's github-hosted 8.0.2 tarball. My feeling is that github is generally a more stable place for code than a project like qhull's website. Would that be a welcome PR?

@tacaswell
Copy link
Member

I think moving to use the GH hosted tarball is reasonable (so we stop hammering the qhull server).

When we were doing the downloading I think the tarballs were included in the directories that we had CI restore from cache.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants