pip3 install mfem --install-option="--with-parallel" not working

[benzwick]

When I install PyMFEM using pip3 install mfem --install-option="--with-parallel" --verbose I cannot import mfem.ser or mfem.par:

In [1]: import mfem.ser
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-1-d6ad452c785f> in <module>
----> 1 import mfem.ser

ModuleNotFoundError: No module named 'mfem.ser'

In [2]: import mfem.par
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-2-6f8793407d7f> in <module>
----> 1 import mfem.par

ModuleNotFoundError: No module named 'mfem.par'

Despite the installation appearing to be successful:

...
  byte-compiling /home/ben/.local/local/lib/python3.10/dist-packages/mfem/_par/communication.py to communication.cpython-310.pyc
  byte-compiling /home/ben/.local/local/lib/python3.10/dist-packages/mfem/_par/fe_nurbs.py to fe_nurbs.cpython-310.pyc
  byte-compiling /home/ben/.local/local/lib/python3.10/dist-packages/mfem/_par/datacollection.py to datacollection.cpython-310.pyc
  byte-compiling /home/ben/.local/local/lib/python3.10/dist-packages/mfem/_par/gslib.py to gslib.cpython-310.pyc
  byte-compiling /home/ben/.local/local/lib/python3.10/dist-packages/mfem/_par/eltrans.py to eltrans.cpython-310.pyc
  byte-compiling /home/ben/.local/local/lib/python3.10/dist-packages/mfem/__sha__.py to __sha__.cpython-310.pyc
  running install_egg_info
  running egg_info
  writing mfem.egg-info/PKG-INFO
  writing dependency_links to mfem.egg-info/dependency_links.txt
  deleting mfem.egg-info/requires.txt
  writing top-level names to mfem.egg-info/top_level.txt
  reading manifest file 'mfem.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no files found matching 'external/README.txt'
  adding license file 'LICENSE'
  writing manifest file 'mfem.egg-info/SOURCES.txt'
  removing '/home/ben/.local/local/lib/python3.10/dist-packages/mfem-4.4.0.1-py3.10.egg-info' (and everything under it)
  Copying mfem.egg-info to /home/ben/.local/local/lib/python3.10/dist-packages/mfem-4.4.0.1-py3.10.egg-info
  running install_scripts
  writing list of installed files to '/tmp/pip-record-9fs919kq/install-record.txt'
  Running setup.py install for mfem ... done
Successfully installed mfem

[benzwick]

Maybe the problem is that the files are put here:

/home/ben/.local/local/lib/python3.10/dist-packages/mfem

instead of here:

/home/ben/.local/lib/python3.10/dist-packages/mfem

i.e., there is an additional local subdirectory in the path so Python cannot find these.

See these output lines:

  byte-compiling /home/ben/.local/local/lib/python3.10/dist-packages/mfem/_par/communication.py to communication.cpython-310.pyc

Also, these files:

/home/ben/.local/local/lib/python3.10/dist-packages/mfem/ser.py
/home/ben/.local/local/lib/python3.10/dist-packages/mfem/par.py

are in a different location than

/home/ben/.local/lib/python3.10/site-packages/mfem/external

which appears to be in the correct location.

[sshiraiwa]

@benzwick , thank you for reporting this. This is a bit puzzling, since it looks like mfem is installed under /home/ben/.local/local/lib/python3.10. In my environment, it goes under xxxx/.local/lib/python.....
For example, I just tried two cases and I got

byte-compiling /home/shiraiwa/.local/lib/python3.8/site-packages/mfem/_par/fespace.py to fespace.cpython-38.pyc (when using my user account)
byte-compiling /root/.local/lib/python3.10/site-packages/mfem/_par/setup.py to setup.cpython-310.pyc (when using docker)

When neither --prefix nor --user is given, this function (https://github.com/mfem/PyMFEM/blob/master/setup.py#L221) looks for a place to install. What is your sys.prefix and site.getusersitepackages()? You may somehow need to give --user option?

[benzwick]

I tried with --prefix=~/.local option but it didn't change anything. As a temporary solution, I copied the files to the correct location and it seems to work.

In [1]: import sys

In [2]: sys.prefix
Out[2]: '/usr'

In [3]: import site

In [4]: site.getusersitepackages()
Out[4]: '/home/ben/.local/lib/python3.10/site-packages'

[benzwick]

I deleted all the existing PyMFEM installations and tried again but now I get this error instead and cannot install the parallel version at all:

$ pip3 install mfem --install-option="--with-parallel" --install-option="--mfem-branch=master" --no-cache-dir --verbose
WARNING: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
Using pip 22.2.2 from /home/ben/.local/lib/python3.10/site-packages/pip (python 3.10)
Defaulting to user installation because normal site-packages is not writeable
Collecting mfem
  Downloading mfem-4.4.0.1.tar.gz (401 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 401.2/401.2 kB 1.8 MB/s eta 0:00:00
  Running command python setup.py egg_info
  running egg_info
  creating /tmp/pip-pip-egg-info-3m984k_c/mfem.egg-info
  writing /tmp/pip-pip-egg-info-3m984k_c/mfem.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-pip-egg-info-3m984k_c/mfem.egg-info/dependency_links.txt
  writing top-level names to /tmp/pip-pip-egg-info-3m984k_c/mfem.egg-info/top_level.txt
  writing manifest file '/tmp/pip-pip-egg-info-3m984k_c/mfem.egg-info/SOURCES.txt'
  reading manifest file '/tmp/pip-pip-egg-info-3m984k_c/mfem.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no files found matching 'mfem/*/*.cxx'
  warning: no files found matching 'mfem/*/*.h'
  warning: no files found matching 'external/README.txt'
  warning: no previously-included files found matching 'mfem/*/*.so'
  adding license file 'LICENSE'
  writing manifest file '/tmp/pip-pip-egg-info-3m984k_c/mfem.egg-info/SOURCES.txt'
  Preparing metadata (setup.py) ... done
Skipping wheel build for mfem, due to binaries being disabled for it.
Installing collected packages: mfem
  Running command Running setup.py install for mfem
  running install
  /home/ben/.local/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
    warnings.warn(
  Traceback (most recent call last):
    File "<string>", line 2, in <module>
    File "<pip-setuptools-caller>", line 34, in <module>
    File "/tmp/pip-install-567i8pva/mfem_5eda993984ac425f8c8ea3ac1a21f08f/setup.py", line 1764, in <module>
      main()
    File "/tmp/pip-install-567i8pva/mfem_5eda993984ac425f8c8ea3ac1a21f08f/setup.py", line 1760, in main
      run_setup()
    File "/tmp/pip-install-567i8pva/mfem_5eda993984ac425f8c8ea3ac1a21f08f/setup.py", line 1748, in run_setup
      setup(
    File "/home/ben/.local/lib/python3.10/site-packages/setuptools/__init__.py", line 153, in setup
      return distutils.core.setup(**attrs)
    File "/usr/lib/python3.10/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/lib/python3.10/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/usr/lib/python3.10/distutils/dist.py", line 983, in run_command
      cmd_obj = self.get_command_obj(command)
    File "/usr/lib/python3.10/distutils/dist.py", line 858, in get_command_obj
      cmd_obj = self.command_obj[command] = klass(self)
    File "/home/ben/.local/lib/python3.10/site-packages/setuptools/__init__.py", line 172, in __init__
      _Command.__init__(self, dist)
    File "/usr/lib/python3.10/distutils/cmd.py", line 62, in __init__
      self.initialize_options()
    File "/tmp/pip-install-567i8pva/mfem_5eda993984ac425f8c8ea3ac1a21f08f/setup.py", line 1442, in initialize_options
      _install.initialize_options(self)
    File "/home/ben/.local/lib/python3.10/site-packages/setuptools/command/install.py", line 40, in initialize_options
      orig.install.initialize_options(self)
    File "/usr/lib/python3.10/_distutils_system_mod.py", line 33, in initialize_options
      super().initialize_options()
  TypeError: super(type, obj): obj must be an instance or subtype of type
  error: subprocess-exited-with-error
  
  × Running setup.py install for mfem did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /usr/bin/python3 -u -c '
  exec(compile('"'"''"'"''"'"'
  # This is <pip-setuptools-caller> -- a caller that pip uses to run setup.py
  #
  # - It imports setuptools before invoking setup.py, to enable projects that directly
  #   import from `distutils.core` to work with newer packaging standards.
  # - It provides a clear error message when setuptools is not installed.
  # - It sets `sys.argv[0]` to the underlying `setup.py`, when invoking `setup.py` so
  #   setuptools doesn'"'"'t think the script is `-c`. This avoids the following warning:
  #     manifest_maker: standard file '"'"'-c'"'"' not found".
  # - It generates a shim setup.py, for handling setup.cfg-only projects.
  import os, sys, tokenize
  
  try:
      import setuptools
  except ImportError as error:
      print(
          "ERROR: Can not execute `setup.py` since setuptools is not available in "
          "the build environment.",
          file=sys.stderr,
      )
      sys.exit(1)
  
  __file__ = %r
  sys.argv[0] = __file__
  
  if os.path.exists(__file__):
      filename = __file__
      with tokenize.open(__file__) as f:
          setup_py_code = f.read()
  else:
      filename = "<auto-generated setuptools caller>"
      setup_py_code = "from setuptools import setup; setup()"
  
  exec(compile(setup_py_code, filename, "exec"))
  '"'"''"'"''"'"' % ('"'"'/tmp/pip-install-567i8pva/mfem_5eda993984ac425f8c8ea3ac1a21f08f/setup.py'"'"',), "<pip-setuptools-caller>", "exec"))' install --record /tmp/pip-record-60vjt9ov/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /home/ben/.local/include/python3.10/mfem --with-parallel --mfem-branch=master
  cwd: /tmp/pip-install-567i8pva/mfem_5eda993984ac425f8c8ea3ac1a21f08f/
  Running setup.py install for mfem ... error
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> mfem

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

[sshiraiwa]

Hmmm, I can not reproduce this.
Reading this line number 172, could it be possible that your setuptools is slightly old?

setuptools/__init__.py", line 172, in __init__
      _Command.__init__(self, dist)

Also, what is your environment? I see _distutils_system_mod.py in you error message. But I don't have it either my virtualenv or docker.

[benzwick]

I am using Python 3.10 on Ubuntu 22.04.

It looks like setuptools may have been downgraded when I installed numba:

$ pip3 install --upgrade numba
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: numba in /usr/lib/python3/dist-packages (0.55.1)
Collecting numba
  Downloading numba-0.56.2-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.5/3.5 MB 5.2 MB/s eta 0:00:00
Requirement already satisfied: numpy<1.24,>=1.18 in /usr/lib/python3/dist-packages (from numba) (1.21.5)
Collecting setuptools<60
  Downloading setuptools-59.8.0-py3-none-any.whl (952 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 952.8/952.8 kB 5.7 MB/s eta 0:00:00
Collecting llvmlite<0.40,>=0.39.0dev0
  Downloading llvmlite-0.39.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.6/34.6 MB 4.8 MB/s eta 0:00:00
Installing collected packages: setuptools, llvmlite, numba
  Attempting uninstall: setuptools
    Found existing installation: setuptools 65.4.1
    Uninstalling setuptools-65.4.1:
      Successfully uninstalled setuptools-65.4.1
Successfully installed llvmlite-0.39.1 numba-0.56.2 setuptools-59.8.0

$ pip install --upgrade setuptools
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: setuptools in /home/ben/.local/lib/python3.10/site-packages (59.8.0)
Collecting setuptools
  Using cached setuptools-65.4.1-py3-none-any.whl (1.2 MB)
Installing collected packages: setuptools
  Attempting uninstall: setuptools
    Found existing installation: setuptools 59.8.0
    Uninstalling setuptools-59.8.0:
      Successfully uninstalled setuptools-59.8.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
numba 0.56.2 requires setuptools<60, but you have setuptools 65.4.1 which is incompatible.
Successfully installed setuptools-65.4.1

With the upgraded setuptools I no longer get the above error and the installation is working again with this command:

pip3 install mfem --user --install-option="--with-parallel" --install-option="--mfem-branch=master" --no-cache-dir --verbose

However, the files are still installed in the wrong place as before. It works after I manually move these directories:

/home/ben/.local/local/lib/python3.10/dist-packages/mfem
/home/ben/.local/local/lib/python3.10/dist-packages/mfem-4.4.0.1-py3.10.egg-info

to here:

/home/ben/.local/lib/python3.10/site-packages/mfem
/home/ben/.local/lib/python3.10/site-packages/mfem-4.4.0.1-py3.10.egg-info

Maybe it is worth putting a requirement for the version of setuptools that is required for pymfem.

[sshiraiwa]

Closing this for now. Please feel free to re-open this if the error persist.
A new version (4.5.0.1) should not have this issue.