python modules not working in python 3.10

[xobs]

Describe the bug

The OpenLane project contains scripts that require external python modules. For example, lefutil.py requires a module called click.

openroad includes a python interpreter, but does not appear to include support for installing 3rd party modules. As a result, openlane fails to run:

$ openroad -exit -no_init -python /opt/Si/OpenLane/scripts/odbpy/lefutil.py get_metal_layers -o /opt/Si/work/inverter/runs/RUN_2022.11.21_13.35.35/tmp/layers.list /opt/Si/work/caravel_example/pdks//sky130A/libs.ref/sky130_fd_sc_hd/techlef/sky130_fd_sc_hd.tlef
OpenROAD v2.0-5616-g7f4902a7b
This program is licensed under the BSD-3 license. See the LICENSE file for details.
Components of this program may be licensed under more restrictive licenses which must be honored.
Traceback (most recent call last):
  File "/opt/Si/OpenLane/scripts/odbpy/lefutil.py", line 15, in <module>
    import click
ModuleNotFoundError: No module named 'click'
$

The build steps at https://github.com/The-OpenROAD-Project/OpenROAD#build do not document how to install these modules, and since the interactive interpreter does not support pip it is unclear how to proceed.

Expected Behavior

Openroad should be usable with the scripts from openlane

Environment

Git commit: d1b69464717b4b93e568051688bc99d2fc16fdeb
kernel: Linux 5.15.74.2-microsoft-standard-WSL2
os: Ubuntu 22.04.1 LTS (Jammy Jellyfish)
cmake version 3.24.1
gcc (Ubuntu 11.2.0-19ubuntu1) 11.2.0
Ubuntu clang version 14.0.0-1ubuntu1

To Reproduce

Run the following:

openroad -exit -no_init -python ${OPENLANE_PATH}/scripts/odbpy/lefutil.py

Relevant log output

No response

Screenshots

No response

Additional Context

I have tried the following, none of which have worked:

1. Run openroad from inside of a python venv
2. Copy the lib directory from the venv to /opt/Si/ which is the prefix used by openroad
3. Run openroad from the venv directory

[xobs]

I did discover that manually setting PYTHONPATH does get it to work. Perhaps that should be documented somewhere?

$ PYTHONPATH=$VIRTUAL_ENV/lib/python3.10/site-packages/ openroad -python
OpenROAD v2.0-5790-ge6f8378ab
This program is licensed under the BSD-3 license. See the LICENSE file for details.
Components of this program may be licensed under more restrictive licenses which must be honored.
[WARNING ORD-0039] .openroad ignored with -python
Python 3.10.6 (main, Aug 10 2022, 11:40:04) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import click
>>>

[maliberty]

@proppy any thoughts?

[proppy]

@xobs I think #1424 is ultimatly the right way to solve this. In the meantime the workaround you're pointing sounds appropriate!

[xobs]

For reference, I'm trying to understand the design flow better which is very difficult to do with a Docker container. This was the last piece I had to solve in order to get it to work.

I've succeeded in hardening the inverter sample project using the following invocation:

 PYTHONPATH=$VIRTUAL_ENV/lib/python3.10/site-packages/ \
 STD_CELL_LIBRARY_OPT=sky130_fd_sc_hd \
 STD_CELL_LIBRARY=sky130_fd_sc_hd \
 PDK_ROOT=/opt/Si/PDKs/share/pdk \
 PDK=sky130B \
 ./flow.tcl \
 -design /opt/Si/work/inverter/ \
 -ignore_mismatches

[proppy]

I'm trying to understand the design flow better which is very difficult to do with a Docker container.

Did you gave the conda-eda installation a try?
https://github.com/chipsalliance/silicon-notebooks/blob/main/digital-inverter-openlane.ipynb

This shouldn't require any container, can should give you an easy to get an environment bootstrapped, also see hdl/conda-eda#247

[xobs]

I did see that, and I've been referring to that quite a bit when setting things up. It's been super helpful.

I couldn't see where the PDK files were sourced from, which is what led me to start from there.

[maliberty]

@proppy do you understand why it wouldn't import the package in OR?

[maliberty]

What is sys.path in python3 versus OR? I just tried it locally and they are the same.

[proppy]

@maliberty it must be because the system python lib directory doesn't get added to the module search path.

This could be fixed by calling https://docs.python.org/3/c-api/init.html#c.Py_SetPath before init, but you'd have to detect it from the host python environment.

[xobs]

I'm not very well-versed in the ways of Python, but I thought the general rule was to avoid using system python. Besides, users may not have the rights to install packages in the root.

That's the reason why I went with a venv, and why the VIRTUAL_ENV path was set.

It seems as though the build process ignores any sort of venv and will use the system python regardless of whether there is another interpreter in the path. I just did a clean build of openroad after making sure I had the venv activated, and it still wouldn't work without having PYTHONPATH manually set.

[maliberty]

@proppy from https://docs.python.org/3/glossary.html#term-import-path

A list of locations (or [path entries](https://docs.python.org/3/glossary.html#term-path-entry)) that are searched by the [path based finder](https://docs.python.org/3/glossary.html#term-path-based-finder) for modules to import. During import, this list of locations usually comes from [sys.path](https://docs.python.org/3/library/sys.html#sys.path), but for subpackages it may also come from the parent package’s __path__ attribute.

so it does seem to come from sys.path.

[proppy]

It seems as though the build process ignores any sort of venv and will use the system python regardless of whether there is another interpreter in the path

Small precision, openroad is its own python interpreter, so having a different python in PATH wouldn't really make a difference. PYTHONPATH seems like the right way to discover the venv site-packages.

so it does seem to come from sys.path

Yes but in that case I believe @xobs has installed additional packages in a venv, which does not seem discoverable by the embedded python interpreter unless provided with PYTHONPATH explicitly.

@xobs can you check if setting PYTHONHOME to the venv path helps, maybe we need to forward it explicitly to the embedded interpreter using https://docs.python.org/3/c-api/init.html#c.Py_SetPythonHome

[maliberty]

@proppy however venv is passing that to the regular python interpreter should work the same for OR.
I'm assuming that you can import click into python3.

[xobs]

@xobs can you check if setting PYTHONHOME to the venv path helps, maybe we need to forward it explicitly to the embedded interpreter using https://docs.python.org/3/c-api/init.html#c.Py_SetPythonHome

Setting PYTHONHOME is a good thought, but it makes things even less happy:

$ PYTHONHOME=$VIRTUAL_ENV openroad -python
OpenROAD v2.0-5790-ge6f8378ab
This program is licensed under the BSD-3 license. See the LICENSE file for details.
Components of this program may be licensed under more restrictive licenses which must be honored.
[WARNING ORD-0039] .openroad ignored with -python
Python path configuration:
  PYTHONHOME = '/opt/Si/pyenv'
  PYTHONPATH = (not set)
  program name = 'python3'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/opt/Si/pyenv/bin/python3'
  sys.base_prefix = '/opt/Si/pyenv'
  sys.base_exec_prefix = '/opt/Si/pyenv'
  sys.platlibdir = 'lib'
  sys.executable = '/opt/Si/pyenv/bin/python3'
  sys.prefix = '/opt/Si/pyenv'
  sys.exec_prefix = '/opt/Si/pyenv'
  sys.path = [
    '/opt/Si/pyenv/lib/python310.zip',
    '/opt/Si/pyenv/lib/python3.10',
    '/opt/Si/pyenv/lib/python3.10/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007fd8591f7080 (most recent call first):
  <no Python frame>

[proppy]

however venv is passing that to the regular python interpreter should work the same for OR

by default venv create a symlink to the system python:

1 🍙 ls -al env/bin/python3
lrwxrwxrwx 1 proppy proppy 16 Nov 23 01:50 env/bin/python3 -> /usr/bin/python3

The default interpreter has a quite a bit logic in it, to resolve the default import path depending of the location executable/symlinks, the presence of a pyvenv.cfg and the include-system-site-packages option, see:
https://github.com/python/cpython/blob/c2102136be569e6fc8ed90181f229b46d07142f8/Modules/getpath.py#L69
https://github.com/python/cpython/blob/c2102136be569e6fc8ed90181f229b46d07142f8/Lib/site.py#L16

In the case of openroad since we embed the interpreter, my initial guess was that none of things that compute default import path relative to the prefix/exec_prefix would really apply, but https://docs.python.org/3.10/c-api/intro.html#embedding-python seems to implies that some "guesswork" is being done wrt to the system python path:

On most systems (in particular, on Unix and Windows, although the details are slightly different), Py_Initialize() calculates the module search path based upon its best guess for the location of the standard Python interpreter executable, assuming that the Python library is found in a fixed location relative to the Python interpreter executable. In particular, it looks for a directory named lib/pythonX.Y relative to the parent directory where the executable named python is found on the shell command search path (the environment variable PATH).

https://docs.python.org/3/library/sys_path_init.html#embedded-python

[openroadie]

Have we considered some sort of virtual environment management tool? It is very problematic to mess with the system python or install libraries there. Two possible tools to use:

https://python-poetry.org/docs/
OR
https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html

I personally prefer poetry because it is pretty straightforward to figure out and once it has created a "lock" file with all the dependencies... installation is a breeze. Also trivial to maintain multiple environments if your machine does other things than OpenRoad.

[proppy]

@openroadie we do have openroad packages for conda, see https://anaconda.org/LiteX-Hub/openroad/ but this problem is a bit different as it refers to the python context embedded in the OpenROAD app executable (which would be different from the system Python).

Since will eventually becomes moot once openroad starts shipping as a proper python packages (see #2308, #1812, #1424 and #2311)

[xobs]

Incidentally, I'm experiencing this same issue in Conda:

$ python -c 'import sys;print(sys.path);import click' 
['', '/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64/lib/python310.zip', '/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64/lib/python3.10', '/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64/lib/python3.10/lib-dynload', '/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64/lib/python3.10/site-packages']

$ openroad -python -c 'import sys;print(sys.path);import click'
OpenROAD 477bf6f24a6aec95fd1a2ac4d14f703b08738c5b
This program is licensed under the BSD-3 license. See the LICENSE file for details.
Components of this program may be licensed under more restrictive licenses which must be honored.
[WARNING ORD-0039] .openroad ignored with -python
['', '/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64/lib/python310.zip', '/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64/lib/python3.10', '/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64/lib/python3.10/lib-dynload']
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'click'
>>>

Somehow, site-packages isn't added to the paths.

Similar to venvs, I can get it to work if I run:

export PYTHONPATH=$CONDA_PREFIX/lib/python3.10/site-packages

[proppy]

export PYTHONPATH=$CONDA_PREFIX/lib/python3.10/site-packages

That should already happen if you activate the conda environment.

[xobs]

I've updated the bug to reflect what I believe is an issue with Python 3.10. It may be an issue for any version of Python after 3.7.

This may be related to python/cpython#78189 where it seems somewhat questionable to call Py_Main() after calling Py_Initialize().

As an example, here is what happens if I use a venv on Linux with Python 3.10:

(pyenv) user@Ondo:/opt/Si/pyenv$ openroad -python
OpenROAD v2.0-6773-g74d2fb645
This program is licensed under the BSD-3 license. See the LICENSE file for details.
Components of this program may be licensed under more restrictive licenses which must be honored.
[WARNING ORD-0039] .openroad ignored with -python
Python 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import click
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'click'
>>> import sys
>>> sys.path
['', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload']
>>> import site; from importlib import reload; reload(site)
<module 'site' from '/usr/lib/python3.10/site.py'>
>>> sys.path
['/opt/Si/pyenv', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/opt/Si/pyenv/lib/python3.10/site-packages']
>>> import click
>>>

You can see that reloading site correctly picked up site-packages.

In Conda on macos I see the exact same thing, where site-packages is not included unless I do a reload():

bash-3.2$ openroad -python
OpenROAD 477bf6f24a6aec95fd1a2ac4d14f703b08738c5b
This program is licensed under the BSD-3 license. See the LICENSE file for details.
Components of this program may be licensed under more restrictive licenses which must be honored.
[WARNING ORD-0039] .openroad ignored with -python
Python 3.10.9 (main, Jan 11 2023, 09:20:18) [Clang 14.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys; print(sys.path)
['', '/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64/lib/python310.zip', '/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64/lib/python3.10', '/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64/lib/python3.10/lib-dynload']
>>> import site; from importlib import reload; reload(site); print(sys.path)
<module 'site' from '/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64/lib/python3.10/site.py'>
['/opt/homebrew/Caskroom/miniconda/base/envs/eda/conda-bld', '/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64/lib/python310.zip', '/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64/lib/python3.10', '/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64/lib/python3.10/lib-dynload', '/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64/lib/python3.10/site-packages']
>>>

When running Python by itself, the path is correct:

bash-3.2$ python
Python 3.10.9 (main, Jan 11 2023, 09:18:18) [Clang 14.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys; print(sys.path)
['', '/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64/lib/python310.zip', '/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64/lib/python3.10', '/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64/lib/python3.10/lib-dynload', '/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64/lib/python3.10/site-packages']
>>>

HOWEVER, when running python -S, the path exactly matches the path given by openroad:

bash-3.2$ python -S
Python 3.10.9 (main, Jan 11 2023, 09:18:18) [Clang 14.0.6 ] on darwin
>>> import sys; print(sys.path)
['', '/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64/lib/python310.zip', '/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64/lib/python3.10', '/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64/lib/python3.10/lib-dynload']
>>>

If I patch site.py I see it running when doing openroad -python. I see it correctly set sys.path. Then the path is thrown away.

I beleive what is happening is this:

1. openroad runs Py_Initialize()
2. python evaluates site.py and sets sys.path
3. openroad runs Py_Main()
4. Py_Main() reinitializes the environment, throwing away the work done in (2)

This is probably an issue in newer pythons, but I can't be sure.

[xobs]

Here's an example showing that sys.path is just thrown away:

bash-3.2$ openroad -python print-site.py -exit
OpenROAD 477bf6f24a6aec95fd1a2ac4d14f703b08738c5b
This program is licensed under the BSD-3 license. See the LICENSE file for details.
Components of this program may be licensed under more restrictive licenses which must be honored.
[WARNING ORD-0039] .openroad ignored with -python
in site.py! sys.path: ['/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64/lib/python310.zip', '/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64/lib/python3.10', '/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64/lib/python3.10/lib-dynload', '/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64/lib/python3.10/site-packages']
in openroad! sys.path: ['/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64', '/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64/lib/python310.zip', '/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64/lib/python3.10', '/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64/lib/python3.10/lib-dynload']
bash-3.2$ openroad -python print-site.py
OpenROAD 477bf6f24a6aec95fd1a2ac4d14f703b08738c5b
This program is licensed under the BSD-3 license. See the LICENSE file for details.
Components of this program may be licensed under more restrictive licenses which must be honored.
[WARNING ORD-0039] .openroad ignored with -python
in site.py! sys.path: ['/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64/lib/python310.zip', '/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64/lib/python3.10', '/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64/lib/python3.10/lib-dynload', '/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64/lib/python3.10/site-packages']
in openroad! sys.path: ['/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64', '/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64/lib/python310.zip', '/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64/lib/python3.10', '/opt/homebrew/Caskroom/miniconda/base/envs/eda-arm64/lib/python3.10/lib-dynload']
>>>

[proppy]

We might be able to get python to reuse the existing initialized environment by using https://docs.python.org/3/c-api/init_config.html#c.Py_RunMain on >= 3.8.