20221105: pytest is failing because it cannot find tools module

[kloczek]

I'm packaging your module as an rpm package so I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

• python3 -sBm build -w --no-isolation
• because I'm calling build with --no-isolation I'm using during all processes only locally installed modules
• install .whl file in </install/prefix> using 'installer` module
• run pytest with $PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>
• build is performed in env which is cut off from access to the public network (pytest is executed with -m "not network")

Here is pytest output:

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-pdfminer-20221105-7.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-pdfminer-20221105-7.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra -m 'not network'
==================================================================================== test session starts ====================================================================================
platform linux -- Python 3.8.17, pytest-7.4.0, pluggy-1.0.0
benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/tkloczko/rpmbuild/BUILD/pdfminer.six-20221105
plugins: hypothesis-6.75.3, benchmark-4.0.0, timeout-2.1.0
collected 118 items / 2 errors

========================================================================================== ERRORS ===========================================================================================
_______________________________________________________________________ ERROR collecting tests/test_tools_dumppdf.py ________________________________________________________________________
ImportError while importing test module '/home/tkloczko/rpmbuild/BUILD/pdfminer.six-20221105/tests/test_tools_dumppdf.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib64/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_tools_dumppdf.py:7: in <module>
    from tools import dumppdf
E   ModuleNotFoundError: No module named 'tools'
_______________________________________________________________________ ERROR collecting tests/test_tools_pdf2txt.py ________________________________________________________________________
ImportError while importing test module '/home/tkloczko/rpmbuild/BUILD/pdfminer.six-20221105/tests/test_tools_pdf2txt.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib64/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_tools_pdf2txt.py:6: in <module>
    import tools.pdf2txt as pdf2txt
E   ModuleNotFoundError: No module named 'tools'
================================================================================== short test summary info ==================================================================================
ERROR tests/test_tools_dumppdf.py
ERROR tests/test_tools_pdf2txt.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
===================================================================================== 2 errors in 0.37s =====================================================================================

Here is list of installed modules in build env

Package                       Version
----------------------------- ---------
alabaster                     0.7.13
asciidoc                      10.2.0
asttokens                     2.2.1
attrs                         23.1.0
Babel                         2.12.1
backcall                      0.2.0
beautifulsoup4                4.12.2
black                         23.3.0
build                         0.10.0
cffi                          1.15.1
charset-normalizer            3.1.0
click                         8.1.3
codespell                     2.2.5
cryptography                  40.0.2
decorator                     5.1.1
distro                        1.8.0
docutils                      0.19
exceptiongroup                1.1.1
executing                     1.2.0
gpg                           1.20.0
html2text                     2020.1.16
hypothesis                    6.75.3
idna                          3.4
imagesize                     1.4.1
importlib-metadata            6.7.0
iniconfig                     2.0.0
installer                     0.7.0
ipython                       8.12.0
isodate                       0.6.1
isort                         5.12.0
jedi                          0.18.2
Jinja2                        3.1.2
libcomps                      0.1.19
lxml                          4.9.3
Markdown                      3.4.1
MarkupSafe                    2.1.2
matplotlib-inline             0.1.6
maturin                       1.1.0
metaextract                   1.0.9
mypy-extensions               1.0.0
packaging                     23.1
parso                         0.8.3
pathspec                      0.11.1
pexpect                       4.8.0
pickleshare                   0.7.5
platformdirs                  3.8.1
pluggy                        1.0.0
ply                           3.11
prompt-toolkit                3.0.38
ptyprocess                    0.7.0
pure-eval                     0.2.2
py-cpuinfo                    9.0.0
py2pack                       0.8.7
pycparser                     2.21
Pygments                      2.15.1
pyparsing                     3.1.0
pypi-search                   1.2.1
pyproject_hooks               1.0.0
pytest                        7.4.0
pytest-benchmark              4.0.0
pytest-timeout                2.1.0
python-dateutil               2.8.2
pytz                          2023.2
PyYAML                        6.0
rdflib                        6.4.0a0
requests                      2.31.0
setuptools                    68.0.0
setuptools-scm                7.1.0
six                           1.16.0
snowballstemmer               2.2.0
sortedcontainers              2.4.0
soupsieve                     2.4.1
Sphinx                        6.2.1
sphinx-argparse               0.4.0
sphinxcontrib-applehelp       1.0.4
sphinxcontrib-devhelp         1.0.2
sphinxcontrib-htmlhelp        2.0.0
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.3
sphinxcontrib-serializinghtml 1.1.5
stack-data                    0.6.2
toml                          0.10.2
tomli                         2.0.1
traitlets                     5.9.0
typing_extensions             4.7.1
urllib3                       1.26.15
wcwidth                       0.2.6
wheel                         0.40.0
yargs                         0.8.1
zipp                          3.15.0

[pietermarsman]

Probably the best solution here is to move everything except the entrypoint into the pdfminer package. And then only test things from the pdfminer package.

Another solution is to call pdf2txt in a subprocess. And not import pdf2txt from the tools directory solely to parse the arguments.

The tools directory/package has been around for a long time so people probably depend on it. But we can make it as slim as possible.