tests don't run through the test framework

[yurivict]

Every test runs fine individually, but when they are called through Python, through the function ctest_runner - each one hangs forever.

Version: 1.7
Python 3.9
FreeBSD 13.1 STABLE

[loriab]

The ctest_runner functions are designed to be collected and called by pytest.

[yurivict]

Yes. They hang from pytest, and they also hang individually.

[loriab]

Ok, that sounds like a problem then. I'm not sure how to help debugging. From the CI, they run from source, they run from conda packages, they run on Linux, Mac, and Windows; so I think the broad setup is sound. You might try seeing if the "api" tests (tests/pytests/) run differently than the "cli" tests tests/*/input.dat .

[yurivict]

"api" tests fail with the timeout error:

==================================================================================== test session starts =====================================================================================
platform freebsd13 -- Python 3.9.15, pytest-7.2.0, pluggy-1.0.0
Using --randomly-seed=3542811706
rootdir: /disk-samsung/freebsd-ports/science/psi4/work/psi4-1.7, configfile: pytest.ini
plugins: cov-2.9.0, randomly-3.12.0, timeout-2.1.0, rerunfailures-10.1, flaky-3.7.0, xdist-2.5.0, env-0.6.2, freezegun-0.4.2, mock-1.10.4, forked-1.4.0, hypothesis-6.56.2
collected 1337 items / 19 errors                                                                                                                                                             

=========================================================================================== ERRORS ===========================================================================================
________________________________________________________________________ ERROR collecting tests/pytests/test_adcc.py _________________________________________________________________________
test_adcc.py:9: in <module>
    from addons import using
addons.py:9: in <module>
    from qcengine.testing import _programs as _programs_qcng
/usr/local/lib/python3.9/site-packages/qcengine/testing.py:173: in <module>
    "psi4": is_program_new_enough("psi4", "1.2"),
/usr/local/lib/python3.9/site-packages/qcengine/testing.py:62: in is_program_new_enough
    candidate_version = qcng.get_program(program).get_version()
/usr/local/lib/python3.9/site-packages/qcengine/programs/psi4.py:90: in get_version
    exc["proc"].wait(timeout=30)
/usr/local/lib/python3.9/subprocess.py:1189: in wait
    return self._wait(timeout=timeout)
/usr/local/lib/python3.9/subprocess.py:1909: in _wait
    raise TimeoutExpired(self.args, timeout)
E   subprocess.TimeoutExpired: Command '['/usr/local/bin/psi4', '--version']' timed out after 30 seconds

I think that most tests themselves pass, but there's something wrong with the python test harness.

[loriab]

If there's any chance you haven't got tags, run "pull tags" from https://psicode.org/psi4manual/master/manage_git.html#how-to-create-and-remove-an-annotated-git-tag-on-a-remote and re-make. then make sure that qcengine info (cmdline) is finding psi in the expected place and version.

[yurivict]

psi4 --version prints correct version.

qcengine info fails:

$ qcengine info psi4
Traceback (most recent call last):
  File "/usr/local/bin/qcengine", line 33, in <module>
    sys.exit(load_entry_point('qcengine==0.26.0', 'console_scripts', 'qcengine')())
  File "/usr/local/bin/qcengine", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/local/lib/python3.9/importlib/metadata.py", line 86, in load
    module = import_module(match.group('module'))
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/lib/python3.9/site-packages/qcengine/__init__.py", line 5, in <module>
    from . import config, exceptions
  File "/usr/local/lib/python3.9/site-packages/qcengine/config.py", line 340, in <module>
    _load_defaults()
  File "/usr/local/lib/python3.9/site-packages/qcengine/config.py", line 173, in _load_defaults
    test_paths = [os.getcwd(), os.path.join(os.path.expanduser("~"), ".qcarchive")]
FileNotFoundError: [Errno 2] No such file or directory

[loriab]

test_paths = [os.getcwd(), os.path.join(os.path.expanduser("~"), ".qcarchive")] form for instance, ['/Users/me/Desktop/html2021', '/Users/me/.qcarchive]. The files don't have to exist. Is your OS checking existence at that point somehow?

[yurivict]

No, this program:

import os

test_paths = [os.getcwd(), os.path.join(os.path.expanduser("~"), ".qcarchive")]
print("test_paths={}".format(test_paths))

prints

test_paths=['/tmp', '/home/yuri/.qcarchive']

and /home/yuri/.qcarchive doesn't exist.

[loriab]

I'm confused, too, as that doesn't seem congruent with your reported

    test_paths = [os.getcwd(), os.path.join(os.path.expanduser("~"), ".qcarchive")]
FileNotFoundError: [Errno 2] No such file or directory

Unless you have a stray /tmp/qcengine.yaml, I'd expect https://github.com/MolSSI/QCEngine/blob/master/qcengine/config.py#L186 to be invoked. Here's a CI run on psi4 1.7 that shows up in qcengine info https://github.com/MolSSI/QCEngine/actions/runs/3654021638/jobs/6174071225#step:9:19 and there's no special configuration or qcengine.yaml files. So all I can think of to debug on your setup is to print stuff until the error becomes clear. Sorry for the trouble.