Bug? FileNotFound on cleanup

[jmrgibson]

Neat plugin, great docs. I'm running into the following issue, outlined below:

Scenario:

• pytest 6.0.1
• pytest-harvest 1.10.0
• pytest-xdist 2.0.0
• running via a shiv
• running multiple instances of pytest simultaneously

stacktrace:

  File "/home/builduser/.shiv/pytest_f6cdbae7-eadd-41ab-aef2-dd73926c7212/site-packages/pytest_harvest/plugin.py", line 435, in pytest_harvest_xdist_cleanup
    rmtree(self.results_path)
  File "/usr/lib/python3.7/shutil.py", line 485, in rmtree
    onerror(os.lstat, path, sys.exc_info())
  File "/usr/lib/python3.7/shutil.py", line 483, in rmtree
    orig_st = os.lstat(path)
FileNotFoundError: [Errno 2] No such file or directory: '.xdist_harvested'

How is self.results_path generated? it it unique per process? or would it be shared if I was running two copies of pytest simultaneously, thus leading to the following error?

EDIT: I should probably mention I'm only using the following hook in my root conftest.py

def pytest_sessionfinish(session):
    """ Gather all results and save them to a csv.
    Works both on worker and master nodes, and also with xdist disabled"""

    session_results_df = get_session_results_df(session)
    suffix = 'all' if is_main_process(session) else get_xdist_worker_id(session)
    session_results_df.to_csv('results_%s.csv' % suffix)

[smarie]

Hi @jmrgibson, thanks for submitting this issue ! Indeed results_path is not unique per process:

python-pytest-harvest/pytest_harvest/plugin.py

Line 398 in 86a259b

class DefaultXDistHarvester(object):

That is currently documented here: https://smarie.github.io/python-pytest-harvest/#pytest-x-dist

We'll probably need to change this. Do you have an alternative in mind ? Maybe as simple as appending os.getpid() to that path ?

[jmrgibson]

I've implement the hooks on my side using a tempdir, which also helps with cleanup as the space gets reclaimed when the process exits. I can make that the default and put up a PR if that suits.

…

On Thu, Dec 10, 2020, 9:33 PM Sylvain Marié ***@***.***> wrote: Hi @jmrgibson <https://github.com/jmrgibson>, thanks for submitting this issue ! Indeed results_path is not unique per process: https://github.com/smarie/python-pytest-harvest/blob/86a259bcb7ac09a643d42a447d7c2e791e403159/pytest_harvest/plugin.py#L398 That is currently documented here: https://smarie.github.io/python-pytest-harvest/#pytest-x-dist We'll probably need to change this. Do you have an alternative in mind ? Maybe as simple as appending os.getpid() to that path ? If so, do not hesitate to propose a PR. Thanks! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#46 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB4NBMYHJ6G3RUALLNCUNJLSUCBVDANCNFSM4US3B4DA> .

[smarie]

This seems like a viable alternative. However I am concerned with debugging. Most users won't be aware that this temp dir is created behind the scenes, and therefore will not be able to perform a bit of debugging themselves by inspecting the files created inside.

Even if this is less elegant, is there any drawback that you might see with appending the process number to this dir name ?

I'm trying to weight one against the other... puzzling