Make the lock a thread local variable #219

csm10495 · 2023-04-05T00:25:59Z

Fixes #82

With this, i can successfully run on win32 and on wsl (ubuntu)

import tempfile
import pathlib
import threading
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor
from filelock import FileLock
import time
import logging

#log = logging.getLogger("filelock")
#log.setLevel(logging.DEBUG)
#log.addHandler(logging.StreamHandler())

TEST_FILE = pathlib.Path(tempfile.gettempdir()) / 'test_file.txt'
TEST_LOCK_FILE =  pathlib.Path(tempfile.gettempdir()) / 'test_file.txt.lock'
LOCK = FileLock(TEST_LOCK_FILE)

def test():
    with LOCK:
        assert TEST_FILE.read_text() == 'hi'
        TEST_FILE.write_text('')
        assert TEST_FILE.read_text() == ''
        TEST_FILE.write_text('hi')
        assert TEST_FILE.read_text() == 'hi'
        return True

if __name__ == '__main__':
    print(f"Test file: {TEST_FILE}")
    print(f"Test lock file: {TEST_LOCK_FILE}")
    TEST_FILE.write_text('hi')

    results = []

    # works with ProcessPoolExecutor but not with ThreadPoolExecutor
    # It also is more likely to work if we sleep after calling submit()
    with ThreadPoolExecutor(max_workers=256) as pool:
        for i in range(1000):
            if i % 10 == 0:
                print (f"Loop: {i+1}")
            results.append(pool.submit(test))

        for idx, result in enumerate(results):
            print (f"Checking result: {idx + 1}")
            assert result.result() == True

(which is a version of the original failing code from #82 )

gaborbernat · 2023-04-05T03:24:28Z

Making the lock thread local shouldn't be a default. Instead, you should consider your use case to attach the lock to a thread local variable.

csm10495 · 2023-04-05T03:55:06Z

I'm curious though: Is there ever a case where this would break someone? I guess if someone expected the object to be the same across threads.. but then why would they be using the lock to begin with?

It seemed reasonable to think that the lock was similar to a multiprocessing.Lock in that it would be safe to try to use it across threads/processes. We found via #82 that that isn't exactly the case.

Should it instead be documented that default behavior is not compatible with multithreading?

gaborbernat · 2023-04-05T04:07:41Z

My worry is about backwards compatibility. What if today someone attaches the lock to a thread local object. Wouldn't this break those people? Before today you could have used a global variable to lock out threads, however, this change would move that functionality to the file system level. Which could be an issue if the lock type is a soft file.

csm10495 · 2023-04-05T04:13:25Z

If they did that, they would have to have done something like this already:

local = threading.local()

def get_lock():
    local.LOCK = FileLock(TEST_LOCK_FILE)
    return local.LOCK

then use get_lock() inside the thread. This still works for me in my testing.

If they did something like this:

local = threading.local()
local.LOCK = FileLock(TEST_LOCK_FILE)

def test():
    with local.LOCK:

either way (with this change or without it) they would get

AttributeError: '_thread._local' object has no attribute 'LOCK'

from inside the thread (so they need to recreate it).

gaborbernat · 2023-04-05T04:37:23Z

Okay, based on this I am willing to accept this tomorrow, however I would like you to stick around and handle any potential follow-up pull request that might be needed to address regressions introduced by this. Thank you

csm10495 · 2023-04-05T04:40:27Z

Sure thing. If anything weird comes up, happy to help figure it out. (fingers crossed that it just works though :) )

TheMatt2 · 2023-04-05T13:07:06Z

I'm curious though: Is there ever a case where this would break someone? I guess if someone expected the object to be the same across threads.. but then why would they be using the lock to begin with?

I must agree that I can not think of a reason this would break "someone". I'm concerned this will somehow break something somewhere, as is the nature of these things. Perhaps a major version change? (v3.11 or v4).

TheMatt2 · 2023-04-05T13:17:39Z

Just pointing out. BaseFileLock already has an explicit _thread_lock that it uses to atomically change its rentrant count.
If you make BaseFileLock local, I believe this secondary lock can be removed completely.

https://github.com/tox-dev/py-filelock/blob/33c07264904bda9ee9369513e0f6ddcfbad13b10/src/filelock/_api.py#L170-L172

gaborbernat

If you make BaseFileLock local, I believe this secondary lock can be removed completely.

Let's make the CI also pass.

TheMatt2 · 2023-04-05T15:14:03Z

For what it's worth, virtualenv already implements a custom wrapper around filelock to create this functionality.
And this change would not break virtualenv.

https://github.com/pypa/virtualenv/blob/0e4cad0e06668fbd93be6605c9a13f871ce647c7/src/virtualenv/util/lock.py#L13-L30

gaborbernat · 2023-04-05T19:21:28Z

@csm10495 seems pypy does not like it 🤔

csm10495 · 2023-04-05T19:25:06Z

What went wrong? pre-commit looks ok. Have a log?

gaborbernat · 2023-04-05T19:28:27Z

Check the CI those jobs just hang 🤷

csm10495 · 2023-04-05T19:34:05Z

.. oh lol. I'll probably come back to it a bit later, will set up pypy in wsl to see.

csm10495 · 2023-04-05T19:43:07Z

... uhh umm:

csm10495@csm10495-desk:~/pypy/pypy3.9-v7.3.11-linux64/bin $ ./pypy -m pip freeze
attrs==22.2.0
cffi==1.15.1
exceptiongroup==1.1.1
filelock @ file:///mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock
greenlet==0.4.13
hpy==0.0.4.dev179+g9b5d200
iniconfig==2.0.0
packaging==23.0
pluggy==1.0.0
pytest==7.2.2
pytest-mock==3.10.0
readline==6.2.4.1
tomli==2.0.1
csm10495@csm10495-desk:~/pypy/pypy3.9-v7.3.11-linux64/bin $ ./pypy -m pytest /mnt/c/Users/csm10495/Desktop/filelock_testing -vvv
===================================================================== test session starts ======================================================================platform linux -- Python 3.9.16[pypy-7.3.11-final], pytest-7.2.2, pluggy-1.0.0 -- /home/csm10495/pypy/pypy3.9-v7.3.11-linux64/bin/pypy
cachedir: .pytest_cache
rootdir: /mnt/c/Users/csm10495/Desktop/filelock_testing
plugins: mock-3.10.0
collected 56 items

../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_error.py::test_timeout_str PASSED                                    [  1%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_error.py::test_timeout_repr PASSED                                   [  3%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_error.py::test_timeout_lock_file PASSED                              [  5%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_error.py::test_timeout_pickle PASSED                                 [  7%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_simple[UnixFileLock-str] PASSED                    [  8%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_simple[UnixFileLock-PurePath] PASSED               [ 10%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_simple[UnixFileLock-Path] PASSED                   [ 12%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_simple[SoftFileLock-str] PASSED                    [ 14%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_simple[SoftFileLock-PurePath] PASSED               [ 16%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_simple[SoftFileLock-Path] PASSED                   [ 17%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_ro_folder[UnixFileLock] PASSED                     [ 19%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_ro_folder[SoftFileLock] PASSED                     [ 21%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_ro_file[UnixFileLock] PASSED                       [ 23%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_ro_file[SoftFileLock] PASSED                       [ 25%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_missing_directory[UnixFileLock] PASSED             [ 26%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_missing_directory[SoftFileLock] PASSED             [ 28%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_nested_context_manager[UnixFileLock] PASSED        [ 30%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_nested_context_manager[SoftFileLock] PASSED        [ 32%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_nested_acquire[UnixFileLock] PASSED                [ 33%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_nested_acquire[SoftFileLock] PASSED                [ 35%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_nested_forced_release[UnixFileLock] PASSED         [ 37%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_nested_forced_release[SoftFileLock] PASSED         [ 39%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_threaded_shared_lock_obj[UnixFileLock] PASSED      [ 41%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_threaded_shared_lock_obj[SoftFileLock] PASSED      [ 42%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_threaded_lock_different_lock_obj[UnixFileLock] PASSED [ 44%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_threaded_lock_different_lock_obj[SoftFileLock] PASSED [ 46%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_timeout[UnixFileLock] PASSED                       [ 48%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_timeout[SoftFileLock] PASSED                       [ 50%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_non_blocking[UnixFileLock] PASSED                  [ 51%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_non_blocking[SoftFileLock] PASSED                  [ 53%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_default_timeout[UnixFileLock] PASSED               [ 55%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_default_timeout[SoftFileLock] PASSED               [ 57%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_context_release_on_exc[UnixFileLock] PASSED        [ 58%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_context_release_on_exc[SoftFileLock] PASSED        [ 60%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_acquire_release_on_exc[UnixFileLock] PASSED        [ 62%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_acquire_release_on_exc[SoftFileLock] PASSED        [ 64%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_del[UnixFileLock] SKIPPED (del() does not trigger GC in PyPy) [ 66%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_del[SoftFileLock] SKIPPED (del() does not trigger GC in PyPy) [ 67%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_cleanup_soft_lock PASSED                           [ 69%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_poll_intervall_deprecated[UnixFileLock] PASSED     [ 71%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_poll_intervall_deprecated[SoftFileLock] PASSED     [ 73%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_context_decorator[UnixFileLock] PASSED             [ 75%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_context_decorator[SoftFileLock] PASSED             [ 76%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_lock_mode PASSED                                   [ 78%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_lock_mode_soft PASSED                              [ 80%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_umask PASSED                                       [ 82%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_umask_soft PASSED                                  [ 83%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_wrong_platform PASSED                              [ 85%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_flock_not_implemented_unix PASSED                  [ 87%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_soft_errors PASSED                                 [ 89%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_thrashing_with_thread_pool_passing_lock_to_threads[UnixFileLock] PASSED [ 91%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_thrashing_with_thread_pool_passing_lock_to_threads[SoftFileLock] PASSED [ 92%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_thrashing_with_thread_pool_global_lock[UnixFileLock] PASSED [ 94%]
                                                                                                                                                           PASSED               [ 96%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_thrashing_with_thread_pool_lock_recreated_in_each_thread[UnixFileLock]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_thrashing_with_thread_pool_global_lock[SoftFileLock]                   PASSED            [ 98%]
../../../../../mnt/c/Users/csm10495/Desktop/filelock_testing/py-filelock/tests/test_filelock.py::test_thrashing_with_thread_pool_lock_recreated_in_each_thread[SoftFileLock] PASSED                     [100%]
======================================================================================== 54 passed, 2 skipped in 4.43s =========================================================================================
csm10495@csm10495-desk:~/pypy/pypy3.9-v7.3.11-linux64/bin $

...

csm10495@csm10495-desk:~/pypy/pypy3.9-v7.3.11-linux64/bin 2 $ ./pypy -m tox --workdir /mnt/c/Users/csm10495/Desktop/filelock_testing -vvv -e dev
ROOT: 158 D setup logging to NOTSET on pid 1814 [tox/report.py:221]
ROOT: 213 W No tox.ini or setup.cfg or pyproject.toml found, assuming empty tox.ini at /home/csm10495/pypy/pypy3.9-v7.3.11-linux64/bin [tox/config/source/discover.py:77]
dev: 275 D created app data folder /home/csm10495/.local/share/virtualenv [virtualenv/app_data/__init__.py:40]
dev: 284 I find interpreter for spec PythonSpec(path=/home/csm10495/pypy/pypy3.9-v7.3.11-linux64/bin/pypy) [virtualenv/discovery/builtin.py:56]
dev: 285 I proposed PythonInfo(spec=PyPy3.9.16.final.0-64, exe=/home/csm10495/pypy/pypy3.9-v7.3.11-linux64/bin/pypy, platform=linux, version='3.9.16 (feeb267ead3e6771d3f2f49b83e1894839f64fb7, Dec 29 2022, 14:23:21)\n[PyPy 7.3.11 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)]', encoding_fs_io=utf-8-UTF-8) [virtualenv/discovery/builtin.py:63]
dev: 285 D accepted PythonInfo(spec=PyPy3.9.16.final.0-64, exe=/home/csm10495/pypy/pypy3.9-v7.3.11-linux64/bin/pypy, platform=linux, version='3.9.16 (feeb267ead3e6771d3f2f49b83e1894839f64fb7, Dec 29 2022, 14:23:21)\n[PyPy 7.3.11 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)]', encoding_fs_io=utf-8-UTF-8) [virtualenv/discovery/builtin.py:65]
dev: 286 D filesystem is case-sensitive [virtualenv/info.py:24]
dev: 347 I create virtual environment via PyPy3Posix(dest=/mnt/c/Users/csm10495/Desktop/filelock_testing/dev, clear=False, no_vcs_ignore=False, global=False) [virtualenv/run/session.py:48]
dev: 352 D create folder /mnt/c/Users/csm10495/Desktop/filelock_testing/dev/bin [virtualenv/util/path/_sync.py:9]
dev: 360 D create folder /mnt/c/Users/csm10495/Desktop/filelock_testing/dev/lib/pypy3.9/site-packages [virtualenv/util/path/_sync.py:9]
dev: 378 D write /mnt/c/Users/csm10495/Desktop/filelock_testing/dev/pyvenv.cfg [virtualenv/create/pyenv_cfg.py:30]
dev: 379 D      home = /home/csm10495/pypy/pypy3.9-v7.3.11-linux64/bin [virtualenv/create/pyenv_cfg.py:34]
dev: 379 D      implementation = PyPy [virtualenv/create/pyenv_cfg.py:34]
dev: 379 D      version_info = 3.9.16.final.0 [virtualenv/create/pyenv_cfg.py:34]
dev: 379 D      virtualenv = 20.21.0 [virtualenv/create/pyenv_cfg.py:34]
dev: 379 D      include-system-site-packages = false [virtualenv/create/pyenv_cfg.py:34]
dev: 379 D      base-prefix = /home/csm10495/pypy/pypy3.9-v7.3.11-linux64 [virtualenv/create/pyenv_cfg.py:34]
dev: 379 D      base-exec-prefix = /home/csm10495/pypy/pypy3.9-v7.3.11-linux64 [virtualenv/create/pyenv_cfg.py:34]
dev: 379 D      base-executable = /home/csm10495/pypy/pypy3.9-v7.3.11-linux64/bin/pypy [virtualenv/create/pyenv_cfg.py:34]
dev: 387 D symlink /home/csm10495/pypy/pypy3.9-v7.3.11-linux64/bin/pypy to /mnt/c/Users/csm10495/Desktop/filelock_testing/dev/bin/pypy [virtualenv/util/path/_sync.py:28]
dev: 422 D symlink /home/csm10495/pypy/pypy3.9-v7.3.11-linux64/bin/libpypy3.9-c.so to /mnt/c/Users/csm10495/Desktop/filelock_testing/dev/bin/libpypy3.9-c.so [virtualenv/util/path/_sync.py:28]
dev: 432 D symlink /home/csm10495/pypy/pypy3.9-v7.3.11-linux64/bin/libpypy3.9-c.so.debug to /mnt/c/Users/csm10495/Desktop/filelock_testing/dev/bin/libpypy3.9-c.so.debug [virtualenv/util/path/_sync.py:28]
dev: 442 D symlink /home/csm10495/pypy/pypy3.9-v7.3.11-linux64/lib/libsqlite3.so.0 to /mnt/c/Users/csm10495/Desktop/filelock_testing/dev/lib/libsqlite3.so.0 [virtualenv/util/path/_sync.py:28]
dev: 451 D symlink directory /home/csm10495/pypy/pypy3.9-v7.3.11-linux64/lib/tk8.5 to /mnt/c/Users/csm10495/Desktop/filelock_testing/dev/lib/tk8.5 [virtualenv/util/path/_sync.py:28]
dev: 461 D symlink /home/csm10495/pypy/pypy3.9-v7.3.11-linux64/lib/libtk8.5.so to /mnt/c/Users/csm10495/Desktop/filelock_testing/dev/lib/libtk8.5.so [virtualenv/util/path/_sync.py:28]
dev: 470 D symlink /home/csm10495/pypy/pypy3.9-v7.3.11-linux64/lib/libtinfow.so.6 to /mnt/c/Users/csm10495/Desktop/filelock_testing/dev/lib/libtinfow.so.6 [virtualenv/util/path/_sync.py:28]
dev: 479 D symlink /home/csm10495/pypy/pypy3.9-v7.3.11-linux64/lib/libgdbm.so.4 to /mnt/c/Users/csm10495/Desktop/filelock_testing/dev/lib/libgdbm.so.4 [virtualenv/util/path/_sync.py:28]
dev: 489 D symlink directory /home/csm10495/pypy/pypy3.9-v7.3.11-linux64/lib/tcl8.5 to /mnt/c/Users/csm10495/Desktop/filelock_testing/dev/lib/tcl8.5 [virtualenv/util/path/_sync.py:28]
dev: 498 D symlink /home/csm10495/pypy/pypy3.9-v7.3.11-linux64/lib/libpanelw.so.6 to /mnt/c/Users/csm10495/Desktop/filelock_testing/dev/lib/libpanelw.so.6 [virtualenv/util/path/_sync.py:28]
dev: 508 D symlink /home/csm10495/pypy/pypy3.9-v7.3.11-linux64/lib/libexpat.so.1 to /mnt/c/Users/csm10495/Desktop/filelock_testing/dev/lib/libexpat.so.1 [virtualenv/util/path/_sync.py:28]
dev: 518 D symlink /home/csm10495/pypy/pypy3.9-v7.3.11-linux64/lib/liblzma.so.5 to /mnt/c/Users/csm10495/Desktop/filelock_testing/dev/lib/liblzma.so.5 [virtualenv/util/path/_sync.py:28]
dev: 527 D symlink /home/csm10495/pypy/pypy3.9-v7.3.11-linux64/lib/libffi.so.6 to /mnt/c/Users/csm10495/Desktop/filelock_testing/dev/lib/libffi.so.6 [virtualenv/util/path/_sync.py:28]
dev: 536 D symlink /home/csm10495/pypy/pypy3.9-v7.3.11-linux64/lib/libncursesw.so.6 to /mnt/c/Users/csm10495/Desktop/filelock_testing/dev/lib/libncursesw.so.6 [virtualenv/util/path/_sync.py:28]
dev: 546 D symlink /home/csm10495/pypy/pypy3.9-v7.3.11-linux64/lib/libtcl8.5.so to /mnt/c/Users/csm10495/Desktop/filelock_testing/dev/lib/libtcl8.5.so [virtualenv/util/path/_sync.py:28]
dev: 550 D create virtualenv import hook file /mnt/c/Users/csm10495/Desktop/filelock_testing/dev/lib/pypy3.9/site-packages/_virtualenv.pth [virtualenv/create/via_global_ref/api.py:89]
dev: 555 D create /mnt/c/Users/csm10495/Desktop/filelock_testing/dev/lib/pypy3.9/site-packages/_virtualenv.py [virtualenv/create/via_global_ref/api.py:92]
dev: 569 D ============================== target debug ============================== [virtualenv/run/session.py:50]
dev: 569 D debug via /mnt/c/Users/csm10495/Desktop/filelock_testing/dev/bin/pypy3 /home/csm10495/pypy/pypy3.9-v7.3.11-linux64/lib/pypy3.9/site-packages/virtualenv/create/debug.py [virtualenv/create/creator.py:193]
dev: 569 D {
  "sys": {
    "executable": "/mnt/c/Users/csm10495/Desktop/filelock_testing/dev/bin/pypy3",
    "_base_executable": "/mnt/c/Users/csm10495/Desktop/filelock_testing/dev/bin/pypy3",
    "prefix": "/mnt/c/Users/csm10495/Desktop/filelock_testing/dev",
    "base_prefix": "/home/csm10495/pypy/pypy3.9-v7.3.11-linux64",
    "real_prefix": null,
    "exec_prefix": "/mnt/c/Users/csm10495/Desktop/filelock_testing/dev",
    "base_exec_prefix": "/home/csm10495/pypy/pypy3.9-v7.3.11-linux64",
    "path": [
      "/home/csm10495/pypy/pypy3.9-v7.3.11-linux64/lib/pypy3.9",
      "/mnt/c/Users/csm10495/Desktop/filelock_testing/dev/lib/pypy3.9/site-packages"
    ],
    "meta_path": [
      "<class '_virtualenv._Finder'>",
      "<class '_frozen_importlib.BuiltinImporter'>",
      "<class '_frozen_importlib.FrozenImporter'>",
      "<class '_frozen_importlib_external.PathFinder'>"
    ],
    "fs_encoding": "utf-8",
    "io_encoding": "UTF-8"
  },
  "version": "3.9.16 (feeb267ead3e6771d3f2f49b83e1894839f64fb7, Dec 29 2022, 14:23:21)\n[PyPy 7.3.11 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)]",
  "makefile_filename": "/home/csm10495/pypy/pypy3.9-v7.3.11-linux64/lib/pypy3.9/config-3.9-x86_64-linux-gnu/Makefile",
  "os": "<module 'os' from '/home/csm10495/pypy/pypy3.9-v7.3.11-linux64/lib/pypy3.9/os.py'>",
  "site": "<module 'site' from '/home/csm10495/pypy/pypy3.9-v7.3.11-linux64/lib/pypy3.9/site.py'>",
  "datetime": "<module 'datetime' from '/home/csm10495/pypy/pypy3.9-v7.3.11-linux64/lib/pypy3.9/datetime.py'>",
  "math": "<module 'math' (built-in)>",
  "json": "<module 'json' from '/home/csm10495/pypy/pypy3.9-v7.3.11-linux64/lib/pypy3.9/json/__init__.py'>"
} [virtualenv/run/session.py:51]
dev: 787 I add seed packages via FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/csm10495/.local/share/virtualenv) [virtualenv/run/session.py:55]
dev: 790 D install setuptools from wheel /home/csm10495/pypy/pypy3.9-v7.3.11-linux64/lib/pypy3.9/site-packages/virtualenv/seed/wheels/embed/setuptools-67.4.0-py3-none-any.whl via CopyPipInstall [virtualenv/seed/embed/via_app_data/via_app_data.py:47]
dev: 790 D install pip from wheel /home/csm10495/pypy/pypy3.9-v7.3.11-linux64/lib/pypy3.9/site-packages/virtualenv/seed/wheels/embed/pip-23.0.1-py3-none-any.whl via CopyPipInstall [virtualenv/seed/embed/via_app_data/via_app_data.py:47]
dev: 791 D build install image for pip-23.0.1-py3-none-any.whl to /home/csm10495/.local/share/virtualenv/wheel/3.9/image/1/CopyPipInstall/pip-23.0.1-py3-none-any [virtualenv/seed/embed/via_app_data/pip_install/base.py:45]
dev: 792 D build install image for setuptools-67.4.0-py3-none-any.whl to /home/csm10495/.local/share/virtualenv/wheel/3.9/image/1/CopyPipInstall/setuptools-67.4.0-py3-none-any [virtualenv/seed/embed/via_app_data/pip_install/base.py:45]
dev: 791 D install wheel from wheel /home/csm10495/pypy/pypy3.9-v7.3.11-linux64/lib/pypy3.9/site-packages/virtualenv/seed/wheels/embed/wheel-0.38.4-py3-none-any.whl via CopyPipInstall [virtualenv/seed/embed/via_app_data/via_app_data.py:47]
dev: 794 D build install image for wheel-0.38.4-py3-none-any.whl to /home/csm10495/.local/share/virtualenv/wheel/3.9/image/1/CopyPipInstall/wheel-0.38.4-py3-none-any [virtualenv/seed/embed/via_app_data/pip_install/base.py:45]
dev: 832 D copy directory /home/csm10495/.local/share/virtualenv/wheel/3.9/image/1/CopyPipInstall/wheel-0.38.4-py3-none-any/wheel to /mnt/c/Users/csm10495/Desktop/filelock_testing/dev/lib/pypy3.9/site-packages/wheel [virtualenv/util/path/_sync.py:36]
dev: 967 D copy /home/csm10495/.local/share/virtualenv/wheel/3.9/image/1/CopyPipInstall/setuptools-67.4.0-py3-none-any/setuptools-67.4.0.virtualenv to /mnt/c/Users/csm10495/Desktop/filelock_testing/dev/lib/pypy3.9/site-packages/setuptools-67.4.0.virtualenv [virtualenv/util/path/_sync.py:36]
dev: 972 D copy directory /home/csm10495/.local/share/virtualenv/wheel/3.9/image/1/CopyPipInstall/setuptools-67.4.0-py3-none-any/_distutils_hack to /mnt/c/Users/csm10495/Desktop/filelock_testing/dev/lib/pypy3.9/site-packages/_distutils_hack [virtualenv/util/path/_sync.py:36]
dev: 994 D copy directory /home/csm10495/.local/share/virtualenv/wheel/3.9/image/1/CopyPipInstall/setuptools-67.4.0-py3-none-any/setuptools-67.4.0.dist-info to /mnt/c/Users/csm10495/Desktop/filelock_testing/dev/lib/pypy3.9/site-packages/setuptools-67.4.0.dist-info [virtualenv/util/path/_sync.py:36]
dev: 1061 D copy directory /home/csm10495/.local/share/virtualenv/wheel/3.9/image/1/CopyPipInstall/pip-23.0.1-py3-none-any/pip to /mnt/c/Users/csm10495/Desktop/filelock_testing/dev/lib/pypy3.9/site-packages/pip [virtualenv/util/path/_sync.py:36]
dev: 1109 D copy directory /home/csm10495/.local/share/virtualenv/wheel/3.9/image/1/CopyPipInstall/setuptools-67.4.0-py3-none-any/setuptools to /mnt/c/Users/csm10495/Desktop/filelock_testing/dev/lib/pypy3.9/site-packages/setuptools [virtualenv/util/path/_sync.py:36]
dev: 1202 D copy /home/csm10495/.local/share/virtualenv/wheel/3.9/image/1/CopyPipInstall/wheel-0.38.4-py3-none-any/wheel-0.38.4.virtualenv to /mnt/c/Users/csm10495/Desktop/filelock_testing/dev/lib/pypy3.9/site-packages/wheel-0.38.4.virtualenv [virtualenv/util/path/_sync.py:36]
dev: 1209 D copy directory /home/csm10495/.local/share/virtualenv/wheel/3.9/image/1/CopyPipInstall/wheel-0.38.4-py3-none-any/wheel-0.38.4.dist-info to /mnt/c/Users/csm10495/Desktop/filelock_testing/dev/lib/pypy3.9/site-packages/wheel-0.38.4.dist-info [virtualenv/util/path/_sync.py:36]
dev: 1370 D generated console scripts wheel wheel3 wheel-3.9 wheel3.9 [virtualenv/seed/embed/via_app_data/pip_install/base.py:41]
dev: 3768 D copy directory /home/csm10495/.local/share/virtualenv/wheel/3.9/image/1/CopyPipInstall/setuptools-67.4.0-py3-none-any/pkg_resources to /mnt/c/Users/csm10495/Desktop/filelock_testing/dev/lib/pypy3.9/site-packages/pkg_resources [virtualenv/util/path/_sync.py:36]
dev: 4477 D copy /home/csm10495/.local/share/virtualenv/wheel/3.9/image/1/CopyPipInstall/setuptools-67.4.0-py3-none-any/distutils-precedence.pth to /mnt/c/Users/csm10495/Desktop/filelock_testing/dev/lib/pypy3.9/site-packages/distutils-precedence.pth [virtualenv/util/path/_sync.py:36]
dev: 4481 D generated console scripts  [virtualenv/seed/embed/via_app_data/pip_install/base.py:41]
dev: 15198 D copy directory /home/csm10495/.local/share/virtualenv/wheel/3.9/image/1/CopyPipInstall/pip-23.0.1-py3-none-any/pip-23.0.1.dist-info to /mnt/c/Users/csm10495/Desktop/filelock_testing/dev/lib/pypy3.9/site-packages/pip-23.0.1.dist-info [virtualenv/util/path/_sync.py:36]
dev: 15387 D copy /home/csm10495/.local/share/virtualenv/wheel/3.9/image/1/CopyPipInstall/pip-23.0.1-py3-none-any/pip-23.0.1.virtualenv to /mnt/c/Users/csm10495/Desktop/filelock_testing/dev/lib/pypy3.9/site-packages/pip-23.0.1.virtualenv [virtualenv/util/path/_sync.py:36]
dev: 15489 D generated console scripts pip pip3 pip-3.9 pip3.9 [virtualenv/seed/embed/via_app_data/pip_install/base.py:41]
dev: 15490 I add activators for Bash, CShell, Fish, Nushell, PowerShell, Python [virtualenv/run/session.py:61]
dev: 15523 D write /mnt/c/Users/csm10495/Desktop/filelock_testing/dev/pyvenv.cfg [virtualenv/create/pyenv_cfg.py:30]
dev: 15523 D    home = /home/csm10495/pypy/pypy3.9-v7.3.11-linux64/bin [virtualenv/create/pyenv_cfg.py:34]
dev: 15523 D    implementation = PyPy [virtualenv/create/pyenv_cfg.py:34]
dev: 15523 D    version_info = 3.9.16.final.0 [virtualenv/create/pyenv_cfg.py:34]
dev: 15523 D    virtualenv = 20.21.0 [virtualenv/create/pyenv_cfg.py:34]
dev: 15523 D    include-system-site-packages = false [virtualenv/create/pyenv_cfg.py:34]
dev: 15523 D    base-prefix = /home/csm10495/pypy/pypy3.9-v7.3.11-linux64 [virtualenv/create/pyenv_cfg.py:34]
dev: 15524 D    base-exec-prefix = /home/csm10495/pypy/pypy3.9-v7.3.11-linux64 [virtualenv/create/pyenv_cfg.py:34]
dev: 15524 D    base-executable = /home/csm10495/pypy/pypy3.9-v7.3.11-linux64/bin/pypy [virtualenv/create/pyenv_cfg.py:34]
  dev: OK (15.31 seconds)
  congratulations :) (15.40 seconds)
csm10495@csm10495-desk:~/pypy/pypy3.9-v7.3.11-linux64/bin $

.. interesting. Has it ever hung before?

csm10495 · 2023-04-05T19:47:46Z

I can try to spinup a VM later to try a bit more in depth if we don't see something obvious

gaborbernat · 2023-04-05T19:52:30Z

Has it ever hung before?

Sometimes in the past, pypy is buggy 😆 we had to alter the code to support it.

csm10495 · 2023-04-05T22:56:45Z

I spun-up an Ubuntu 20.04 LTS VM and still can't seem to get it to fail:

csm10495@csm10495-VirtualBox:~/pypy3.9-v7.3.11-linux64/bin/py-filelock$ ../pypy -m pytest 

================================================================ test session starts ================================================================

platform linux -- Python 3.9.16[pypy-7.3.11-final], pytest-7.2.2, pluggy-1.0.0

rootdir: /home/csm10495/pypy3.9-v7.3.11-linux64/bin/py-filelock

plugins: mock-3.10.0

collected 56 items                                                                                                                                  



tests/test_error.py ....                                                                                                                      [  7%]

tests/test_filelock.py ................................ss..................                                                                   [100%]



=========================================================== 54 passed, 2 skipped in 4.80s ===========================================================

csm10495@csm10495-VirtualBox:~/pypy3.9-v7.3.11-linux64/bin/py-filelock$ cat /etc/lsb-release 

DISTRIB_ID=Ubuntu

DISTRIB_RELEASE=22.04

DISTRIB_CODENAME=jammy

DISTRIB_DESCRIPTION="Ubuntu 22.04.2 LTS"

csm10495@csm10495-VirtualBox:~/pypy3.9-v7.3.11-linux64/bin/py-filelock$ ../pypy --version

Python 3.9.16 (feeb267ead3e6771d3f2f49b83e1894839f64fb7, Dec 29 2022, 14:23:21)

[PyPy 7.3.11 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)]

csm10495@csm10495-VirtualBox:~/pypy3.9-v7.3.11-linux64/bin/py-filelock$

I'm running out of ideas to repro :/

csm10495 · 2023-04-05T23:05:40Z

So this is kind of weird: If I use --cov it seems like it hangs. If I don't use --cov it seems to pass normally:

csm10495@csm10495-VirtualBox:~/pypy3.9-v7.3.11-linux64/bin/py-filelock$ ../pypy -m pytest --color=yes --junitxml /home/csm10495/pypy3.9-v7.3.11-linux64/bin/py-filelock/.tox/junit.py310.xml  -vvv --cov

================================================================ test session starts ================================================================

platform linux -- Python 3.9.16[pypy-7.3.11-final], pytest-7.2.2, pluggy-1.0.0 -- /home/csm10495/pypy3.9-v7.3.11-linux64/bin/pypy

cachedir: .pytest_cache

rootdir: /home/csm10495/pypy3.9-v7.3.11-linux64/bin/py-filelock

plugins: cov-4.0.0, mock-3.10.0

collected 56 items                                                                                                                                  



tests/test_error.py::test_timeout_str PASSED                                                                                                  [  1%]

tests/test_error.py::test_timeout_repr PASSED                                                                                                 [  3%]

tests/test_error.py::test_timeout_lock_file PASSED                                                                                            [  5%]

tests/test_error.py::test_timeout_pickle PASSED                                                                                               [  7%]

tests/test_filelock.py::test_simple[UnixFileLock-str] PASSED                                                                                  [  8%]

tests/test_filelock.py::test_simple[UnixFileLock-PurePath] PASSED                                                                             [ 10%]

tests/test_filelock.py::test_simple[UnixFileLock-Path] PASSED                                                                                 [ 12%]

tests/test_filelock.py::test_simple[SoftFileLock-str] PASSED                                                                                  [ 14%]

tests/test_filelock.py::test_simple[SoftFileLock-PurePath] PASSED                                                                             [ 16%]

tests/test_filelock.py::test_simple[SoftFileLock-Path] PASSED                                                                                 [ 17%]

tests/test_filelock.py::test_ro_folder[UnixFileLock] PASSED                                                                                   [ 19%]

tests/test_filelock.py::test_ro_folder[SoftFileLock] PASSED                                                                                   [ 21%]

tests/test_filelock.py::test_ro_file[UnixFileLock] PASSED                                                                                     [ 23%]

tests/test_filelock.py::test_ro_file[SoftFileLock] PASSED                                                                                     [ 25%]

tests/test_filelock.py::test_missing_directory[UnixFileLock] PASSED                                                                           [ 26%]

tests/test_filelock.py::test_missing_directory[SoftFileLock] PASSED                                                                           [ 28%]

tests/test_filelock.py::test_nested_context_manager[UnixFileLock] PASSED                                                                      [ 30%]

tests/test_filelock.py::test_nested_context_manager[SoftFileLock] PASSED                                                                      [ 32%]

tests/test_filelock.py::test_nested_acquire[UnixFileLock] PASSED                                                                              [ 33%]

tests/test_filelock.py::test_nested_acquire[SoftFileLock] PASSED                                                                              [ 35%]

tests/test_filelock.py::test_nested_forced_release[UnixFileLock] PASSED                                                                       [ 37%]

tests/test_filelock.py::test_nested_forced_release[SoftFileLock] PASSED                                                                       [ 39%]

tests/test_filelock.py::test_threaded_shared_lock_obj[UnixFileLock]

gaborbernat · 2023-04-05T23:40:18Z

I'd be ok to disable coverage reporting for pypy...

csm10495 · 2023-04-05T23:52:24Z

Maybe? Let me try debugging it a tiny bit more later today. If I still have no idea, then yeah i guess disable.

csm10495 · 2023-04-06T03:46:49Z

@TheMatt2 with pypy? Or is that regular cpython?

TheMatt2 · 2023-04-06T03:51:56Z

cpython. ~~I'll test pypy in a sec~~

pypy ran in 47.8 seconds (of filelock before this PR)

pypy3 -m pytest -vvv --cov
=========================================================================================== test session starts ============================================================================================
platform darwin -- Python 3.7.13[pypy-7.3.9-final], pytest-7.2.2, pluggy-1.0.0 -- /usr/local/bin/pypy3
cachedir: .pytest_cache
rootdir: .../py-filelock
plugins: timeout-2.1.0, mock-3.10.0, cov-4.0.0
collected 50 items                                                                                                                                                                                         

tests/test_error.py::test_timeout_str PASSED                                                                                                                                                         [  2%]
tests/test_error.py::test_timeout_repr PASSED                                                                                                                                                        [  4%]
tests/test_error.py::test_timeout_lock_file PASSED                                                                                                                                                   [  6%]
tests/test_error.py::test_timeout_pickle PASSED                                                                                                                                                      [  8%]
tests/test_filelock.py::test_simple[UnixFileLock-str] PASSED                                                                                                                                         [ 10%]
tests/test_filelock.py::test_simple[UnixFileLock-PurePath] PASSED                                                                                                                                    [ 12%]
tests/test_filelock.py::test_simple[UnixFileLock-Path] PASSED                                                                                                                                        [ 14%]
tests/test_filelock.py::test_simple[SoftFileLock-str] PASSED                                                                                                                                         [ 16%]
tests/test_filelock.py::test_simple[SoftFileLock-PurePath] PASSED                                                                                                                                    [ 18%]
tests/test_filelock.py::test_simple[SoftFileLock-Path] PASSED                                                                                                                                        [ 20%]
tests/test_filelock.py::test_ro_folder[UnixFileLock] PASSED                                                                                                                                          [ 22%]
tests/test_filelock.py::test_ro_folder[SoftFileLock] PASSED                                                                                                                                          [ 24%]
tests/test_filelock.py::test_ro_file[UnixFileLock] PASSED                                                                                                                                            [ 26%]
tests/test_filelock.py::test_ro_file[SoftFileLock] PASSED                                                                                                                                            [ 28%]
tests/test_filelock.py::test_missing_directory[UnixFileLock] PASSED                                                                                                                                  [ 30%]
tests/test_filelock.py::test_missing_directory[SoftFileLock] PASSED                                                                                                                                  [ 32%]
tests/test_filelock.py::test_nested_context_manager[UnixFileLock] PASSED                                                                                                                             [ 34%]
tests/test_filelock.py::test_nested_context_manager[SoftFileLock] PASSED                                                                                                                             [ 36%]
tests/test_filelock.py::test_nested_acquire[UnixFileLock] PASSED                                                                                                                                     [ 38%]
tests/test_filelock.py::test_nested_acquire[SoftFileLock] PASSED                                                                                                                                     [ 40%]
tests/test_filelock.py::test_nested_forced_release[UnixFileLock] PASSED                                                                                                                              [ 42%]
tests/test_filelock.py::test_nested_forced_release[SoftFileLock] PASSED                                                                                                                              [ 44%]
tests/test_filelock.py::test_threaded_shared_lock_obj[UnixFileLock] PASSED                                                                                                                           [ 46%]
tests/test_filelock.py::test_threaded_shared_lock_obj[SoftFileLock] PASSED                                                                                                                           [ 48%]
tests/test_filelock.py::test_threaded_lock_different_lock_obj[UnixFileLock] PASSED                                                                                                                   [ 50%]
tests/test_filelock.py::test_threaded_lock_different_lock_obj[SoftFileLock] PASSED                                                                                                                   [ 52%]
tests/test_filelock.py::test_timeout[UnixFileLock] PASSED                                                                                                                                            [ 54%]
tests/test_filelock.py::test_timeout[SoftFileLock] PASSED                                                                                                                                            [ 56%]
tests/test_filelock.py::test_non_blocking[UnixFileLock] PASSED                                                                                                                                       [ 58%]
tests/test_filelock.py::test_non_blocking[SoftFileLock] PASSED                                                                                                                                       [ 60%]
tests/test_filelock.py::test_default_timeout[UnixFileLock] PASSED                                                                                                                                    [ 62%]
tests/test_filelock.py::test_default_timeout[SoftFileLock] PASSED                                                                                                                                    [ 64%]
tests/test_filelock.py::test_context_release_on_exc[UnixFileLock] PASSED                                                                                                                             [ 66%]
tests/test_filelock.py::test_context_release_on_exc[SoftFileLock] PASSED                                                                                                                             [ 68%]
tests/test_filelock.py::test_acquire_release_on_exc[UnixFileLock] PASSED                                                                                                                             [ 70%]
tests/test_filelock.py::test_acquire_release_on_exc[SoftFileLock] PASSED                                                                                                                             [ 72%]
tests/test_filelock.py::test_del[UnixFileLock] SKIPPED (del() does not trigger GC in PyPy)                                                                                                           [ 74%]
tests/test_filelock.py::test_del[SoftFileLock] SKIPPED (del() does not trigger GC in PyPy)                                                                                                           [ 76%]
tests/test_filelock.py::test_cleanup_soft_lock PASSED                                                                                                                                                [ 78%]
tests/test_filelock.py::test_poll_intervall_deprecated[UnixFileLock] PASSED                                                                                                                          [ 80%]
tests/test_filelock.py::test_poll_intervall_deprecated[SoftFileLock] PASSED                                                                                                                          [ 82%]
tests/test_filelock.py::test_context_decorator[UnixFileLock] PASSED                                                                                                                                  [ 84%]
tests/test_filelock.py::test_context_decorator[SoftFileLock] PASSED                                                                                                                                  [ 86%]
tests/test_filelock.py::test_lock_mode PASSED                                                                                                                                                        [ 88%]
tests/test_filelock.py::test_lock_mode_soft PASSED                                                                                                                                                   [ 90%]
tests/test_filelock.py::test_umask PASSED                                                                                                                                                            [ 92%]
tests/test_filelock.py::test_umask_soft PASSED                                                                                                                                                       [ 94%]
tests/test_filelock.py::test_wrong_platform PASSED                                                                                                                                                   [ 96%]
tests/test_filelock.py::test_flock_not_implemented_unix PASSED                                                                                                                                       [ 98%]
tests/test_filelock.py::test_soft_errors PASSED                                                                                                                                                      [100%]

---------- coverage: platform darwin, python 3.7.13-final-0 ----------
Name                     Stmts   Miss Branch BrPart  Cover   Missing
--------------------------------------------------------------------
tests/test_filelock.py     323     13     24      1    96%   364-382, 445, 505
--------------------------------------------------------------------
TOTAL                      342     13     24      1    96%

1 file skipped due to complete coverage.

Required test coverage of 76.0% reached. Total coverage: 96.17%

====================================================================================== 48 passed, 2 skipped in 47.88s ======================================================================================

csm10495 · 2023-04-06T04:00:29Z

@TheMatt2 Can you run the same pytest for pypy against this changeset instead of the one before it?

TheMatt2 · 2023-04-06T04:10:16Z

Sure... oh interesting.

~~It failed in 56 seconds.~~

~~Soooo... shoot. Or I guess this is a good thing.~~
~~This is with the code from this PR~~
(wrong version was being tested)

``` pypy3 -m pytest -vvv --cov =========================================================================================== test session starts ============================================================================================ platform darwin -- Python 3.7.13[pypy-7.3.9-final], pytest-7.2.2, pluggy-1.0.0 -- /usr/local/bin/pypy3 cachedir: .pytest_cache rootdir: .../py-filelock plugins: timeout-2.1.0, mock-3.10.0, cov-4.0.0 collected 56 items

tests/test_error.py::test_timeout_str PASSED [ 1%]
tests/test_error.py::test_timeout_repr PASSED [ 3%]
tests/test_error.py::test_timeout_lock_file PASSED [ 5%]
tests/test_error.py::test_timeout_pickle PASSED [ 7%]
tests/test_filelock.py::test_simple[UnixFileLock-str] PASSED [ 8%]
tests/test_filelock.py::test_simple[UnixFileLock-PurePath] PASSED [ 10%]
tests/test_filelock.py::test_simple[UnixFileLock-Path] PASSED [ 12%]
tests/test_filelock.py::test_simple[SoftFileLock-str] PASSED [ 14%]
tests/test_filelock.py::test_simple[SoftFileLock-PurePath] PASSED [ 16%]
tests/test_filelock.py::test_simple[SoftFileLock-Path] PASSED [ 17%]
tests/test_filelock.py::test_ro_folder[UnixFileLock] PASSED [ 19%]
tests/test_filelock.py::test_ro_folder[SoftFileLock] PASSED [ 21%]
tests/test_filelock.py::test_ro_file[UnixFileLock] PASSED [ 23%]
tests/test_filelock.py::test_ro_file[SoftFileLock] PASSED [ 25%]
tests/test_filelock.py::test_missing_directory[UnixFileLock] PASSED [ 26%]
tests/test_filelock.py::test_missing_directory[SoftFileLock] PASSED [ 28%]
tests/test_filelock.py::test_nested_context_manager[UnixFileLock] PASSED [ 30%]
tests/test_filelock.py::test_nested_context_manager[SoftFileLock] PASSED [ 32%]
tests/test_filelock.py::test_nested_acquire[UnixFileLock] PASSED [ 33%]
tests/test_filelock.py::test_nested_acquire[SoftFileLock] PASSED [ 35%]
tests/test_filelock.py::test_nested_forced_release[UnixFileLock] PASSED [ 37%]
tests/test_filelock.py::test_nested_forced_release[SoftFileLock] PASSED [ 39%]
tests/test_filelock.py::test_threaded_shared_lock_obj[UnixFileLock] PASSED [ 41%]
tests/test_filelock.py::test_threaded_shared_lock_obj[SoftFileLock] PASSED [ 42%]
tests/test_filelock.py::test_threaded_lock_different_lock_obj[UnixFileLock] PASSED [ 44%]
tests/test_filelock.py::test_threaded_lock_different_lock_obj[SoftFileLock] PASSED [ 46%]
tests/test_filelock.py::test_timeout[UnixFileLock] PASSED [ 48%]
tests/test_filelock.py::test_timeout[SoftFileLock] PASSED [ 50%]
tests/test_filelock.py::test_non_blocking[UnixFileLock] PASSED [ 51%]
tests/test_filelock.py::test_non_blocking[SoftFileLock] PASSED [ 53%]
tests/test_filelock.py::test_default_timeout[UnixFileLock] PASSED [ 55%]
tests/test_filelock.py::test_default_timeout[SoftFileLock] PASSED [ 57%]
tests/test_filelock.py::test_context_release_on_exc[UnixFileLock] PASSED [ 58%]
tests/test_filelock.py::test_context_release_on_exc[SoftFileLock] PASSED [ 60%]
tests/test_filelock.py::test_acquire_release_on_exc[UnixFileLock] PASSED [ 62%]
tests/test_filelock.py::test_acquire_release_on_exc[SoftFileLock] PASSED [ 64%]
tests/test_filelock.py::test_del[UnixFileLock] SKIPPED (del() does not trigger GC in PyPy) [ 66%]
tests/test_filelock.py::test_del[SoftFileLock] SKIPPED (del() does not trigger GC in PyPy) [ 67%]
tests/test_filelock.py::test_cleanup_soft_lock PASSED [ 69%]
tests/test_filelock.py::test_poll_intervall_deprecated[UnixFileLock] PASSED [ 71%]
tests/test_filelock.py::test_poll_intervall_deprecated[SoftFileLock] PASSED [ 73%]
tests/test_filelock.py::test_context_decorator[UnixFileLock] PASSED [ 75%]
tests/test_filelock.py::test_context_decorator[SoftFileLock] PASSED [ 76%]
tests/test_filelock.py::test_lock_mode PASSED [ 78%]
tests/test_filelock.py::test_lock_mode_soft PASSED [ 80%]
tests/test_filelock.py::test_umask PASSED [ 82%]
tests/test_filelock.py::test_umask_soft PASSED [ 83%]
tests/test_filelock.py::test_wrong_platform PASSED [ 85%]
tests/test_filelock.py::test_flock_not_implemented_unix PASSED [ 87%]
tests/test_filelock.py::test_soft_errors PASSED [ 89%]
tests/test_filelock.py::test_thrashing_with_threadpool_passing_lock_to_threads[UnixFileLock] FAILED [ 91%]
tests/test_filelock.py::test_thrashing_with_threadpool_passing_lock_to_threads[SoftFileLock] FAILED [ 92%]
tests/test_filelock.py::test_thrashing_with_threadpool_global_lock[UnixFileLock] FAILED [ 94%]
tests/test_filelock.py::test_thrashing_with_threadpool_global_lock[SoftFileLock] FAILED [ 96%]
tests/test_filelock.py::test_thrashing_with_threadpool_lock_recreated_in_each_thread[UnixFileLock] PASSED [ 98%]
tests/test_filelock.py::test_thrashing_with_threadpool_lock_recreated_in_each_thread[SoftFileLock] PASSED [100%]

================================================================================================= FAILURES =================================================================================================
___________________________________________________________________ test_thrashing_with_threadpool_passing_lock_to_threads[UnixFileLock] ___________________________________________________________________

tmp_path = PosixPath('.../pytest-5/test_thrashing_with_threadpool0'), lock_type = <class 'filelock._unix.UnixFileLock'>

@pytest.mark.parametrize("lock_type", [FileLock, SoftFileLock])
def test_thrashing_with_threadpool_passing_lock_to_threads(tmp_path: Path, lock_type: BaseFileLock) -> None:
    lock_file = tmp_path / "test.txt.lock"
    txt_file = tmp_path / "test.txt"

    # notice how lock is passed to the function below
    lock = lock_type(lock_file)

    def mess_with_file(lock, txt_file):
        with lock:
            for _ in range(3):
                u = str(uuid4())
                txt_file.write_text(u)
                assert txt_file.read_text() == u

        return True

    results = []
    with ThreadPoolExecutor() as executor:
        for _ in range(100):
            results.append(executor.submit(mess_with_file, lock, txt_file))

  assert all([r.result() for r in results])

tests/test_filelock.py:538:

tests/test_filelock.py:538: in
assert all([r.result() for r in results])
/usr/local/Cellar/pypy3/7.3.9/libexec/lib-python/3/concurrent/futures/_base.py:428: in result
return self.__get_result()
/usr/local/Cellar/pypy3/7.3.9/libexec/lib-python/3/concurrent/futures/_base.py:384: in __get_result
raise self._exception
/usr/local/Cellar/pypy3/7.3.9/libexec/lib-python/3/concurrent/futures/thread.py:57: in run
result = self.fn(*self.args, **self.kwargs)

lock = <filelock._unix.UnixFileLock object at 0x00007fc5964f74b0>
txt_file = PosixPath('.../pytest-5/test_thrashing_with_threadpool0/test.txt')

def mess_with_file(lock, txt_file):
    with lock:
        for _ in range(3):
            u = str(uuid4())
            txt_file.write_text(u)

          assert txt_file.read_text() == u

E AssertionError: assert '' == '30550e77-d36e-4f61-8d63-f0797175de86'
E - 30550e77-d36e-4f61-8d63-f0797175de86

tests/test_filelock.py:529: AssertionError
___________________________________________________________________ test_thrashing_with_threadpool_passing_lock_to_threads[SoftFileLock] ___________________________________________________________________

tmp_path = PosixPath('.../pytest-5/test_thrashing_with_threadpool1'), lock_type = <class 'filelock._soft.SoftFileLock'>

@pytest.mark.parametrize("lock_type", [FileLock, SoftFileLock])
def test_thrashing_with_threadpool_passing_lock_to_threads(tmp_path: Path, lock_type: BaseFileLock) -> None:
    lock_file = tmp_path / "test.txt.lock"
    txt_file = tmp_path / "test.txt"

    # notice how lock is passed to the function below
    lock = lock_type(lock_file)

    def mess_with_file(lock, txt_file):
        with lock:
            for _ in range(3):
                u = str(uuid4())
                txt_file.write_text(u)
                assert txt_file.read_text() == u

        return True

    results = []
    with ThreadPoolExecutor() as executor:
        for _ in range(100):
            results.append(executor.submit(mess_with_file, lock, txt_file))

  assert all([r.result() for r in results])

tests/test_filelock.py:538:

tests/test_filelock.py:538: in
assert all([r.result() for r in results])
/usr/local/Cellar/pypy3/7.3.9/libexec/lib-python/3/concurrent/futures/_base.py:428: in result
return self.__get_result()
/usr/local/Cellar/pypy3/7.3.9/libexec/lib-python/3/concurrent/futures/_base.py:384: in __get_result
raise self._exception
/usr/local/Cellar/pypy3/7.3.9/libexec/lib-python/3/concurrent/futures/thread.py:57: in run
result = self.fn(*self.args, **self.kwargs)

lock = <filelock._soft.SoftFileLock object at 0x00007fc5958fb830>
txt_file = PosixPath('.../pytest-5/test_thrashing_with_threadpool1/test.txt')

def mess_with_file(lock, txt_file):
    with lock:
        for _ in range(3):
            u = str(uuid4())
            txt_file.write_text(u)

          assert txt_file.read_text() == u

E AssertionError: assert '193e41cf-9b40-4692-bce6-99d62a955a45' == 'b71e7bda-f44d-4852-82da-34acf75888d0'
E - b71e7bda-f44d-4852-82da-34acf75888d0
E + 193e41cf-9b40-4692-bce6-99d62a955a45

tests/test_filelock.py:529: AssertionError
_________________________________________________________________________ test_thrashing_with_threadpool_global_lock[UnixFileLock] _________________________________________________________________________

tmp_path = PosixPath('.../pytest-5/test_thrashing_with_threadpool2'), lock_type = <class 'filelock._unix.UnixFileLock'>

@pytest.mark.parametrize("lock_type", [FileLock, SoftFileLock])
def test_thrashing_with_threadpool_global_lock(tmp_path: Path, lock_type: BaseFileLock) -> None:
    lock_file = tmp_path / "test.txt.lock"
    txt_file = tmp_path / "test.txt"

    # Notice how lock is scoped to be allowed in the nested function below
    lock = lock_type(lock_file)

    def mess_with_file(txt_file):
        with lock:
            for _ in range(3):
                u = str(uuid4())
                txt_file.write_text(u)
                assert txt_file.read_text() == u

        return True

    results = []
    with ThreadPoolExecutor() as executor:
        for _ in range(100):
            results.append(executor.submit(mess_with_file, txt_file))

  assert all([r.result() for r in results])

tests/test_filelock.py:563:

tests/test_filelock.py:563: in
assert all([r.result() for r in results])
/usr/local/Cellar/pypy3/7.3.9/libexec/lib-python/3/concurrent/futures/_base.py:428: in result
return self.__get_result()
/usr/local/Cellar/pypy3/7.3.9/libexec/lib-python/3/concurrent/futures/_base.py:384: in __get_result
raise self._exception
/usr/local/Cellar/pypy3/7.3.9/libexec/lib-python/3/concurrent/futures/thread.py:57: in run
result = self.fn(*self.args, **self.kwargs)

txt_file = PosixPath('.../pytest-5/test_thrashing_with_threadpool2/test.txt')

def mess_with_file(txt_file):
    with lock:
        for _ in range(3):
            u = str(uuid4())
            txt_file.write_text(u)

          assert txt_file.read_text() == u

E AssertionError: assert '' == '89c14325-6222-4f02-b872-1f6376fa3c2e'
E - 89c14325-6222-4f02-b872-1f6376fa3c2e

tests/test_filelock.py:554: AssertionError
_________________________________________________________________________ test_thrashing_with_threadpool_global_lock[SoftFileLock] _________________________________________________________________________

tmp_path = PosixPath('.../pytest-5/test_thrashing_with_threadpool3'), lock_type = <class 'filelock._soft.SoftFileLock'>

@pytest.mark.parametrize("lock_type", [FileLock, SoftFileLock])
def test_thrashing_with_threadpool_global_lock(tmp_path: Path, lock_type: BaseFileLock) -> None:
    lock_file = tmp_path / "test.txt.lock"
    txt_file = tmp_path / "test.txt"

    # Notice how lock is scoped to be allowed in the nested function below
    lock = lock_type(lock_file)

    def mess_with_file(txt_file):
        with lock:
            for _ in range(3):
                u = str(uuid4())
                txt_file.write_text(u)
                assert txt_file.read_text() == u

        return True

    results = []
    with ThreadPoolExecutor() as executor:
        for _ in range(100):
            results.append(executor.submit(mess_with_file, txt_file))

  assert all([r.result() for r in results])

tests/test_filelock.py:563:

tests/test_filelock.py:563: in
assert all([r.result() for r in results])
/usr/local/Cellar/pypy3/7.3.9/libexec/lib-python/3/concurrent/futures/_base.py:428: in result
return self.__get_result()
/usr/local/Cellar/pypy3/7.3.9/libexec/lib-python/3/concurrent/futures/_base.py:384: in __get_result
raise self._exception
/usr/local/Cellar/pypy3/7.3.9/libexec/lib-python/3/concurrent/futures/thread.py:57: in run
result = self.fn(*self.args, **self.kwargs)

txt_file = PosixPath('.../pytest-5/test_thrashing_with_threadpool3/test.txt')

def mess_with_file(txt_file):
    with lock:
        for _ in range(3):
            u = str(uuid4())
            txt_file.write_text(u)

          assert txt_file.read_text() == u

E AssertionError: assert '' == '14be5b17-20ad-438b-aa51-7c116b1a8984'
E - 14be5b17-20ad-438b-aa51-7c116b1a8984

tests/test_filelock.py:554: AssertionError

---------- coverage: platform darwin, python 3.7.13-final-0 ----------
Name Stmts Miss Branch BrPart Cover Missing

tests/test_filelock.py 375 13 42 1 97% 366-384, 447, 507

TOTAL 394 13 42 1 97%

1 file skipped due to complete coverage.

Required test coverage of 76.0% reached. Total coverage: 96.79%
========================================================================================= short test summary info ==========================================================================================
FAILED tests/test_filelock.py::test_thrashing_with_threadpool_passing_lock_to_threads[UnixFileLock] - AssertionError: assert '' == '30550e77-d36e-4f61-8d63-f0797175de86'
FAILED tests/test_filelock.py::test_thrashing_with_threadpool_passing_lock_to_threads[SoftFileLock] - AssertionError: assert '193e41cf-9b40-4692-bce6-99d62a955a45' == 'b71e7bda-f44d-4852-82da-34acf75888d0'
FAILED tests/test_filelock.py::test_thrashing_with_threadpool_global_lock[UnixFileLock] - AssertionError: assert '' == '89c14325-6222-4f02-b872-1f6376fa3c2e'
FAILED tests/test_filelock.py::test_thrashing_with_threadpool_global_lock[SoftFileLock] - AssertionError: assert '' == '14be5b17-20ad-438b-aa51-7c116b1a8984'
================================================================================= 4 failed, 50 passed, 2 skipped in 56.80s =================================================================================

</details>

csm10495 · 2023-04-06T04:11:39Z

.. Well i didn't expect that at all. I get those failures when i forget to pip install my dev version (and instead the version from pypi ran).. any chance that happened here? (Please make sure that the dev version of filelock is installed in the env)

TheMatt2 · 2023-04-06T04:25:26Z

.. Well i didn't expect that at all. I get those failures when i forget to pip install my dev version (and instead the version from pypi ran).. any chance that happened here?

(Slightly blushing) um... yep that's what happened. I installed the dev version, but I didn't realize the default filelock was already installed...

Let's try that again

TheMatt2 · 2023-04-06T04:30:41Z

Code version in this PR

2017 Macbook Air macOS 12.6.4

Pypy passed in 21.63 seconds without coverage
Pypy passed in 72.63 seconds with coverage

Cache emptied between tests

pypy3 -m pytest -vvv
=========================================================================================== test session starts ============================================================================================
platform darwin -- Python 3.7.13[pypy-7.3.9-final], pytest-7.2.2, pluggy-1.0.0 -- /usr/local/bin/pypy3
cachedir: .pytest_cache
rootdir: .../py-filelock
plugins: timeout-2.1.0, mock-3.10.0, cov-4.0.0
collected 56 items                                                                                                                                                                                         

tests/test_error.py::test_timeout_str PASSED                                                                                                                                                         [  1%]
tests/test_error.py::test_timeout_repr PASSED                                                                                                                                                        [  3%]
tests/test_error.py::test_timeout_lock_file PASSED                                                                                                                                                   [  5%]
tests/test_error.py::test_timeout_pickle PASSED                                                                                                                                                      [  7%]
tests/test_filelock.py::test_simple[UnixFileLock-str] PASSED                                                                                                                                         [  8%]
tests/test_filelock.py::test_simple[UnixFileLock-PurePath] PASSED                                                                                                                                    [ 10%]
tests/test_filelock.py::test_simple[UnixFileLock-Path] PASSED                                                                                                                                        [ 12%]
tests/test_filelock.py::test_simple[SoftFileLock-str] PASSED                                                                                                                                         [ 14%]
tests/test_filelock.py::test_simple[SoftFileLock-PurePath] PASSED                                                                                                                                    [ 16%]
tests/test_filelock.py::test_simple[SoftFileLock-Path] PASSED                                                                                                                                        [ 17%]
tests/test_filelock.py::test_ro_folder[UnixFileLock] PASSED                                                                                                                                          [ 19%]
tests/test_filelock.py::test_ro_folder[SoftFileLock] PASSED                                                                                                                                          [ 21%]
tests/test_filelock.py::test_ro_file[UnixFileLock] PASSED                                                                                                                                            [ 23%]
tests/test_filelock.py::test_ro_file[SoftFileLock] PASSED                                                                                                                                            [ 25%]
tests/test_filelock.py::test_missing_directory[UnixFileLock] PASSED                                                                                                                                  [ 26%]
tests/test_filelock.py::test_missing_directory[SoftFileLock] PASSED                                                                                                                                  [ 28%]
tests/test_filelock.py::test_nested_context_manager[UnixFileLock] PASSED                                                                                                                             [ 30%]
tests/test_filelock.py::test_nested_context_manager[SoftFileLock] PASSED                                                                                                                             [ 32%]
tests/test_filelock.py::test_nested_acquire[UnixFileLock] PASSED                                                                                                                                     [ 33%]
tests/test_filelock.py::test_nested_acquire[SoftFileLock] PASSED                                                                                                                                     [ 35%]
tests/test_filelock.py::test_nested_forced_release[UnixFileLock] PASSED                                                                                                                              [ 37%]
tests/test_filelock.py::test_nested_forced_release[SoftFileLock] PASSED                                                                                                                              [ 39%]
tests/test_filelock.py::test_threaded_shared_lock_obj[UnixFileLock] PASSED                                                                                                                           [ 41%]
tests/test_filelock.py::test_threaded_shared_lock_obj[SoftFileLock] PASSED                                                                                                                           [ 42%]
tests/test_filelock.py::test_threaded_lock_different_lock_obj[UnixFileLock] PASSED                                                                                                                   [ 44%]
tests/test_filelock.py::test_threaded_lock_different_lock_obj[SoftFileLock] PASSED                                                                                                                   [ 46%]
tests/test_filelock.py::test_timeout[UnixFileLock] PASSED                                                                                                                                            [ 48%]
tests/test_filelock.py::test_timeout[SoftFileLock] PASSED                                                                                                                                            [ 50%]
tests/test_filelock.py::test_non_blocking[UnixFileLock] PASSED                                                                                                                                       [ 51%]
tests/test_filelock.py::test_non_blocking[SoftFileLock] PASSED                                                                                                                                       [ 53%]
tests/test_filelock.py::test_default_timeout[UnixFileLock] PASSED                                                                                                                                    [ 55%]
tests/test_filelock.py::test_default_timeout[SoftFileLock] PASSED                                                                                                                                    [ 57%]
tests/test_filelock.py::test_context_release_on_exc[UnixFileLock] PASSED                                                                                                                             [ 58%]
tests/test_filelock.py::test_context_release_on_exc[SoftFileLock] PASSED                                                                                                                             [ 60%]
tests/test_filelock.py::test_acquire_release_on_exc[UnixFileLock] PASSED                                                                                                                             [ 62%]
tests/test_filelock.py::test_acquire_release_on_exc[SoftFileLock] PASSED                                                                                                                             [ 64%]
tests/test_filelock.py::test_del[UnixFileLock] SKIPPED (del() does not trigger GC in PyPy)                                                                                                           [ 66%]
tests/test_filelock.py::test_del[SoftFileLock] SKIPPED (del() does not trigger GC in PyPy)                                                                                                           [ 67%]
tests/test_filelock.py::test_cleanup_soft_lock PASSED                                                                                                                                                [ 69%]
tests/test_filelock.py::test_poll_intervall_deprecated[UnixFileLock] PASSED                                                                                                                          [ 71%]
tests/test_filelock.py::test_poll_intervall_deprecated[SoftFileLock] PASSED                                                                                                                          [ 73%]
tests/test_filelock.py::test_context_decorator[UnixFileLock] PASSED                                                                                                                                  [ 75%]
tests/test_filelock.py::test_context_decorator[SoftFileLock] PASSED                                                                                                                                  [ 76%]
tests/test_filelock.py::test_lock_mode PASSED                                                                                                                                                        [ 78%]
tests/test_filelock.py::test_lock_mode_soft PASSED                                                                                                                                                   [ 80%]
tests/test_filelock.py::test_umask PASSED                                                                                                                                                            [ 82%]
tests/test_filelock.py::test_umask_soft PASSED                                                                                                                                                       [ 83%]
tests/test_filelock.py::test_wrong_platform PASSED                                                                                                                                                   [ 85%]
tests/test_filelock.py::test_flock_not_implemented_unix PASSED                                                                                                                                       [ 87%]
tests/test_filelock.py::test_soft_errors PASSED                                                                                                                                                      [ 89%]
tests/test_filelock.py::test_thrashing_with_threadpool_passing_lock_to_threads[UnixFileLock] PASSED                                                                                                  [ 91%]
tests/test_filelock.py::test_thrashing_with_threadpool_passing_lock_to_threads[SoftFileLock] PASSED                                                                                                  [ 92%]
tests/test_filelock.py::test_thrashing_with_threadpool_global_lock[UnixFileLock] PASSED                                                                                                              [ 94%]
tests/test_filelock.py::test_thrashing_with_threadpool_global_lock[SoftFileLock] PASSED                                                                                                              [ 96%]
tests/test_filelock.py::test_thrashing_with_threadpool_lock_recreated_in_each_thread[UnixFileLock] PASSED                                                                                            [ 98%]
tests/test_filelock.py::test_thrashing_with_threadpool_lock_recreated_in_each_thread[SoftFileLock] PASSED                                                                                            [100%]

====================================================================================== 54 passed, 2 skipped in 21.63s ======================================================================================

pypy3 -m pytest -vvv --cov
=========================================================================================== test session starts ============================================================================================
platform darwin -- Python 3.7.13[pypy-7.3.9-final], pytest-7.2.2, pluggy-1.0.0 -- /usr/local/bin/pypy3
cachedir: .pytest_cache
rootdir: .../py-filelock
plugins: timeout-2.1.0, mock-3.10.0, cov-4.0.0
collected 56 items                                                                                                                                                                                         

tests/test_error.py::test_timeout_str PASSED                                                                                                                                                         [  1%]
tests/test_error.py::test_timeout_repr PASSED                                                                                                                                                        [  3%]
tests/test_error.py::test_timeout_lock_file PASSED                                                                                                                                                   [  5%]
tests/test_error.py::test_timeout_pickle PASSED                                                                                                                                                      [  7%]
tests/test_filelock.py::test_simple[UnixFileLock-str] PASSED                                                                                                                                         [  8%]
tests/test_filelock.py::test_simple[UnixFileLock-PurePath] PASSED                                                                                                                                    [ 10%]
tests/test_filelock.py::test_simple[UnixFileLock-Path] PASSED                                                                                                                                        [ 12%]
tests/test_filelock.py::test_simple[SoftFileLock-str] PASSED                                                                                                                                         [ 14%]
tests/test_filelock.py::test_simple[SoftFileLock-PurePath] PASSED                                                                                                                                    [ 16%]
tests/test_filelock.py::test_simple[SoftFileLock-Path] PASSED                                                                                                                                        [ 17%]
tests/test_filelock.py::test_ro_folder[UnixFileLock] PASSED                                                                                                                                          [ 19%]
tests/test_filelock.py::test_ro_folder[SoftFileLock] PASSED                                                                                                                                          [ 21%]
tests/test_filelock.py::test_ro_file[UnixFileLock] PASSED                                                                                                                                            [ 23%]
tests/test_filelock.py::test_ro_file[SoftFileLock] PASSED                                                                                                                                            [ 25%]
tests/test_filelock.py::test_missing_directory[UnixFileLock] PASSED                                                                                                                                  [ 26%]
tests/test_filelock.py::test_missing_directory[SoftFileLock] PASSED                                                                                                                                  [ 28%]
tests/test_filelock.py::test_nested_context_manager[UnixFileLock] PASSED                                                                                                                             [ 30%]
tests/test_filelock.py::test_nested_context_manager[SoftFileLock] PASSED                                                                                                                             [ 32%]
tests/test_filelock.py::test_nested_acquire[UnixFileLock] PASSED                                                                                                                                     [ 33%]
tests/test_filelock.py::test_nested_acquire[SoftFileLock] PASSED                                                                                                                                     [ 35%]
tests/test_filelock.py::test_nested_forced_release[UnixFileLock] PASSED                                                                                                                              [ 37%]
tests/test_filelock.py::test_nested_forced_release[SoftFileLock] PASSED                                                                                                                              [ 39%]
tests/test_filelock.py::test_threaded_shared_lock_obj[UnixFileLock] PASSED                                                                                                                           [ 41%]
tests/test_filelock.py::test_threaded_shared_lock_obj[SoftFileLock] PASSED                                                                                                                           [ 42%]
tests/test_filelock.py::test_threaded_lock_different_lock_obj[UnixFileLock] PASSED                                                                                                                   [ 44%]
tests/test_filelock.py::test_threaded_lock_different_lock_obj[SoftFileLock] PASSED                                                                                                                   [ 46%]
tests/test_filelock.py::test_timeout[UnixFileLock] PASSED                                                                                                                                            [ 48%]
tests/test_filelock.py::test_timeout[SoftFileLock] PASSED                                                                                                                                            [ 50%]
tests/test_filelock.py::test_non_blocking[UnixFileLock] PASSED                                                                                                                                       [ 51%]
tests/test_filelock.py::test_non_blocking[SoftFileLock] PASSED                                                                                                                                       [ 53%]
tests/test_filelock.py::test_default_timeout[UnixFileLock] PASSED                                                                                                                                    [ 55%]
tests/test_filelock.py::test_default_timeout[SoftFileLock] PASSED                                                                                                                                    [ 57%]
tests/test_filelock.py::test_context_release_on_exc[UnixFileLock] PASSED                                                                                                                             [ 58%]
tests/test_filelock.py::test_context_release_on_exc[SoftFileLock] PASSED                                                                                                                             [ 60%]
tests/test_filelock.py::test_acquire_release_on_exc[UnixFileLock] PASSED                                                                                                                             [ 62%]
tests/test_filelock.py::test_acquire_release_on_exc[SoftFileLock] PASSED                                                                                                                             [ 64%]
tests/test_filelock.py::test_del[UnixFileLock] SKIPPED (del() does not trigger GC in PyPy)                                                                                                           [ 66%]
tests/test_filelock.py::test_del[SoftFileLock] SKIPPED (del() does not trigger GC in PyPy)                                                                                                           [ 67%]
tests/test_filelock.py::test_cleanup_soft_lock PASSED                                                                                                                                                [ 69%]
tests/test_filelock.py::test_poll_intervall_deprecated[UnixFileLock] PASSED                                                                                                                          [ 71%]
tests/test_filelock.py::test_poll_intervall_deprecated[SoftFileLock] PASSED                                                                                                                          [ 73%]
tests/test_filelock.py::test_context_decorator[UnixFileLock] PASSED                                                                                                                                  [ 75%]
tests/test_filelock.py::test_context_decorator[SoftFileLock] PASSED                                                                                                                                  [ 76%]
tests/test_filelock.py::test_lock_mode PASSED                                                                                                                                                        [ 78%]
tests/test_filelock.py::test_lock_mode_soft PASSED                                                                                                                                                   [ 80%]
tests/test_filelock.py::test_umask PASSED                                                                                                                                                            [ 82%]
tests/test_filelock.py::test_umask_soft PASSED                                                                                                                                                       [ 83%]
tests/test_filelock.py::test_wrong_platform PASSED                                                                                                                                                   [ 85%]
tests/test_filelock.py::test_flock_not_implemented_unix PASSED                                                                                                                                       [ 87%]
tests/test_filelock.py::test_soft_errors PASSED                                                                                                                                                      [ 89%]
tests/test_filelock.py::test_thrashing_with_thread_pool_passing_lock_to_threads[UnixFileLock] PASSED                                                                                                 [ 91%]
tests/test_filelock.py::test_thrashing_with_thread_pool_passing_lock_to_threads[SoftFileLock] PASSED                                                                                                 [ 92%]
tests/test_filelock.py::test_thrashing_with_thread_pool_global_lock[UnixFileLock] PASSED                                                                                                             [ 94%]
tests/test_filelock.py::test_thrashing_with_thread_pool_global_lock[SoftFileLock] PASSED                                                                                                             [ 96%]
tests/test_filelock.py::test_thrashing_with_thread_pool_lock_recreated_in_each_thread[UnixFileLock] PASSED                                                                                           [ 98%]
tests/test_filelock.py::test_thrashing_with_thread_pool_lock_recreated_in_each_thread[SoftFileLock] PASSED                                                                                           [100%]

---------- coverage: platform darwin, python 3.7.13-final-0 ----------
Name                     Stmts   Miss Branch BrPart  Cover   Missing
--------------------------------------------------------------------
tests/test_filelock.py     367     13     38      1    97%   374-392, 455, 515
--------------------------------------------------------------------
TOTAL                      386     13     38      1    97%

1 file skipped due to complete coverage.

Required test coverage of 76.0% reached. Total coverage: 96.70%

================================================================================= 54 passed, 2 skipped in 72.63s (0:01:12) =================================================================================

csm10495 · 2023-04-06T04:35:27Z

Phew. A pass with reasonable speed. Yeah I'm not sure why the hang is so long in ci.

TheMatt2 · 2023-04-06T05:57:45Z

I've been unable to replicate the hang in pypy

TheMatt2 · 2023-04-06T15:39:00Z

It seems the test_threaded_lock_different_lock_obj takes significantly longer depending on how many cores the computer has. It may be the ci servers have lots of cores, which is causing the test to take a long time, but not actually hang. Is this possible?

If so, this could be prevented by reducing the number of concurrent threads and increasing how many lock acquires are done per thread:

https://github.com/csm10495/py-filelock/blob/902d659fc47b4e2ae941b5acd5281e08a8a17763/tests/test_filelock.py#L215-L220

gaborbernat · 2023-04-06T15:41:45Z

It may be the ci servers have lots of cores,

Github actions VMs are always provisioned with 2 cores 🤔 See https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources

TheMatt2 · 2023-04-06T15:48:37Z

Well... so much for that theory.
In fairness, that link says the MacOS servers have 3 cores, which does explain why those are slower.

Though, the reason that test_threaded_shared_lock_obj is slower now is because this PR makes sure all threads grab the lock exclusively. Previously, multiple threads were able to all enter the lock together.

I'm still wondering if its possible these "hangs" are just really slow, not actual crashes.

EDIT: Geez yeah.
Pypy takes 2m 40s to run a test that takes Python 3.10 5 seconds at worst.
(Ubuntu 6 cores)

Also the CI is going on.. 20 minutes now? It was 20 seconds for the non-pypy test
These times were taken from the current HEAD, without the changes of this PR

gaborbernat · 2023-04-06T16:08:35Z

So I've put in #224, that will remove coverage reporting for pypy 👍

for more information, see https://pre-commit.ci

Signed-off-by: Bernát Gábor <bgabor8@bloomberg.net>

…ython run as root

TheMatt2 · 2023-04-06T16:36:10Z

Interesting. The prior CI with coverage Ubuntu was canceled due to time on test_filelock.py::test_threaded_shared_lock_obj[SoftFileLock]

MacOS was canceled due to time on test_filelock.py::test_threaded_shared_lock_obj[UnixFileLock]
(The CI report doesn't actually say, it only shows the last test that passed. But from testing it seems this would have been the test to go next)

The tests I ran yesterday on MacOS that completed in reasonable times were done using pypy 3.8

Is this a pypy 3.9 issue?

TheMatt2 · 2023-04-06T16:49:38Z

Update

Ubuntu 22.10

Pypy 3.9

umask 022
git checkout origin/fix-windows-thread-locking
pypy3 -m pip uninstall filelock
pypy3 -m pip install ".[testing]"
# Successfully installed filelock-0.1.dev238+ga2ccce3
rm -rf .coverage .pytest_cache **/__pycache__
time pypy3 -m pytest -vvv --cov

Test completed after

real	26m33.687s
user	9m25.055s
sys	4m43.054s

So it seems that this can complete given enough time.
But it takes forever.

And it only takes 10 seconds if --cov is removed from the arguments.

csm10495 · 2023-04-06T16:56:58Z

That long with multiple cores. I think with just 2 in ci, it's suppppper slow.

TheMatt2 · 2023-04-06T17:55:53Z

I know this is closed now. But I thought I would add the final set of timing results on MacOS.
In short, I can not reproduce the results taking so long.
(Is this going to be another crazy issue where Pypy behaves differently, depending on how it was compiled?)

2017 MacBook Air with 2 hyperthreaded cores

Test on Pypy 3.8: Finished with coverage in 47 seconds

Python 3.8.16 (a9dbdca6fc3286b0addd2240f11d97d8e8de187a, Dec 29 2022, 12:50:26)
[PyPy 7.3.11 with GCC Apple LLVM 13.1.6 (clang-1316.0.21.2.5)] on darwin
filelock-0.1.dev238+ga2ccce3 (this PR)

rm -rf .coverage .pytest_cache **/__pycache__
time ../pypy3.8-v7.3.11-macos_x86_64/bin/pypy3 -m pytest -vvv --cov
...
======================================== 48 passed, 2 skipped in 47.15s =========================================
../pypy3.8-v7.3.11-macos_x86_64/bin/pypy3 -m pytest -vvv --cov  31.77s user 14.81s system 95% cpu 49.025 total

Test on Pypy 3.9: Finished with coverage in 54 seconds

Python 3.9.16 (feeb267ead3e6771d3f2f49b83e1894839f64fb7, Dec 29 2022, 15:36:58)
[PyPy 7.3.11 with GCC Apple LLVM 13.1.6 (clang-1316.0.21.2.5)] on darwin
filelock-0.1.dev238+ga2ccce3

rm -rf .coverage .pytest_cache **/__pycache__
time ../pypy3.9-v7.3.11-macos_x86_64/bin/pypy3 -m pytest -vvv --cov
...
======================================== 54 passed, 2 skipped in 54.14s =========================================
../pypy3.9-v7.3.11-macos_x86_64/bin/pypy3 -m pytest -vvv --cov  37.11s user 16.40s system 94% cpu 56.440 total

csm10495 mentioned this pull request Apr 5, 2023

Does not successfully lock on Windows #82

Closed

csm10495 changed the title ~~Surprisingly simple fix to not working properly with Python threads in Windows~~ Surprisingly simple fix to not working properly with Python threads in various platforms Apr 5, 2023

gaborbernat requested changes Apr 5, 2023

View reviewed changes

gaborbernat changed the title ~~Surprisingly simple fix to not working properly with Python threads in various platforms~~ Make the local a thread local variable Apr 5, 2023

csm10495 force-pushed the fix-windows-thread-locking branch from 33c0726 to ec6b246 Compare April 5, 2023 18:44

csm10495 requested a review from gaborbernat April 5, 2023 18:47

gaborbernat approved these changes Apr 5, 2023

View reviewed changes

gaborbernat force-pushed the fix-windows-thread-locking branch 3 times, most recently from 2df9a89 to 9796392 Compare April 5, 2023 19:06

csm10495 changed the title ~~Make the local a thread local variable~~ Make the lock a thread local variable Apr 6, 2023

csm10495 and others added 8 commits April 6, 2023 09:11

Surprisingly simple fix to not working properly with Python threads

e7b61a4

Move lower to fix issue on all platforms

199c361

[pre-commit.ci] auto fixes from pre-commit.com hooks

b8d671f

for more information, see https://pre-commit.ci

Add some threadpool-thrash tests

1f704f0

Fix CI

d0b41cf

Remove _thread_lock as now the file lock itself will be thread-local

1b62a85

PR feedback

b29fbb8

Signed-off-by: Bernát Gábor <bgabor8@bloomberg.net>

Add some more skips for if tests are run in an environment that has p…

a2ccce3

…ython run as root

gaborbernat force-pushed the fix-windows-thread-locking branch from 902d659 to a2ccce3 Compare April 6, 2023 16:12

gaborbernat merged commit eb09492 into tox-dev:main Apr 6, 2023

slanzmich mentioned this pull request Apr 12, 2023

filelock 3.11 breaks our multi-threaded parallel processing use case #230

Closed

Make the lock a thread local variable #219

Make the lock a thread local variable #219

Conversation

csm10495 commented Apr 5, 2023 • edited Loading

gaborbernat commented Apr 5, 2023

csm10495 commented Apr 5, 2023 • edited Loading

gaborbernat commented Apr 5, 2023

csm10495 commented Apr 5, 2023

gaborbernat commented Apr 5, 2023

csm10495 commented Apr 5, 2023

TheMatt2 commented Apr 5, 2023 • edited Loading

TheMatt2 commented Apr 5, 2023 • edited Loading

gaborbernat left a comment

Choose a reason for hiding this comment

TheMatt2 commented Apr 5, 2023

gaborbernat commented Apr 5, 2023

csm10495 commented Apr 5, 2023

gaborbernat commented Apr 5, 2023

csm10495 commented Apr 5, 2023

csm10495 commented Apr 5, 2023

csm10495 commented Apr 5, 2023

gaborbernat commented Apr 5, 2023

csm10495 commented Apr 5, 2023

csm10495 commented Apr 5, 2023

gaborbernat commented Apr 5, 2023

csm10495 commented Apr 5, 2023

csm10495 commented Apr 6, 2023 • edited Loading

TheMatt2 commented Apr 6, 2023 • edited Loading

csm10495 commented Apr 6, 2023

TheMatt2 commented Apr 6, 2023 • edited Loading

---------- coverage: platform darwin, python 3.7.13-final-0 ---------- Name Stmts Miss Branch BrPart Cover Missing

tests/test_filelock.py 375 13 42 1 97% 366-384, 447, 507

csm10495 commented Apr 6, 2023 • edited Loading

TheMatt2 commented Apr 6, 2023

TheMatt2 commented Apr 6, 2023 • edited Loading

csm10495 commented Apr 6, 2023

TheMatt2 commented Apr 6, 2023

TheMatt2 commented Apr 6, 2023

gaborbernat commented Apr 6, 2023 • edited Loading

TheMatt2 commented Apr 6, 2023 • edited Loading

gaborbernat commented Apr 6, 2023

TheMatt2 commented Apr 6, 2023 • edited Loading

TheMatt2 commented Apr 6, 2023 • edited Loading

csm10495 commented Apr 6, 2023

TheMatt2 commented Apr 6, 2023

csm10495 commented Apr 5, 2023 •

edited

Loading

csm10495 commented Apr 5, 2023 •

edited

Loading

TheMatt2 commented Apr 5, 2023 •

edited

Loading

TheMatt2 commented Apr 5, 2023 •

edited

Loading

csm10495 commented Apr 6, 2023 •

edited

Loading

TheMatt2 commented Apr 6, 2023 •

edited

Loading

TheMatt2 commented Apr 6, 2023 •

edited

Loading

---------- coverage: platform darwin, python 3.7.13-final-0 ----------
Name Stmts Miss Branch BrPart Cover Missing

csm10495 commented Apr 6, 2023 •

edited

Loading

TheMatt2 commented Apr 6, 2023 •

edited

Loading

gaborbernat commented Apr 6, 2023 •

edited

Loading

TheMatt2 commented Apr 6, 2023 •

edited

Loading

TheMatt2 commented Apr 6, 2023 •

edited

Loading

TheMatt2 commented Apr 6, 2023 •

edited

Loading