Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a timeout functionality to common locking operations #51565

Closed
pitrou opened this issue Nov 13, 2009 · 13 comments
Closed

Add a timeout functionality to common locking operations #51565

pitrou opened this issue Nov 13, 2009 · 13 comments
Assignees
Labels
interpreter-core Interpreter core (Objects, Python, Grammar, and Parser dirs) stdlib Python modules in the Lib dir type-feature A feature request or enhancement

Comments

@pitrou
Copy link
Member

pitrou commented Nov 13, 2009

BPO 7316
Nosy @pitrou, @briancurtin, @asvetlov, @florentx, @Bluehorn
Files
  • timedlock5.patch
  • timedlock6.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/pitrou'
    closed_at = <Date 2010-04-14.15:45:10.167>
    created_at = <Date 2009-11-13.17:31:47.283>
    labels = ['interpreter-core', 'type-feature', 'library']
    title = 'Add a timeout functionality to common locking operations'
    updated_at = <Date 2010-04-14.21:26:33.991>
    user = 'https://github.com/pitrou'

    bugs.python.org fields:

    activity = <Date 2010-04-14.21:26:33.991>
    actor = 'torsten'
    assignee = 'pitrou'
    closed = True
    closed_date = <Date 2010-04-14.15:45:10.167>
    closer = 'pitrou'
    components = ['Interpreter Core', 'Library (Lib)']
    creation = <Date 2009-11-13.17:31:47.283>
    creator = 'pitrou'
    dependencies = []
    files = ['16609', '16911']
    hgrepos = []
    issue_num = 7316
    keywords = ['patch', 'needs review']
    message_count = 13.0
    messages = ['95192', '95379', '95387', '95388', '95392', '96304', '97436', '97498', '101413', '103070', '103103', '103134', '103148']
    nosy_count = 8.0
    nosy_names = ['pitrou', 'jyasskin', 'gps', 'jnoller', 'brian.curtin', 'asvetlov', 'flox', 'torsten']
    pr_nums = []
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue7316'
    versions = ['Python 3.2']

    @pitrou
    Copy link
    Member Author

    pitrou commented Nov 13, 2009

    Here is a patch which adds a timeout feature to the locking operations
    provided by Python. This feature is added at two levels:

    • the C API level, with a new function PyThread_acquire_lock_timed()
    • the Python level, with an optional timeout argument to the acquire()
      method of Lock and RLock objects (it also helps simplify the wait()
      function of Condition objects)

    The timeout duration is expressed in microseconds at the C API level,
    and in seconds at the Python API level. There is also a new Python-level
    constant, _thread.TIMEOUT_MAX, indicating the max allowable timeout
    value (values above this raise an OverflowError).

    At the C level, the max timeout is PY_TIMEOUT_MAX (in microseconds). The
    caller should check the value him/herself.

    The patch contains both a POSIX implementation and a Windows
    implementation. It still lacks docs.

    @pitrou pitrou added interpreter-core Interpreter core (Objects, Python, Grammar, and Parser dirs) stdlib Python modules in the Lib dir type-feature A feature request or enhancement labels Nov 13, 2009
    @pitrou
    Copy link
    Member Author

    pitrou commented Nov 17, 2009

    This patch adds some docs and comments. It also adds the feature in the
    non-semaphore path of thread_pthread.h, which I had forgotten to address.

    @jyasskin
    Copy link
    Mannequin

    jyasskin mannequin commented Nov 17, 2009

    I don't object strongly, but since locks are "supposed" to be held for
    short amounts of time, a timeout shouldn't be that useful, and when
    people really need it they can put it together with a condition
    variable. Timeouts also interact poorly with condition variables: you
    can time out the initial acquire, but if you wait on a condition there's
    no place to put the timeout on the reacquire.

    Given that it's hard to pick a timeout in most cases anyway, I think
    it'd be a much bigger win to figure out thread interruption. (Yes, I
    know that's hard, and that I promised to do it a long while ago and
    never got around to it.)

    That said, I have no objections at all to adding an internal timeout
    ability for use by Condition.wait, and if you're still enthusiastic
    about adding the timeout given the above argument, I won't block you.

    @pitrou
    Copy link
    Member Author

    pitrou commented Nov 17, 2009

    Timeouts also interact poorly with condition variables: you
    can time out the initial acquire, but if you wait on a condition there's
    no place to put the timeout on the reacquire.

    I don't see how it's an objection. If you have a condition variable you
    just use the cv's timeout feature, don't you? I guess there are already
    tons of combinations which don't make sense anyway.

    Given that it's hard to pick a timeout in most cases anyway, I think
    it'd be a much bigger win to figure out thread interruption. (Yes, I
    know that's hard, and that I promised to do it a long while ago and
    never got around to it.)

    What do you mean by thread interruption? Cancellation?

    That said, I have no objections at all to adding an internal timeout
    ability for use by Condition.wait, and if you're still enthusiastic
    about adding the timeout given the above argument, I won't block you.

    Well, it's pretty basic functionality provided by the underlying OS
    APIs, which is why I think it would be good to expose it. I remember
    being annoyed by its absence, but it was a long time ago and I don't
    remember which problem I was trying to solve.

    (and it's safer than thread cancellation ;-))

    @jyasskin
    Copy link
    Mannequin

    jyasskin mannequin commented Nov 17, 2009

    > Timeouts also interact poorly with condition variables: you
    > can time out the initial acquire, but if you wait on a condition there's
    > no place to put the timeout on the reacquire.

    I don't see how it's an objection. If you have a condition variable you
    just use the cv's timeout feature, don't you? I guess there are already
    tons of combinations which don't make sense anyway.

    The cv's timeout stops waiting for the cv to be notified, but then it
    just calls acquire() with no timeout.

    > Given that it's hard to pick a timeout in most cases anyway, I think
    > it'd be a much bigger win to figure out thread interruption. (Yes, I
    > know that's hard, and that I promised to do it a long while ago and
    > never got around to it.)

    What do you mean by thread interruption? Cancellation?

    Yes, sorry, I was using the Java term, which isn't particularly accurate.

    > That said, I have no objections at all to adding an internal timeout
    > ability for use by Condition.wait, and if you're still enthusiastic
    > about adding the timeout given the above argument, I won't block you.

    Well, it's pretty basic functionality provided by the underlying OS
    APIs, which is why I think it would be good to expose it. I remember
    being annoyed by its absence, but it was a long time ago and I don't
    remember which problem I was trying to solve.

    Fair enough.

    @pitrou pitrou self-assigned this Dec 9, 2009
    @pitrou
    Copy link
    Member Author

    pitrou commented Dec 12, 2009

    Updated patch against py3k.

    @pitrou
    Copy link
    Member Author

    pitrou commented Jan 8, 2010

    Updated patch against newest py3k.

    @jyasskin
    Copy link
    Mannequin

    jyasskin mannequin commented Jan 10, 2010

    diff -r 8089902215a5 Doc/library/_thread.rst
    --- a/Doc/library/_thread.rst   Fri Jan 08 18:54:23 2010 +0100
    +++ b/Doc/library/_thread.rst   Fri Jan 08 20:33:54 2010 +0100
    @@ -103,18 +103,29 @@ It defines the following constant and fu
       Availability: Windows, systems with POSIX threads.

    +.. data:: TIMEOUT_MAX

    Above here, it says, "It defines the following constant and
    functions:", which should be updated to "constants" now that there are
    2.

    •   The maximum value allowed for the *timeout* parameter of
    •   :meth:`Lock.acquire`. Specifiying a timeout greater than this value will
    •   raise an :exc:`OverflowError`.

    Do we want to document what this value is likely to be? Or guarantee
    that it's at least 2000?

    I believe we can support arbitrary values here, subject to floating
    point rounding errors, by calling lock-with-timeout in a loop. I'm not
    sure whether that's a good idea, but it fits better with python's
    arbitrary-precision ints.

     Lock objects have the following methods:

    -.. method:: lock.acquire([waitflag])
    +.. method:: lock.acquire(waitflag=1, timeout=-1)

       Without the optional argument, this method acquires the lock
    unconditionally, if

    Since there are now 2 optional arguments, this needs to be updated.

       necessary waiting until it is released by another thread (only one
    thread at a
       time can acquire a lock --- that's their reason for existence).
    If the integer
       waitflag argument is present, the action depends on its value:
    if it is zero,
       the lock is only acquired if it can be acquired immediately without waiting,

    •   while if it is nonzero, the lock is acquired unconditionally as before.  The
    •   return value is True if the lock is acquired successfully,
      False if not.
    •   while if it is nonzero, the lock is acquired unconditionally as before.
    •   If the floating-point timeout argument is present and positive, it
    •   specifies the maximum wait time in seconds before returning.

    You might mention that "lock.acquire(timeout=0)" is equivalent to
    "lock.acquire(waitflag=0)", and that a missing or negative timeout
    causes an unbounded wait.

    •   The return value is ``True`` if the lock is acquired successfully,
    •   ``False`` if not.

     .. method:: lock.release()
    diff -r 8089902215a5 Doc/library/threading.rst
    --- a/Doc/library/threading.rst Fri Jan 08 18:54:23 2010 +0100
    +++ b/Doc/library/threading.rst Fri Jan 08 20:33:54 2010 +0100
    @@ -155,6 +155,16 @@ This module defines the following functi
       Availability: Windows, systems with POSIX threads.

    +This module also defines the following constant:
    +
    +.. data:: TIMEOUT_MAX
    +

    •   The maximum value allowed for the *timeout* parameter of blocking functions
    •   (:meth:`Lock.acquire`, :meth:`RLock.acquire`, :meth:`Condition.wait`, etc.).
    •   Specifiying a timeout greater than this value will raise an
    •   :exc:`OverflowError`.

     Detailed interfaces for the objects are documented below.

     The design of this module is loosely based on Java's threading model. However,
    @@ -349,7 +359,7 @@ and may vary across implementations.
     All methods are executed atomically.

    -.. method:: Lock.acquire(blocking=True)
    +.. method:: Lock.acquire(blocking=True, timeout=-1)

       Acquire a lock, blocking or non-blocking.

    @@ -363,6 +373,13 @@ All methods are executed atomically.
       without an argument would block, return false immediately; otherwise, do the
       same thing as when called without arguments, and return true.

    •   When invoked with the floating-point *timeout* argument set to a positive
    •   value, block for at most the number of seconds specified by *timeout*
    •   and as long as the lock cannot be acquired.

    s/and as long as the lock cannot be acquired./and return False if the
    lock couldn't be acquired by then./ ? Also consider an equivalent
    comment about timeout<=0 as I suggested for _thread.

    •   The return value is ``True`` if the lock is acquired successfully,
    •   ``False`` if not.

     .. method:: Lock.release()

    @@ -396,7 +413,7 @@ pair) resets the lock to unlocked and al
     :meth:`acquire` to proceed.

    -.. method:: RLock.acquire(blocking=True)
    +.. method:: RLock.acquire(blocking=True, timeout=-1)

       Acquire a lock, blocking or non-blocking.

    @@ -415,6 +432,11 @@ pair) resets the lock to unlocked and al
       without an argument would block, return false immediately; otherwise, do the
       same thing as when called without arguments, and return true.

    •   When invoked with the floating-point *timeout* argument set to a positive
    •   value, block for at most the number of seconds specified by *timeout*
    •   and as long as the lock cannot be acquired.  Return true if the lock has
    •   been acquired, false if the timeout has elapsed.

    True and False? And same comment as for Lock.acquire.

     .. method:: RLock.release()

    diff -r 8089902215a5 Include/pythread.h
    --- a/Include/pythread.h        Fri Jan 08 18:54:23 2010 +0100
    +++ b/Include/pythread.h        Fri Jan 08 20:33:54 2010 +0100
    @@ -23,6 +23,30 @@ PyAPI_FUNC(void) PyThread_free_lock(PyTh
     PyAPI_FUNC(int) PyThread_acquire_lock(PyThread_type_lock, int);
     #define WAIT_LOCK      1
     #define NOWAIT_LOCK    0
    +
    +#if defined(HAVE_LONG_LONG)
    +#define PY_TIMEOUT_T PY_LONG_LONG
    +#define PY_TIMEOUT_MAX PY_LLONG_MAX

    I think this deserves a comment that it's not the same as
    _thread.TIMEOUT_MAX and why.

    +#else
    +#define PY_TIMEOUT_T long
    +#define PY_TIMEOUT_MAX LONG_MAX
    +#endif
    +
    +/* In the NT API, the timeout is a DWORD and is expressed in milliseconds */
    +#if defined (NT_THREADS) && (0xFFFFFFFFLL * 1000 < PY_TIMEOUT_MAX)
    +#undef PY_TIMEOUT_MAX
    +#define PY_TIMEOUT_MAX (0xFFFFFFFFLL * 1000)
    +#endif
    +
    +/* If microseconds == 0, the call is non-blocking: it returns immediately

    •   even when the lock can't be acquired.
    •   If microseconds > 0, the call waits up to the specified duration.
    •   If microseconds < 0, the call waits until success (or abnormal failure)
    •   microseconds must be less than PY_TIMEOUT_MAX. Behaviour otherwise is
    •   undefined. */
      +PyAPI_FUNC(int) PyThread_acquire_lock_timed(PyThread_type_lock,
    •                                           PY_TIMEOUT_T microseconds);
       PyAPI_FUNC(void) PyThread_release_lock(PyThread_type_lock);

     PyAPI_FUNC(size_t) PyThread_get_stacksize(void);
    diff -r 8089902215a5 Lib/_dummy_thread.py
    --- a/Lib/dummy_thread.py      Fri Jan 08 18:54:23 2010 +0100
    +++ b/Lib/dummy_thread.py      Fri Jan 08 20:33:54 2010 +0100
    @@ -17,6 +17,10 @@ __all
    = ['error', 'start_new_thread',
               'interrupt_main', 'LockType']

     import traceback as _traceback
    +import time
    +
    +# A dummy value
    +TIMEOUT_MAX = 2**31

    This should probably be the same as the typical value for _thread.TIMEOUT_MAX.

     class error(Exception):
        """Dummy implementation of _thread.error."""
    @@ -92,7 +96,7 @@ class LockType(object):
        def __init__(self):
            self.locked_status = False

    •    def acquire(self, waitflag=None):
    •    def acquire(self, waitflag=None, timeout=-1):
              """Dummy implementation of acquire().

            For blocking calls, self.locked_status is automatically set to
    @@ -111,6 +115,8 @@ class LockType(object):
                    self.locked_status = True
                    return True
                else:

    •                if timeout > 0:
    •                    time.sleep(timeout)
                      return False

        __enter__ = acquire
    diff -r 8089902215a5 Lib/multiprocessing/pool.py
    --- a/Lib/multiprocessing/pool.py       Fri Jan 08 18:54:23 2010 +0100
    +++ b/Lib/multiprocessing/pool.py       Fri Jan 08 20:33:54 2010 +0100
    @@ -379,10 +379,10 @@ class Pool(object):
                    p.terminate()

            debug('joining task handler')

    •        task_handler.join(1e100)
    •        task_handler.join()

    Why is this change here? (Mostly curiosity)

            debug('joining result handler')

    •        result_handler.join(1e100)
    •        task_handler.join()

            if pool and hasattr(pool[0], 'terminate'):
                debug('joining pool workers')
    diff -r 8089902215a5 Lib/test/lock_tests.py
    --- a/Lib/test/lock_tests.py    Fri Jan 08 18:54:23 2010 +0100
    +++ b/Lib/test/lock_tests.py    Fri Jan 08 20:33:54 2010 +0100
    @@ -4,7 +4,7 @@ Various tests for synchronization primit

     import sys
     import time
    -from _thread import start_new_thread, get_ident
    +from _thread import start_new_thread, get_ident, TIMEOUT_MAX
     import threading
     import unittest

    @@ -62,6 +62,14 @@ class BaseTestCase(unittest.TestCase):
            support.threading_cleanup(*self._threads)
            support.reap_children()

    •    def assertTimeout(self, actual, expected):
    •        # The waiting and/or time.time() can be imprecise, which
    •        # is why comparing to the expected value would sometimes fail
    •        # (especially under Windows).
    •        self.assertGreaterEqual(actual, expected * 0.6)
    •        # Test nothing insane happened
    •        self.assertLess(actual, expected * 10.0)

     class BaseLockTests(BaseTestCase):
        """
    @@ -143,6 +151,31 @@ class BaseLockTests(BaseTestCase):
            Bunch(f, 15).wait_for_finished()
            self.assertEqual(n, len(threading.enumerate()))

    •    def test_timeout(self):
    •        lock = self.locktype()
    •        # Can't set timeout if not blocking

    Please add this to the documentation.

    •        self.assertRaises(ValueError, lock.acquire, 0, 1)
    •        # Invalid timeout values
    •        self.assertRaises(ValueError, lock.acquire, timeout=-100)
    •        self.assertRaises(OverflowError, lock.acquire, timeout=1e100)
    •        self.assertRaises(OverflowError, lock.acquire, timeout=TIMEOUT_MAX + 1)
    •        # TIMEOUT_MAX is ok
    •        lock.acquire(timeout=TIMEOUT_MAX)
    •        lock.release()
    •        t1 = time.time()
    •        self.assertTrue(lock.acquire(timeout=5))
    •        t2 = time.time()
    •        self.assertLess(t2 - t1, 5)

    This is just a sanity-check that a successful acquire finishes in a
    sane amount of time, right? Please comment that.

    •        results = []
    •        def f():
    •            t1 = time.time()
    •            results.append(lock.acquire(timeout=0.5))
    •            t2 = time.time()
    •            results.append(t2 - t1)
    •        Bunch(f, 1).wait_for_finished()
    •        self.assertFalse(results[0])
    •        self.assertTimeout(results[1], 0.5)

     class LockTests(BaseLockTests):
        """
    @@ -178,7 +211,7 @@ class LockTests(BaseLockTests):
            b.wait_for_finished()
            lock.acquire()
            lock.release()

     class RLockTests(BaseLockTests):
        """
    @@ -284,14 +317,14 @@ class EventTests(BaseTestCase):
            def f():
                results1.append(evt.wait(0.0))
                t1 = time.time()

    •            r = evt.wait(0.2)
    •            r = evt.wait(0.5)
                  t2 = time.time()
                  results2.append((r, t2 - t1))
              Bunch(f, N).wait_for_finished()
              self.assertEqual(results1, [False] * N)
              for r, dt in results2:
                  self.assertFalse(r)
    •            self.assertTrue(dt >= 0.2, dt)
    •            self.assertTimeout(dt, 0.5)
              # The event is set
              results1 = []
              results2 = []
      @@ -397,14 +430,14 @@ class ConditionTests(BaseTestCase):
              def f():
                  cond.acquire()
                  t1 = time.time()
    •            cond.wait(0.2)
    •            cond.wait(0.5)
                  t2 = time.time()
                  cond.release()
                  results.append(t2 - t1)
              Bunch(f, N).wait_for_finished()
              self.assertEqual(len(results), 5)
              for dt in results:
    •            self.assertTrue(dt >= 0.2, dt)
    •            self.assertTimeout(dt, 0.5)

     class BaseSemaphoreTests(BaseTestCase):
    diff -r 8089902215a5 Lib/threading.py
    --- a/Lib/threading.py  Fri Jan 08 18:54:23 2010 +0100
    +++ b/Lib/threading.py  Fri Jan 08 20:33:54 2010 +0100
    @@ -31,6 +31,7 @@ try:
        _CRLock = _thread.RLock
     except AttributeError:
        _CRLock = None
    +TIMEOUT_MAX = _thread.TIMEOUT_MAX
     del _thread

    @@ -107,14 +108,14 @@ class _RLock(Verbose):
            return "<%s owner=%r count=%d>" % (
                    self.__class
    _.__name__, owner, self._count)

    •    def acquire(self, blocking=True):
    •    def acquire(self, blocking=True, timeout=-1):
              me = _get_ident()
              if self._owner == me:
                  self._count = self._count + 1
                  if __debug__:
                      self._note("%s.acquire(%s): recursive success", self, blocking)
                  return 1
    •        rc = self._block.acquire(blocking)
    •        rc = self._block.acquire(blocking, timeout)
              if rc:
                  self._owner = me
                  self._count = 1
      @@ -234,22 +235,10 @@ class _Condition(Verbose):
                      if __debug
      _:
                          self._note("%s.wait(): got it", self)
                  else:
    •                # Balancing act:  We can't afford a pure busy loop, so we
    •                # have to sleep; but if we sleep the whole timeout time,
    •                # we'll be unresponsive.  The scheme here sleeps very
    •                # little at first, longer as time goes on, but never longer
    •                # than 20 times per second (or the timeout time remaining).
    •                endtime = _time() + timeout
    •                delay = 0.0005 # 500 us -> initial delay of 1 ms
    •                while True:
    •                    gotit = waiter.acquire(0)
    •                    if gotit:
    •                        break
    •                    remaining = endtime - _time()
    •                    if remaining <= 0:
    •                        break
    •                    delay = min(delay * 2, remaining, .05)
    •                    _sleep(delay)
    •                if timeout > 0:
    •                    gotit = waiter.acquire(True, timeout)
    •                else:
    •                    gotit = waiter.acquire(False)
                      if not gotit:
                          if __debug__:
                              self._note("%s.wait(%s): timed out", self, timeout)
      diff -r 8089902215a5 Modules/_threadmodule.c
      --- a/Modules/_threadmodule.c   Fri Jan 08 18:54:23 2010 +0100
      +++ b/Modules/_threadmodule.c   Fri Jan 08 20:33:54 2010 +0100
      @@ -39,18 +39,47 @@ lock_dealloc(lockobject *self)
       }

     static PyObject *
    -lock_PyThread_acquire_lock(lockobject *self, PyObject *args)
    +lock_PyThread_acquire_lock(lockobject *self, PyObject *args, PyObject *kwds)
     {

    •       int i = 1;
    •       char *kwlist[] = {"blocking", "timeout", NULL};
    •       int blocking = 1;
    •       double timeout = -1;
    •       PY_TIMEOUT_T microseconds;
    •       int r;
    •       if (!PyArg_ParseTuple(args, "|i:acquire", &i))
    •       if (!PyArg_ParseTupleAndKeywords(args, kwds, "|id:acquire", kwlist,

    •                                        &blocking, &timeout))
                     return NULL;

    •       if (!blocking && timeout != -1) {

    •               PyErr_SetString(PyExc_ValueError, "can't specify a timeout "

    •                               "for a non-blocking call");

    •               return NULL;

    •       }

    •       if (timeout < 0 && timeout != -1) {

    •               PyErr_SetString(PyExc_ValueError, "timeout value must be "

    •                               "strictly positive");

    •               return NULL;

    •       }

    •       if (!blocking)

    •               microseconds = 0;

    •       else if (timeout == -1)

    •               microseconds = -1;

    •       else {

    •               timeout *= 1e6;

    •               if (timeout > PY_TIMEOUT_MAX) {

    I believe it's possible for this comparison to return false, but for
    the conversion to PY_TIMEOUT_T to still overflow:

    $ cat test.c
    #include <stdio.h>
    #include <limits.h>
    
    int main() {
      double d_ll_max = (double)LONG_LONG_MAX;
      if (d_ll_max > LONG_LONG_MAX)
       printf("Bigger\n");
      if (d_ll_max == LONG_LONG_MAX)
       printf("Equal\n");
      printf("%lld %lf %lld\n", LONG_LONG_MAX, d_ll_max, (long long)d_ll_max);
    return 0;
    }
    $ ./test
    Equal
    9223372036854775807 9223372036854775808.000000 -9223372036854775808

    Unfortunately, that overflowing cast back to long long is undefined
    behavior, and I don't know how to check for that overflow before it
    happens.

    •                       PyErr_SetString(PyExc_OverflowError,
    •                                       "timeout value is too large");
    •                       return NULL;
    •               }
    •               microseconds = (PY_TIMEOUT_T) timeout;
    •       }

           Py_BEGIN_ALLOW_THREADS

    •       i = PyThread_acquire_lock(self->lock_lock, i);
    •       r = PyThread_acquire_lock_timed(self->lock_lock, microseconds);
             Py_END_ALLOW_THREADS
    •       return PyBool_FromLong((long)i);
    •       return PyBool_FromLong(r);
       }

     PyDoc_STRVAR(acquire_doc,
    @@ -105,9 +134,9 @@ Return whether the lock is in the locked

     static PyMethodDef lock_methods[] = {
           {"acquire_lock", (PyCFunction)lock_PyThread_acquire_lock,

    •        METH_VARARGS, acquire_doc},
    •        METH_VARARGS | METH_KEYWORDS, acquire_doc},
             {"acquire",      (PyCFunction)lock_PyThread_acquire_lock,
    •        METH_VARARGS, acquire_doc},
    •        METH_VARARGS | METH_KEYWORDS, acquire_doc},
             {"release_lock", (PyCFunction)lock_PyThread_release_lock,
              METH_NOARGS, release_doc},
             {"release",      (PyCFunction)lock_PyThread_release_lock,
      @@ -117,7 +146,7 @@ static PyMethodDef lock_methods[] = {
             {"locked",       (PyCFunction)lock_locked_lock,
              METH_NOARGS, locked_doc},
             {"__enter__",    (PyCFunction)lock_PyThread_acquire_lock,
    •        METH_VARARGS, acquire_doc},
    •        METH_VARARGS | METH_KEYWORDS, acquire_doc},
             {"__exit__",    (PyCFunction)lock_PyThread_release_lock,
              METH_VARARGS, release_doc},
             {NULL,           NULL}          /* sentinel */
      @@ -182,15 +211,41 @@ rlock_dealloc(rlockobject *self)
       static PyObject *
       rlock_acquire(rlockobject *self, PyObject *args, PyObject *kwds)
       {
    •       char *kwlist[] = {"blocking", NULL};
    •       char *kwlist[] = {"blocking", "timeout", NULL};
             int blocking = 1;
    •       double timeout = -1;
    •       PY_TIMEOUT_T microseconds;
             long tid;
             int r = 1;
    •       if (!PyArg_ParseTupleAndKeywords(args, kwds, "|i:acquire", kwlist,
    •                                        &blocking))
    •       if (!PyArg_ParseTupleAndKeywords(args, kwds, "|id:acquire", kwlist,

    •                                        &blocking, &timeout))
                     return NULL;

    •       if (!blocking && timeout != -1) {

    •               PyErr_SetString(PyExc_ValueError, "can't specify a timeout "

    •                               "for a non-blocking call");

    •               return NULL;

    •       }

    •       if (timeout < 0 && timeout != -1) {

    •               PyErr_SetString(PyExc_ValueError, "timeout value must be "

    •                               "strictly positive");

    •               return NULL;

    •       }

    •       if (!blocking)

    •               microseconds = 0;

    •       else if (timeout == -1)

    •               microseconds = -1;

    •       else {

    •               timeout *= 1e6;

    •               if (timeout > PY_TIMEOUT_MAX) {

    •                       PyErr_SetString(PyExc_OverflowError,

    •                                       "timeout value is too large");

    •                       return NULL;

    •               }

    •               microseconds = (PY_TIMEOUT_T) timeout;

    •       }

           tid = PyThread_get_thread_ident();
           if (self->rlock_count > 0 && tid == self->rlock_owner) {
                   unsigned long count = self->rlock_count + 1;
    @@ -205,11 +260,11 @@ rlock_acquire(rlockobject *self, PyObjec

           if (self->rlock_count > 0 ||
               !PyThread_acquire_lock(self->rlock_lock, 0)) {

    •               if (!blocking) {
    •               if (microseconds == 0) {
                             Py_RETURN_FALSE;
                     }
                     Py_BEGIN_ALLOW_THREADS
    •               r = PyThread_acquire_lock(self->rlock_lock, blocking);
    •               r = PyThread_acquire_lock_timed(self->rlock_lock, microseconds);
                     Py_END_ALLOW_THREADS
             }
             if (r) {
      @@ -1012,7 +1067,7 @@ static struct PyModuleDef threadmodule =
       PyMODINIT_FUNC
       PyInit__thread(void)
       {
    •       PyObject *m, *d;
    •       PyObject *m, *d, *timeout_max;

           /* Initialize types: */
           if (PyType_Ready(&localtype) < 0)
    @@ -1027,6 +1082,12 @@ PyInit__thread(void)
           if (m == NULL)
                   return NULL;

    •       timeout_max = PyFloat_FromDouble(PY_TIMEOUT_MAX / 1000000.);
    •       if (!timeout_max)
    •               return NULL;
    •       if (PyModule_AddObject(m, "TIMEOUT_MAX", timeout_max) < 0)
    •               return NULL;

           /* Add a symbolic constant */
           d = PyModule_GetDict(m);
           ThreadError = PyErr_NewException("_thread.error", NULL, NULL);
    diff -r 8089902215a5 Python/thread_nt.h
    --- a/Python/thread_nt.h        Fri Jan 08 18:54:23 2010 +0100
    +++ b/Python/thread_nt.h        Fri Jan 08 20:33:54 2010 +0100
    @@ -34,13 +34,13 @@ DeleteNonRecursiveMutex(PNRMUTEX mutex)
     }

     DWORD
    -EnterNonRecursiveMutex(PNRMUTEX mutex, BOOL wait)
    +EnterNonRecursiveMutex(PNRMUTEX mutex, DWORD milliseconds)
     {
           /* Assume that the thread waits successfully */
           DWORD ret ;

           /* InterlockedIncrement(&mutex->owned) == 0 means that no thread
    currently owns the mutex */

    •       if (!wait)
    •       if (!milliseconds)

    Use ==0 now that this in a real integer?

           {
                   if (InterlockedCompareExchange(&mutex->owned, 0, -1) != -1)
                           return WAIT_TIMEOUT ;
    @@ -49,7 +49,7 @@ EnterNonRecursiveMutex(PNRMUTEX mutex, B
           else
                   ret = InterlockedIncrement(&mutex->owned) ?
                           /* Some thread owns the mutex, let's wait... */

    •                       WaitForSingleObject(mutex->hevent, INFINITE) : WAIT_OBJECT_0 ;
    •                       WaitForSingleObject(mutex->hevent, milliseconds) : WAIT_OBJECT_0 ;

           mutex->thread_id = GetCurrentThreadId() ; /* We own it */
           return ret ;
    @@ -249,17 +249,34 @@ PyThread_free_lock(PyThread_type_lock aL
     * if the lock has already been acquired by this thread!
     */
     int
    +PyThread_acquire_lock_timed(PyThread_type_lock aLock, PY_TIMEOUT_T
    microseconds)
    +{

    •       int success ;
    •       PY_TIMEOUT_T milliseconds;
    •       if (microseconds >= 0) {
    •               milliseconds = (microseconds + 999) / 1000;

    Can (microseconds+999) overflow?

    •               if ((DWORD) milliseconds != milliseconds)
    •                       Py_FatalError("Timeout too large for a DWORD, "
    •                                      "please check PY_TIMEOUT_MAX");
    •       }
    •       else
    •               milliseconds = INFINITE;
    •       dprintf(("%ld: PyThread_acquire_lock_timed(%p, %lld) called\n",
    •                PyThread_get_thread_ident(), aLock, microseconds));
    •       success = aLock && EnterNonRecursiveMutex((PNRMUTEX) aLock, (DWORD)
      milliseconds) == WAIT_OBJECT_0 ;
    •       dprintf(("%ld: PyThread_acquire_lock(%p, %lld) -> %d\n",
    •                PyThread_get_thread_ident(), aLock, microseconds, success));
    •       return success;
      +}
      +int
       PyThread_acquire_lock(PyThread_type_lock aLock, int waitflag)
       {
    •       int success ;
    •       dprintf(("%ld: PyThread_acquire_lock(%p, %d) called\n",
      PyThread_get_thread_ident(),aLock, waitflag));
    •       success = aLock && EnterNonRecursiveMutex((PNRMUTEX) aLock,
      (waitflag ? INFINITE : 0)) == WAIT_OBJECT_0 ;
    •       dprintf(("%ld: PyThread_acquire_lock(%p, %d) -> %d\n",
      PyThread_get_thread_ident(),aLock, waitflag, success));
    •       return success;
    •       return PyThread_acquire_lock_timed(aLock, waitflag ? -1 : 0);
       }

     void
    diff -r 8089902215a5 Python/thread_pthread.h
    --- a/Python/thread_pthread.h   Fri Jan 08 18:54:23 2010 +0100
    +++ b/Python/thread_pthread.h   Fri Jan 08 20:33:54 2010 +0100
    @@ -83,6 +83,14 @@
     #endif

    +/* We assume all modern POSIX systems have gettimeofday() */

    Famous last words. ;) (Not saying you should change anything)

    +#ifdef GETTIMEOFDAY_NO_TZ
    +#define GETTIMEOFDAY(ptv) gettimeofday(ptv)
    +#else
    +#define GETTIMEOFDAY(ptv) gettimeofday(ptv, (struct timezone *)NULL)
    +#endif
    +
    +
     /* A pthread mutex isn't sufficient to model the Python lock type
     * because, according to Draft 5 of the docs (P1003.4a/D5), both of the
     * following are undefined:
    @@ -335,34 +343,61 @@ fix_status(int status)
           return (status == -1) ? errno : status;
     }

    -int
    -PyThread_acquire_lock(PyThread_type_lock lock, int waitflag)
    +int
    +PyThread_acquire_lock_timed(PyThread_type_lock lock, PY_TIMEOUT_T microseconds)
     {
           int success;
           sem_t *thelock = (sem_t *)lock;
           int status, error = 0;

    •       struct timeval tv;
    •       struct timespec ts;
    •       dprintf(("PyThread_acquire_lock(%p, %d) called\n", lock, waitflag));
    •       dprintf(("PyThread_acquire_lock_timed(%p, %lld) called\n",

    •                lock, microseconds));

    •       if (microseconds > 0) {

    •               GETTIMEOFDAY(&tv);

    •               tv.tv_usec += microseconds % 1000000;

    •               tv.tv_sec += microseconds / 1000000;

    •               tv.tv_sec += tv.tv_usec / 1000000;

    •               tv.tv_usec %= 1000000;

    •               ts.tv_sec = tv.tv_sec;

    •               ts.tv_nsec = tv.tv_usec * 1000;

    •       }
             do {

    •               if (waitflag)
    •               if (microseconds > 0)
    •                       status = fix_status(sem_timedwait(thelock, &ts));
    •               else if (microseconds == 0)
    •                       status = fix_status(sem_trywait(thelock));
    •               else
                             status = fix_status(sem_wait(thelock));
    •               else

    •                       status = fix_status(sem_trywait(thelock));
             } while (status == EINTR); /* Retry if interrupted by a signal */

    •       if (waitflag) {

    •       if (microseconds > 0) {
    •               if (status != ETIMEDOUT)
    •                       CHECK_STATUS("sem_timedwait");
    •       }
    •       else if (microseconds == 0) {
    •               if (status != EAGAIN)
    •                       CHECK_STATUS("sem_trywait");
    •       }
    •       else {
                     CHECK_STATUS("sem_wait");
    •       } else if (status != EAGAIN) {
    •               CHECK_STATUS("sem_trywait");
             }

           success = (status == 0) ? 1 : 0;

    •       dprintf(("PyThread_acquire_lock(%p, %d) -> %d\n", lock, waitflag, success));
    •       dprintf(("PyThread_acquire_lock_timed(%p, %lld) -> %d\n",
    •                lock, microseconds, success));
             return success;
       }

    +int
    +PyThread_acquire_lock(PyThread_type_lock lock, int waitflag)
    +{

    •       return PyThread_acquire_lock_timed(lock, waitflag ? -1 : 0);
      +}

     void
     PyThread_release_lock(PyThread_type_lock lock)
     {
    @@ -430,40 +465,70 @@ PyThread_free_lock(PyThread_type_lock lo
           free((void *)thelock);
     }

    -int
    -PyThread_acquire_lock(PyThread_type_lock lock, int waitflag)
    +int
    +PyThread_acquire_lock_timed(PyThread_type_lock lock, PY_TIMEOUT_T microseconds)
     {
           int success;
           pthread_lock *thelock = (pthread_lock *)lock;
           int status, error = 0;

    •       dprintf(("PyThread_acquire_lock(%p, %d) called\n", lock, waitflag));
    •       dprintf(("PyThread_acquire_lock_timed(%p, %lld) called\n",
    •                lock, microseconds));

           status = pthread_mutex_lock( &thelock->mut );
           CHECK_STATUS("pthread_mutex_lock[1]");
           success = thelock->locked == 0;

    •       if ( !success && waitflag ) {
    •       if (!success && microseconds != 0) {
    •               struct timeval tv;
    •               struct timespec ts;
    •               if (microseconds > 0) {
    •                       GETTIMEOFDAY(&tv);
    •                       tv.tv_usec += microseconds % 1000000;
    •                       tv.tv_sec += microseconds / 1000000;
    •                       tv.tv_sec += tv.tv_usec / 1000000;
    •                       tv.tv_usec %= 1000000;
    •                       ts.tv_sec = tv.tv_sec;
    •                       ts.tv_nsec = tv.tv_usec * 1000;

    Pull this into a helper function so it's not duplicated between the
    sem and mutex implementations?

    •               }
                     /* continue trying until we get the lock */

                   /* mut must be locked by me -- part of the condition
                    * protocol */

    •               while ( thelock->locked ) {
    •                       status = pthread_cond_wait(&thelock->lock_released,
    •                                                  &thelock->mut);
    •                       CHECK_STATUS("pthread_cond_wait");
    •               while (thelock->locked) {
    •                       if (microseconds > 0) {
    •                               status = pthread_cond_timedwait(
    •                                       &thelock->lock_released,
    •                                       &thelock->mut, &ts);
    •                               if (status == ETIMEDOUT)
    •                                       break;
    •                               CHECK_STATUS("pthread_cond_timed_wait");
    •                       }
    •                       else {
    •                               status = pthread_cond_wait(
    •                                       &thelock->lock_released,
    •                                       &thelock->mut);
    •                               CHECK_STATUS("pthread_cond_wait");
    •                       }
                     }
    •               success = 1;
    •               success = (status == 0);
             }
             if (success) thelock->locked = 1;
             status = pthread_mutex_unlock( &thelock->mut );
             CHECK_STATUS("pthread_mutex_unlock[1]");

           if (error) success = 0;

    •       dprintf(("PyThread_acquire_lock(%p, %d) -> %d\n", lock, waitflag, success));
    •       dprintf(("PyThread_acquire_lock_timed(%p, %lld) -> %d\n",
    •                lock, microseconds, success));
             return success;
       }

    +int
    +PyThread_acquire_lock(PyThread_type_lock lock, int waitflag)
    +{

    •       return PyThread_acquire_lock_timed(lock, waitflag ? -1 : 0);
      +}

     void
     PyThread_release_lock(PyThread_type_lock lock)
     {

    @pitrou
    Copy link
    Member Author

    pitrou commented Mar 21, 2010

    Here is a new patch fixing most of your comments.

    A couple of answers:

    I believe we can support arbitrary values here, subject to floating
    point rounding errors, by calling lock-with-timeout in a loop. I'm not
    sure whether that's a good idea, but it fits better with python's
    arbitrary-precision ints.

    I'm a bit wary of this, because we can't test it properly.

    •    task_handler.join(1e100)
      
    •    task_handler.join()
      

    Why is this change here? (Mostly curiosity)

    Because 1e100 would raise OverflowError :)

    •           if (timeout \> PY_TIMEOUT_MAX) {
      

    I believe it's possible for this comparison to return false, but for
    the conversion to PY_TIMEOUT_T to still overflow:

    Ok, I've replaced it with the following which should be ok:

    if (timeout >= (double) PY_TIMEOUT_MAX) [...]
    
    •           milliseconds = (microseconds + 999) / 1000;
      

    Can (microseconds+999) overflow?

    Indeed it can (I sincerely hoped that nobody would care...).
    I've replaced it with what might be a more appropriate construct.
    Please note that behaviour is undefined when microseconds exceeds the max timeout, though (this is the low-level C API).

    @pitrou
    Copy link
    Member Author

    pitrou commented Apr 13, 2010

    Updated patch against current py3k.
    About the timeout limit, an useful point of comparison is select(), which doesn't try to loop either: it just throws OverflowError if the specified timeout value is too large to be represented in a timeval struct.

    @jyasskin
    Copy link
    Mannequin

    jyasskin mannequin commented Apr 14, 2010

    Thanks, looks good. Sorry for the delay.

    @pitrou
    Copy link
    Member Author

    pitrou commented Apr 14, 2010

    Thank you! Checked in in r80071.

    @pitrou pitrou closed this as completed Apr 14, 2010
    @Bluehorn
    Copy link
    Mannequin

    Bluehorn mannequin commented Apr 14, 2010

    While you are at it, you might want to submit the patch from http://bugs.python.org/issue850728 as well.

    It adds timeouts for semaphores. :)

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    interpreter-core Interpreter core (Objects, Python, Grammar, and Parser dirs) stdlib Python modules in the Lib dir type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    1 participant