add time.monotonic() method

[kristjanvalur]

BPO	10278
Nosy	@doko42, @abalkin, @pitrou, @kristjanvalur, @vstinner, @merwok, @briancurtin, @anacrolix, @4kir4, @vadmium
Files	wallclock.patch monotonic.py clock_gettime.patch wallclock-2.patch wallclock-3.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/vstinner'
closed_at = <Date 2012-01-18.21:39:43.081>
created_at = <Date 2010-11-01.15:10:08.663>
labels = ['interpreter-core', 'type-feature']
title = 'add time.monotonic() method'
updated_at = <Date 2019-02-20.10:34:21.980>
user = 'https://github.com/kristjanvalur'

bugs.python.org fields:

activity = <Date 2019-02-20.10:34:21.980>
actor = 'vstinner'
assignee = 'vstinner'
closed = True
closed_date = <Date 2012-01-18.21:39:43.081>
closer = 'pitrou'
components = ['Interpreter Core']
creation = <Date 2010-11-01.15:10:08.663>
creator = 'kristjan.jonsson'
dependencies = []
files = ['19455', '19461', '23083', '23799', '23809']
hgrepos = []
issue_num = 10278
keywords = ['patch']
message_count = 40.0
messages = ['120130', '120131', '120141', '120144', '120146', '120149', '120153', '120154', '120192', '120193', '120205', '120207', '126223', '126248', '126274', '139965', '142792', '143205', '143206', '143209', '143364', '146349', '146351', '146355', '146357', '148523', '148530', '148596', '150861', '150888', '150938', '151504', '151506', '151559', '151570', '151666', '151858', '155819', '155826', '336059']
nosy_count = 12.0
nosy_names = ['doko', 'belopolsky', 'pitrou', 'kristjan.jonsson', 'vstinner', 'eric.araujo', 'brian.curtin', 'glenn', 'anacrolix', 'akira', 'python-dev', 'martin.panter']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue10278'
versions = ['Python 3.3']

[kristjanvalur]

If measuring time across blocking calls, such as thread synchronization, one currently must time.time(). This is because time.clock() measures cpu seconds on unix. On windows, however, time.clock() would be more appropriate because it measures wall-clock time.

To avoid having to put platform clauses everywhere, this patch adds time.wallclock().
The current implementation is a simple alias to time.clock on windows and time.time otherwise. Future improvements may add a better implementation on those non-windows platforms that support it.

[voidspace]

+1

[abalkin]

Why does this need to be in stdlib?

AFAICT, the proposed patch is just:

if <appropriate test>:
   wallclock = time.clock
else:
   wallclock = time.time

which is easy enough to stick in your measuring code or a project's utilities module.

If others really want to see this in stdlib, I would prefer to place it in a more specialized module such as profile.

[pitrou]

Well, the problem is that the "appropriate test" is not easy to guess a priori, so it would be useful for the stdlib to provide the right tool for the job.
As for where it should live, I have no strong opinion, but it's true that the time module looks appropriate.

[abalkin]

On Mon, Nov 1, 2010 at 1:43 PM, Antoine Pitrou <report@bugs.python.org> wrote:
..

Well, the problem is that the "appropriate test" is not easy to guess a priori, so it would
be useful for the stdlib to provide the right tool for the job.

This sounds like an argument against this feature, not for it. If it
is hard for the application code to implement an appropriate test "a
priori", what is the chance to get it right in stdlib?

As for where it should live, I have no strong opinion, but it's true that the time module looks appropriate.

Having time.time and time.clock is already confusing enough. Having
the third function which is either the first or the second depending
on some unspecified criterion does not strike me as a clean design.

[pitrou]

> Well, the problem is that the "appropriate test" is not easy to guess a priori, so it would
> be useful for the stdlib to provide the right tool for the job.

This sounds like an argument against this feature, not for it. If it
is hard for the application code to implement an appropriate test "a
priori", what is the chance to get it right in stdlib?

The point of a standard library is to bring together competence and
experience to build a common ground of useful functions. If we
restricted ourselves to easy things then 75% of the stdlib should be
ripped out.

> As for where it should live, I have no strong opinion, but it's true that the time module looks appropriate.

Having time.time and time.clock is already confusing enough. Having
the third function which is either the first or the second depending
on some unspecified criterion does not strike me as a clean design.

The problem is time.clock(), since it does two wildly different things
depending on the OS.
I would suggest to deprecate time.clock() at the same time as we add
time.wallclock(). For the Unix-specific definition of time.clock(),
there is already os.times() (which gives even richer information).

[abalkin]

On Mon, Nov 1, 2010 at 2:09 PM, Antoine Pitrou <report@bugs.python.org> wrote:
..

> > Well, the problem is that the "appropriate test" is not easy to guess a priori, so it would
> > be useful for the stdlib to provide the right tool for the job.
>
> This sounds like an argument against this feature, not for it.  If it
> is hard for the application code to implement an appropriate test "a
> priori", what is the chance to get it right in stdlib?

The point of a standard library is to bring together competence and
experience to build a common ground of useful functions. If we
restricted ourselves to easy things then 75% of the stdlib should be
ripped out.

It looks like I misunderstood what you said. I thought "a priory"
meant without knowing the details of the application rather than "by a
novice."

[abalkin]

On Mon, Nov 1, 2010 at 2:09 PM, Antoine Pitrou <report@bugs.python.org> wrote:
..

The problem is time.clock(), since it does two wildly different things
depending on the OS.
I would suggest to deprecate time.clock() at the same time as we add
time.wallclock(). For the Unix-specific definition of time.clock(),
there is already os.times() (which gives even richer information).

+1, but doing something like that should be discussed on python-ideas
first. We should also weigh this against other proposals such as
exposing gettimeofday.

[kristjanvalur]

Certainly.
I was going to put this simple code in time.py when I realized that time was a C module.

The main point, as Antoine points out, is that time.clock() means two seriously different things on the two main platforms, and time.clock() is potentially inadequate on one of them.

I put in the patch since it was quick to do, but I'll provoke a discussion on python-ideas for now.

[abalkin]

2010/11/1 Kristján Valur Jónsson <report@bugs.python.org>:
..

I put in the patch since it was quick to do, but I'll provoke a discussion
on python-ideas for now.

I am looking forward to it. You may find reviewing the following
issues helpful for your case: bpo-9079, bpo-2736, and bpo-9528.

[vstinner]

I would also be nice to have a "monotonic" clock in Python. See monotonic.py for an example. But I don't know if it solves the same problem than time.wallclock() or not. I need a monotonic clock for a server on which NTP is sometimes called to resynchronize the system clock (and the NTP server may be change by the user).

[kristjanvalur]

Well, that is sort of what I'm trying to achieve. Note that you probably want to use QueryPerformaceCounter on windows (or simply time.clock()) or at least GetTickCount64 which doesn't wrap around after 50 days :). Also, I'm trying to get the highest resolution possible...

[glenn]

I agree with Victor: Python should provide a function to supply monotonic time, which is what's really wanted for measuring time deltas. Far too many applications incorrectly use the system clock for this, and Python makes this worse by not providing any standard library function to allow people to do this correctly.

In Windows, it should probably use GetTickCount64 if available, otherwise GetTickCount with logic to handle wrapping. I think QueryPerformanceCounter is problematic as a general-purpose timer: depending on the hardware and Windows version, it can be offset differently across CPUs, and may not be reliable on some processors. It may be fixed in Vista or Win7, I'm not sure; if so it's much higher resolution than GTC.

[anacrolix]

This is sorely needed. IMO the current behaviour of time.clock works for Windows, and clock_gettime(CLOCK_MONOTONIC) on POSIX or clock_gettime(CLOCK_MONOTONIC_RAW) on Linux>=2.6.28.

There are some related discussions on StackOverflow that may contain useful ideas also:
http://stackoverflow.com/questions/1205722/how-do-i-get-monotonic-time-durations-in-python
http://stackoverflow.com/questions/4687408/retrieve-wall-time-in-python-using-the-standard-library

[briancurtin]

In Windows, it should probably use GetTickCount64 if available,
otherwise GetTickCount with logic to handle wrapping. I think
QueryPerformanceCounter is problematic as a general-purpose timer:
depending on the hardware and Windows version, it can be offset
differently across CPUs, and may not be reliable on some processors. It
may be fixed in Vista or Win7, I'm not sure; if so it's much higher
resolution than GTC.

I don't have a ton of experience with this, but by creating a synchronization point based on when the system clock changes, QueryPerformanceCounter can be pretty accurate and as you already stated, and higher resolution than most other solutions. http://msdn.microsoft.com/en-us/magazine/cc163996.aspx is where I got the idea and most of the implementation from. I'm essentially using the code from Figure 2 in a C extension right now to do something similar to threading.Timer in a performance testing tool.

[anacrolix]

What's the status of this bug? This is a very useful feature, I've had to use and add bindings to monotonic times for numerous applications. Can it make it into 3.3?

[vstinner]

See also bpo-12822.

[vstinner]

I think that we should process in two steps:

• Expose low level C functions
• Write a high level reusing the best low level function depending on the OS

Low level functions:

• Expose clock_gettime() into the time module, with CLOCK_MONOTONIC and CLOCK_MONOTONIC_RAW constants, if available. The wrapper should be thin, so the result would be a tuple (sec, nsec) (to avoid issues with floating point numbers)
• Windows: GetTickCount, GetTickCount64 and QueryPerformaceCounter are monotonic. GetTickCount wraps after 50 days, so GetTickCount64 should be preferred. GetTickCount(64) returns a number of milliseconds. It looks like QueryPerformaceCounter has a better resolution than GetTickCount(64). I don't know when QueryPerformaceCounter does wrap? QueryPerformaceCounter is already exposed as time.clock().

High level:

• Use clock_gettime(CLOCK_MONOTONIC_RAW) on Linux>=2.6.28
• Use clock_gettime(CLOCK_MONOTONIC) on UNIX
• Use time.clock() (QueryPerformaceCounter) on Windows?
• If none of these functions is available, don't define the function

I propose time.monotonic() because I saw this name in different projects.

Pseudo-code for the time module:

monotonic = None
if hasattr(time, 'clock_gettime'):
  if hasattr(time, 'CLOCK_MONOTONIC_RAW'):
    def monotonic(): return time.clock_gettime(CLOCK_MONOTONIC_RAW)
  else:
    # i don't think that we should expose clock_gettime
    # if CLOCK_MONOTONIC is not available (or the function is useless)
    def monotonic(): return time.clock_gettime(CLOCK_MONOTONIC)
elif os.name == 'nt':
    def monotonic(): return time.clock()
if monotonic is not None:
   monotonic.__doc__ = 'monotonic time'
else:
   del monotonic

[vstinner]

Note: it would be very pratical if time.monotonic() has always the same unit on all platforms. clock_gettime() uses seconds (and nanoseconds), time.clock() uses also seconds.

[kristjanvalur]

The problem with QueryPerformanceCounter is that it drifts. It has high resolution, but can drift far out of sync with GetTickCount64.
The best solutions on windows combine the two, but that's tricky to impolement.

QPC will wrap, but only after a long time. its 64 bits, and with a frequency of 1GHz, that takes some 600 years.
Of course, with 10GHz we're down to 60 years, but by that time, we will have python 2.8

[vstinner]

clock_gettime.patch: add time.clock_gettime() function and time.CLOCK_xxx constants. The patch requires to rerun "autoconf".

For the documentation, I don't know the availability of this function. Is it available on Windows? CLOCK_REALTIME doc contains "Setting this clock requires appropriate privileges": this sentence might be removed if we don't expose clock_settime.

The constants are not defined if the function is not available.

timemodule.c and datetimemodule.c are no more linked to libm. I don't know why libm was used? Copy/paste failure?

On Linux, clock_gettime() requires librt. I chose to check for librt in configure.in. To get this info in setup.py, I close to use the TIMEMODULE_LIBS define (in pyconfig.h). I don't know if there is something simpler.

time.clock_gettime() returns a float. It would be nice to keep nanoseconds as an integer, but I chose a float to mimic other time functions. If we need nanosecond resolution, a new function can be added.

The unit test is very simple, but less than time.clock() test :-)

[vstinner]

The issue bpo-13261 has been marked as a duplicate of this issue. Copy of msg146347:
-------------
time.clock () has very poor time resolution on Linux (tested on Ubuntu 11.04).

The result of call to clock () changes once per several seconds. On the other side, on Windows it provides very good resolution.

Here is a doctest that fails on Linux:
"""
 >>> from time import sleep
 >>> prev = clock ()
 >>> res = True
 >>> for i in xrange(10):
 ...     sleep(0.15)
 ...     next = clock ()
 ...     res = res and (next - prev) > 0.1
 ...     prev = next
 >>> print res
 True
"""

Currently on Linux I am using a workaround that is attached to the issue.
-------------

The issue has also a script:
http://bugs.python.org/file23515/monotime_unix.py

[pitrou]

CLOCK_MONOTONIC, CLOCK_PROCESS_CPUTIME_ID and CLOCK_THREAD_CPUTIME_ID are optional according to POSIX, which only mandates CLOCK_REALTIME. You should mention it in the docs.

You might also want to export clock_getres():
http://pubs.opengroup.org/onlinepubs/9699919799/functions/clock_getres.html

[python-dev]

New changeset 35e4b7c4bafa by Victor Stinner in branch 'default':
Close bpo-10278: Add clock_getres(), clock_gettime() and CLOCK_xxx constants to
http://hg.python.org/cpython/rev/35e4b7c4bafa

[vstinner]

I closed maybe this issue too quickly. My commit doesn't solve the initial issue: Python doesn't provide a portable "wallclock" function.

wallclock.patch should be updated to use:

• time.clock() on Windows (use QueryPerformanceCounter)
• or time.clock_gettime(CLOCK_MONOTONIC_RAW) if available
• or time.clock_gettime(CLOCK_MONOTONIC) if available
• or time.time() (which is usually gettimeofday())

Pseudo-code:

wallclock = None
if hasattr(time, 'clock_gettime') and hasattr(time, 'CLOCK_MONOTONIC_RAW'):
  # I understood that CLOCK_MONOTONIC_RAW is more reliable
  # than CLOCK_MONOTONIC, because CLOCK_MONOTONIC may be adjusted
  # by NTP. I don't know if it's correct.
  def wallclock():
    return time.clock_gettime(CLOCK_MONOTONIC_RAW)

elif hasattr(time, 'clock_gettime') and hasattr(time, 'CLOCK_MONOTONIC'):
def wallclock():
return time.clock_gettime(CLOCK_MONOTONIC)

elif os.name == 'nt':
# define a new function to be able to set its docstring
def wallclock():
return time.clock()

else:
def wallclock():
return time.time()
if wallclock is not None:
wallclock.__doc__ = 'monotonic time'
else:
del wallclock

wallclock() doc should also explain that time.time() may be adjusted by NTP or manually by the administrator.

--

By the way, it would be nice to expose the precision of time.clock(): it's 1/divisor or 1/CLOCKS_PER_SEC on Windows, 1/CLOCKS_PER_SEC on UNIX.

[vstinner]

wallclock-2.patch: implement wallclock() using the new clock_getime(CLOCK_MONOTONIC) function (or other, depending on the OS).

I removed description on how the function is implemented from the doc.

[pitrou]

As discussed elsewhere, there should be a fallback when clock_gettime() fails for the given clock.

[vstinner]

Version 3 of the patch: check for clock_gettime() failure. Try CLOCK_MONOTONIC_RAW, or try CLOCK_MONOTONIC, or use time.time().

[doko42]

on linux the underlying functionality is implemented in librt; the extension doesn't check for this or links with -lrt.

[vstinner]

on linux the underlying functionality is implemented in librt; the extension doesn't check for this or links with -lrt.

The changeset 35e4b7c4bafa changed configure.in to check clock_gettime(). It checks without and with librt:

 7.7 +AC_CHECK_FUNCS(clock_gettime, [], [
 7.8 +    AC_CHECK_LIB(rt, clock_gettime, [
 7.9 +        AC_DEFINE(HAVE_CLOCK_GETTIME, 1)
7.10 +        AC_DEFINE(TIMEMODULE_LIB, [rt],
7.11 +                  [Library needed by timemodule.c: librt may be needed for clock_gettime()])
7.12 +    ])
7.13 +])

wallclock-3.patch checks for HAVE_CLOCK_GETTIME, which does use librt in its test (see above).

[doko42]

then something like this is missing?

diff -r ca2a35140e6a Modules/Setup.dist
--- a/Modules/Setup.dist	Mon Jan 09 06:17:39 2012 +0000
+++ b/Modules/Setup.dist	Mon Jan 09 15:25:05 2012 +0100
@@ -166,7 +166,7 @@
 #cmath cmathmodule.c _math.c # -lm # complex math library functions
 #math mathmodule.c _math.c # -lm # math library functions, e.g. sin()
 #_struct _struct.c	# binary structure packing/unpacking
-#time timemodule.c # -lm # time operations and variables
+#time timemodule.c $(TIMEMODULE_LIB) # -lm # time operations and variables
 #_weakref _weakref.c	# basic weak reference support
 #_testcapi _testcapimodule.c    # Python C API test module
 #_random _randommodule.c	# Random number generator

[python-dev]

New changeset bb10cd354e49 by Victor Stinner in branch 'default':
Close bpo-10278: Add time.wallclock() function, monotonic clock.
http://hg.python.org/cpython/rev/bb10cd354e49

[python-dev]

New changeset 1de276420470 by Victor Stinner in branch 'default':
Issue bpo-10278: fix a typo in the doc
http://hg.python.org/cpython/rev/1de276420470

[pitrou]

There are some issue on the Windows buildbots:

======================================================================
FAIL: test_wallclock (test.test_time.TimeTestCase)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\test\test_time.py", line 340, in test_wallclock
    self.assertAlmostEqual(t, 0.1, places=2)
AssertionError: 0.09138312271531701 != 0.1 within 2 places

---

[python-dev]

New changeset 83e8c3a6a81c by Antoine Pitrou in branch 'default':
Be more lenient in test_wallclock (issue bpo-10278).
http://hg.python.org/cpython/rev/83e8c3a6a81c

[python-dev]

New changeset 8502a9236c2e by Victor Stinner in branch 'default':
Issue bpo-10278: Be more explicit in tests than wallclock() is monotonic (cannot
http://hg.python.org/cpython/rev/8502a9236c2e

[python-dev]

New changeset fb0f4fe8123e by Victor Stinner in branch 'default':
Issue bpo-10278: wallclock() cannot go backward, but two consecutive calls
http://hg.python.org/cpython/rev/fb0f4fe8123e

[python-dev]

New changeset c11946846474 by Victor Stinner in branch 'default':
Issue bpo-10278: Drop time.monotonic() function, rename time.wallclock() to time.steady()
http://hg.python.org/cpython/rev/c11946846474

[python-dev]

New changeset 27441e0d6a75 by Victor Stinner in branch 'default':
Issue bpo-10278: Add an optional strict argument to time.steady(), False by default
http://hg.python.org/cpython/rev/27441e0d6a75

[vstinner]

This issue has been followed by the PEP-418 which added time.monotonic(), time.perf_counter() and time.process_time().