Use an accurate clock in timeit

[pitrou]

BPO	13481
Nosy	@birkenfeld, @rhettinger, @pitrou, @vstinner
Files	timeit_clock.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2011-11-29.10:40:21.452>
created_at = <Date 2011-11-26.00:02:01.201>
labels = ['type-feature', 'library']
title = 'Use an accurate clock in timeit'
updated_at = <Date 2011-11-29.10:40:21.451>
user = 'https://github.com/pitrou'

bugs.python.org fields:

activity = <Date 2011-11-29.10:40:21.451>
actor = 'pitrou'
assignee = 'none'
closed = True
closed_date = <Date 2011-11-29.10:40:21.452>
closer = 'pitrou'
components = ['Library (Lib)']
creation = <Date 2011-11-26.00:02:01.201>
creator = 'pitrou'
dependencies = []
files = ['23783']
hgrepos = []
issue_num = 13481
keywords = ['patch']
message_count = 11.0
messages = ['148369', '148381', '148386', '148390', '148391', '148392', '148393', '148471', '148519', '148547', '148550']
nosy_count = 5.0
nosy_names = ['georg.brandl', 'rhettinger', 'pitrou', 'vstinner', 'neologix']
pr_nums = []
priority = 'normal'
resolution = 'rejected'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue13481'
versions = ['Python 3.3']

[pitrou]

This patch uses an accurate POSIX clock if possible in the timeit module.

[birkenfeld]

So does the accuracy outweigh using a Python function to call the actual clock function? (and usuable -> usable)

[pitrou]

Well, you only call the clock at the begining and end of a timing run, not at each iteration.

[neologix]

_clocks = ['CLOCK_PROCESS_CPUTIME_ID', 'CLOCK_MONOTONIC_RAW',
               'CLOCK_MONOTONIC', 'CLOCK_REALTIME']

Beware, we're mixing CPU time and wall-clock time:
$ ./python -c "from time import *; id = CLOCK_REALTIME; t = clock_gettime(id); sleep(1); print(clock_gettime(id) - t)"
1.0011036396026611
$ ./python -c "from time import *; id = CLOCK_PROCESS_CPUTIME_ID; t = clock_gettime(id); sleep(1); print(clock_gettime(id) - t)"
9.480300000003217e-05

Right now, timeit measures wall-clock time:
"""
On either platform, the default timer functions measure wall clock time, not the CPU time. [...] On Unix, you can use clock() to measure CPU time.
"""

With CLOCK_PROCESS_CPUTIME_ID:

• depending on the platform, we'll measure either wall-clock time or CPU time
• preemtion, blocking syscalls, etc won't be accounted for (so, for example, I/O-related tests will be meaningless)

[pitrou]

With CLOCK_PROCESS_CPUTIME_ID:

• depending on the platform, we'll measure either wall-clock time or
  CPU time

Indeed. I thought CPU time would be more useful (and that's the point of
the patch) but perhaps it breaks the spec.

• preemtion, blocking syscalls, etc won't be accounted for (so, for
  example, I/O-related tests will be meaningless)

But does it include kernel CPU time for the given process?

[neologix]

Indeed. I thought CPU time would be more useful (and that's the point
of the patch)

Ah, OK.
Then you should probably rename the issue "make timeit measure CPU time", or something like that, because I really thought this issue was about using a more accurate clock (less jitter, can't go backward, etc). And also update the documentation :-)

but perhaps it breaks the spec.

Well, I almost never use timeit so I can't make a call, but that's definitely a semantics change, and this may puzzle some users, especially since it will really depend on the OS/kernel version in use (and so it won't be documented).

But does it include kernel CPU time for the given process?

Yes. But it won't be reliable, for example, to measure the performance of a new readinto() implentation, since time spent by the process in 'S' or 'D' state won't be accounted for.

[pitrou]

> But does it include kernel CPU time for the given process?

Yes. But it won't be reliable, for example, to measure the performance
of a new readinto() implentation, since time spent by the process in
'S' or 'D' state won't be accounted for.

Then I'm not sure this is a good idea anymore.

[rhettinger]

I think this should be rejected.

[vstinner]

Are CLOCK_MONOTONIC_RAW, CLOCK_MONOTONIC and CLOCK_REALTIME more accurate than gettimeofday (time.time)?

[neologix]

Are CLOCK_MONOTONIC_RAW, CLOCK_MONOTONIC and CLOCK_REALTIME more accurate than gettimeofday (time.time)?

Actually, on Linux gettimeofday() returns CLOCK_REALTIME.
As for CLOCK_MONOTONIC{_RAW}, they're guaranteed not to go backward
(NTP and such).
But I think Antoine was referring to CPU time vs wall clock time (but
see comments above while this is probably a bad idea).

I think this should be rejected.

Agreed.

[pitrou]

Ok, closing.