[time] document that integer nanoseconds may be more precise than float

[vxgmichel]

BPO	39484
Nosy	@malemburg, @rhettinger, @mdickinson, @vstinner, @larryhastings, @serhiy-storchaka, @eryksun, @vxgmichel
Files	Comparing_division_errors_over_10_us.png comparing_errors.py Comparing_conversions_over_5_us.png comparing_conversions.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-01-29.13:02:45.749>
labels = ['3.8', '3.9', '3.10', 'extension-modules', 'type-feature', 'docs']
title = '[time] document that integer nanoseconds may be more precise than float'
updated_at = <Date 2021-03-21.21:08:52.484>
user = 'https://github.com/vxgmichel'

bugs.python.org fields:

activity = <Date 2021-03-21.21:08:52.484>
actor = 'eryksun'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation', 'Extension Modules']
creation = <Date 2020-01-29.13:02:45.749>
creator = 'vxgmichel'
dependencies = []
files = ['48880', '48881', '48882', '48883']
hgrepos = []
issue_num = 39484
keywords = []
message_count = 31.0
messages = ['360958', '360980', '361139', '361140', '361141', '361142', '361143', '361144', '361286', '361288', '361292', '361296', '361300', '361301', '361304', '361306', '361308', '361309', '361314', '361316', '361319', '361351', '361650', '361651', '361652', '361653', '361654', '361665', '361666', '361668', '361669']
nosy_count = 10.0
nosy_names = ['lemburg', 'rhettinger', 'mark.dickinson', 'vstinner', 'larry', 'stutzbach', 'docs@python', 'serhiy.storchaka', 'eryksun', 'vxgmichel']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue39484'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[vxgmichel]

On windows, the timestamps produced by time.time() often end up being equal because of the 15 ms resolution:

    >>> time.time(), time.time()
    (1580301469.6875124, 1580301469.6875124)

The problem I noticed is that a value produced by time_ns() might end up being higher then a value produced time() even though time_ns() was called before:

    >>> a, b = time.time_ns(), time.time()
    >>> a, b
    (1580301619906185300, 1580301619.9061852)
    >>> a / 10**9 <= b
    False

This break in causality can lead to very obscure bugs since timestamps are often compared to one another. Note that those timestamps can also come from non-python sources, i.e a C program using GetSystemTimeAsFileTime.

This problem seems to be related to the conversion _PyTime_AsSecondsDouble:

cpython/Python/pytime.c

Lines 460 to 461 in f1c1903

	d = (double)t;
	d /= 1e9;

    # Float produced by `time.time()`
    >>> b.hex()
    '0x1.78c5f4cf9fef0p+30'
    # Basically what `_PyTime_AsSecondsDouble` does:
    >>> (float(a) / 10**9).hex()
    '0x1.78c5f4cf9fef0p+30'
    # What I would expect from `time.time()`
    >>> (a / 10**9).hex()
    '0x1.78c5f4cf9fef1p+30'

However I don't know if this would be enough to fix all causality issues since, as Tim Peters noted in another thread:

Just noting for the record that a C double (time.time() result) isn't quite enough to hold a full-precision Windows time regardless

(https://bugs.python.org/issue19738#msg204112)

[vxgmichel]

I thought about it a bit more and I realized there is no way to recover the time in hundreds of nanoseconds from the float produced by time.time() (since the windows time currently takes 54 bits and will take 55 bits in 2028).

That means time() and time_ns() cannot be compared by converting time() to nanoseconds, but it might still make sense to compare them by converting time_ns() to seconds (which is apparently broken at the moment).

If that makes sense, a possible roadmap to tackle this problem would be:

• fix _PyTime_AsSecondsDouble so that time.time_ns() / 10**9 == time.time()
• add a warning in the documentation that one should be careful when comparing the timestamps produced by time() and time_ns()(in particular,time()` should not be converted to nanoseconds)

[vstinner]

>>> a / 10**9 <= b
    False

Try to use a/1e9 <= b.

--

The C code to get the system clock is the same for time.time() and time.time_ns(). It's only the conversion of the result which is different:

static PyObject *
time_time(PyObject *self, PyObject *unused)
{
    _PyTime_t t = _PyTime_GetSystemClock();
    return _PyFloat_FromPyTime(t);
}

static PyObject *
time_time_ns(PyObject *self, PyObject *unused)
{
    _PyTime_t t = _PyTime_GetSystemClock();
    return _PyTime_AsNanosecondsObject(t);
}

where _PyTime_t is int64_t: 64-bit signed integer.

Conversions:

static PyObject*
_PyFloat_FromPyTime(_PyTime_t t)
{
    double d = _PyTime_AsSecondsDouble(t);
    return PyFloat_FromDouble(d);
}


double
_PyTime_AsSecondsDouble(_PyTime_t t)
{
    /* volatile avoids optimization changing how numbers are rounded */
    volatile double d;

    if (t % SEC_TO_NS == 0) {
        _PyTime_t secs;
        /* Divide using integers to avoid rounding issues on the integer part.
           1e-9 cannot be stored exactly in IEEE 64-bit. */
        secs = t / SEC_TO_NS;
        d = (double)secs;
    }
    else {
        d = (double)t;
        d /= 1e9;
    }
    return d;
}

PyObject *
_PyTime_AsNanosecondsObject(_PyTime_t t)
{
    Py_BUILD_ASSERT(sizeof(long long) >= sizeof(_PyTime_t));
    return PyLong_FromLongLong((long long)t);
}

In short, time.time() = float(time.time_ns()) / 1e9.

--

The problem can be reproduced in Python:

>>> a=1580301619906185300
>>> b=a/1e9
>>> a / 10**9 <= b
False

I added time.time_ns() because we loose precision if you care about nanosecond resolution, with such "large number". float has a precision around 238 nanoseconds:

>>> import math; ulp=math.ulp(b)
>>> ulp
2.384185791015625e-07
>>> "%.0f +- %.0f" % (b*1e9, ulp*1e9)
'1580301619906185216 +- 238'

int/int and int/float don't give the same result:

>>> a/10**9
1580301619.9061854
>>> a/1e9
1580301619.9061852

I'm not sure which one is "correct". To understand the issue, you can use the next math.nextafter() function to get the next floating point towards -inf:

>>> a/10**9
1580301619.9061854
>>> a/1e9
1580301619.9061852
>>> math.nextafter(a/10**9, -math.inf)
1580301619.9061852
>>> math.nextafter(a/1e9, -math.inf)
1580301619.906185

Handling floating point numbers are hard.

Why don't use only use integers? :-)

[vstinner]

See also bpo-39277 which is similar issue. I proposed a fix which uses nextafter():
#17933

[vstinner]

Another way to understand the problem: nanosecond (int) => seconds (float) => nanoseconds (int) roundtrip looses precison.

>>> a=1580301619906185300
>>> a/1e9*1e9
1.5803016199061852e+18
>>> b=int(a/1e9*1e9)
>>> b
1580301619906185216
>>> a - b
84

The best would be to add a round parameter to _PyTime_AsSecondsDouble(), but I'm not sure how to implement it.

The following rounding mode is used to read a clock:

/* Round towards minus infinity (-inf).
   For example, used to read a clock. */
_PyTime_ROUND_FLOOR=0,

_PyTime_ROUND_FLOOR is used in time.clock_settime(), time.gmtime(), time.localtime() and time.ctime() functions: to round input arguments.

time.time(), time.monotonic() and time.perf_counter() converts _PyTime_t to float using _PyTime_AsSecondsDouble() (which currently has no round parameter) for their output.

See also my rejected PEP-410 ;-)

--

One way to solve this issue is to document how to compare time.time() and time.time_ns() timestamps in a reliable way.

[vstinner]

By the way, I wrote an article about the history on how Python rounds time... https://vstinner.github.io/pytime.html

I also wrote two articles about nanoseconds in Python:

• https://vstinner.github.io/python37-pep-564-nanoseconds.html
• https://vstinner.github.io/python37-perf-counter-nanoseconds.html

Oh and I forgot the main one: PEP-564 :-)

"Example 2: compare times with different resolution" sounds like this issue.

https://www.python.org/dev/peps/pep-0564/#example-2-compare-times-with-different-resolution

[larryhastings]

I don't think this is fixable, because it's not exactly a bug. The problem is we're running out of bits. In converting the time around, we've lost some precision. So the times that come out of time.time() and time.time_ns() should not be considered directly comparable.

Both functions, time.time() and time.time_ns(), call the same underlying function to get the current time. That function is_PyTime_GetSystemClock(); it returns nanoseconds since the 1970 epoch, stored in an int64. Each function then simply converts that time into its return format and returns that.

In the case of time.time_ns(), it loses no precision whatsoever. In the case of time.time(), it (usually) converts to double and divides by 1e9, which is implicitly floor rounding.

Back-of-the-envelope math here: An IEEE double has 53 bits of resolution for the mantissa, not counting the leading 1. The current time in seconds since the 1970 epoch uses about 29 bits of those 53 bits. That leaves 24 bits for the fractional second. But you'd need 30 bits to render all one billion fractional values. We're six bits short.

Unless anybody has an amazing suggestion about how to ameliorate this situation, I think we should close this as wontfix.