add st_*time_ns fields to os.stat(), add ns keyword to os.*utime*(), os.*utimens*() expects a number of nanoseconds

[larryhastings]

BPO	14127
Nosy	@gvanrossum, @loewis, @gpshead, @jcea, @ncoghlan, @pitrou, @vstinner, @larryhastings, @merwok, @bitdancer, @peterjc
Files	larry.st_mtime_ns.patch.1.txt larry.st_mtime_ns.patch.2.txt larry.parsekwonly.diff.1.txt larry.st_mtime_ns.patch.3.txt larry.st_mtime_ns.patch.4.txt larry.st_mtime_ns.patch.6.txt larry.st_mtime_ns.patch.7.txt larry.st_mtime_ns.patch.8.txt larry.st_mtime_ns.patch.9.txt larry.utime.ns.1.patch: Patch adding ns= parameter to utime, futimes, and lutimes. Also removed futimens. larry.utime.ns.2.patch: Patch adding ns= parameter to utime, futimes, and lutimes. Also removed futimens. utime-hack.patch larry.utime.win32.bugfix.1.patch: Patch fixing two bugs for Windows support of utime. utime_read_time_arguments.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/larryhastings'
closed_at = <Date 2012-06-23.03:00:19.086>
created_at = <Date 2012-02-26.09:23:21.502>
labels = ['type-feature', 'library']
title = 'add st_*time_ns fields to os.stat(), add ns keyword to os.*utime*(), os.*utimens*() expects a number of nanoseconds'
updated_at = <Date 2018-01-18.02:59:27.721>
user = 'https://github.com/larryhastings'

bugs.python.org fields:

activity = <Date 2018-01-18.02:59:27.721>
actor = 'jcea'
assignee = 'larry'
closed = True
closed_date = <Date 2012-06-23.03:00:19.086>
closer = 'larry'
components = ['Library (Lib)']
creation = <Date 2012-02-26.09:23:21.502>
creator = 'larry'
dependencies = []
files = ['24838', '24859', '24874', '24875', '24884', '24908', '24951', '24952', '24954', '25354', '25356', '25448', '25451', '25452']
hgrepos = []
issue_num = 14127
keywords = ['patch']
message_count = 86.0
messages = ['154316', '154319', '154334', '154355', '154362', '154363', '154365', '154366', '154367', '154368', '154377', '154378', '154383', '154389', '154863', '154864', '154865', '154866', '154868', '154869', '154870', '154871', '154872', '154873', '154874', '155138', '155605', '155608', '155620', '155632', '155635', '155724', '155740', '155747', '155773', '155774', '155784', '155868', '155919', '155938', '155940', '155941', '155942', '155944', '155946', '155962', '156000', '156181', '156364', '156366', '156368', '156369', '156375', '157423', '158764', '158771', '159219', '159221', '159222', '159239', '159361', '159410', '159416', '159637', '159836', '159837', '159840', '159841', '159857', '159871', '159891', '159904', '159909', '159911', '159912', '159913', '159914', '159919', '159920', '159921', '159922', '159924', '159925', '160050', '160054', '163526']
nosy_count = 16.0
nosy_names = ['gvanrossum', 'loewis', 'gregory.p.smith', 'jcea', 'ncoghlan', 'pitrou', 'vstinner', 'larry', 'eric.araujo', 'Arfrever', 'r.david.murray', 'maubp', 'shaurz', 'rosslagerwall', 'python-dev', 'sbt']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue14127'
versions = ['Python 3.3']

[larryhastings]

Following on from Guido's rejection of PEP-410:
http://mail.python.org/pipermail/python-dev/2012-February/116837.html

This bug is the proposed hammering-out space for how to preserve exact metadata for st_atime / st_mtime between os.stat and os.utime.

(Yes, there's already bpo-11457 -- but that went pretty far down the rabbit hole of Decimal. I thought maybe a fresh start would be best.)

[larryhastings]

Guido proposed st_atime_ns et al. I'll make an alternate proposal.

I'd like to avoid tying ourselves to ns resolution. As MvL said: "I don't want to deal with this issue *again* in my lifetime".

If all we want is to facilitate copying the exact metadata from os.stat to os.utime, then we don't really care about conveniently modifying the timestamp. So I propose we use MvL's suggestion of a tuple of ints. Either (numerator, denominator) or (seconds, fractional_numerator, fractional_denominator). (These are mentioned in PEP-410 under the heading "Tuple of ints" as options A and B respectively.) Name the fields with the suffix "_exact" (e.g. st_mtime_exact). os.stat and its ilk produce it; os.utime and its ilk consume it. If people want to monkey with the values and do math, let 'em--consenting adults.

[loewis]

OTOH, it seems that Guido is very much in favor of hard-coding ns resolution in the API, claiming that expectations of even finer resolution are academic. While I still disagree (the *very* same argument was brought up for ms resolution ten years ago), I think that practicality beats purity here: people apparently can deal with an fixed scale much better than with a floating one. So it may be better to meet the apparent intuition of the majority than follow the purity route.

[bitdancer]

Is it totally insane to think about using a float subclass with an additional attribute instead of a tuple?

[gvanrossum]

@rdm: yes, that's insane for this purpose. The size of the solution doesn't match the size of the problem (the problem is non-existent for most people).

@larry: That seems unnecessarily general. I take full responsibility for fixing the precision at ns. And note that I *am* giving you a way out -- I'm establishing a clear pattern, and if someone really, desperately wants ps or fs, it's completely obvious how to change the API again. But I bet you a case of beer that won't happen in our lifetimes (or in Benjamin's). So the dynamic precision is just unnneeded complexity.

[larryhastings]

I suggest that publishing nanoseconds as a plain int would be a nasty API. Consider what it would do to os.utime:

    if isinstance(mtime, int):
        # must be st_mtime_ns, it's in nanoseconds, use as-is
        value = mtime
    else:
        assert isinstance(mtime, float)
        # must be st_mtime, it's in seconds, multiply by a billion
        value = mtime * 1000000000

Have we ever published an API that treated a parameter as two wildly different numbers based solely on whether the parameter was an int or a float?

[larryhastings]

(D'oh. My pseudo-code should have said

        value = int(mtime * 1000000000)

for that second assignment. Hopefully my point was still clear.)

[pitrou]

How about a separate float attribute for the fractional part (in seconds)? This gives a femtosecond precision (assuming a 50 bits mantissa). The utime() could accept a (integral part, fractional part) tuple to set a timestamp without losing bits.

(PHP does something similar, but badly, with its microtime() function:
http://www.php.net/manual/en/function.microtime.php)

[larryhastings]

@pitrou: What would that look like when passed in to os.utime?

[pitrou]

>>> st = os.stat('LICENSE')
>>> st.st_mtime
1330108216.7984242
>>> st.st_mtime_frac
0.798424152
>>> tup = st.st_mtime, st.st_mtime_frac
>>> os.utime('LICENSE', (tup, tup))

Of course, the fact that utime takes a (atime, mtime) tuple makes this a bit cumbersome.
When passed a tuple, utime would ignore the fractional part of the first element.

[loewis]

I suggest that publishing nanoseconds as a plain int would be a
nasty API. Consider what it would do to os.utime:

No, it wouldn't. Please re-read Guido's proposal. If you want to
specify nanoseconds, you have to pass the ns= parameter. My only
quibble with the specific spelling is that it invokes Godwin's law
(but I can live that that as a theoretical concern, also).

Have we ever published an API that treated a parameter as two wildly
different numbers based solely on whether the parameter was an int
or a float?

No, and Guido is on the record for objecting such APIs. Hence the
keyword parameter.

[gvanrossum]

Thanks Martin. I'd be happy with nsec instead of ns.

[larryhastings]

My mistake! That ought to teach me to bound out of bed Sunday morning with my "brilliant realization". (But it probably won't.)

I volunteer to implement this, with the new custom class for the os.stat object. I'll have it done no later than the PyCon sprints next month.

For the record, I actually prefer "ns". "s" is the SI standard abbreviation for second (as is "n" for nano-):
http://en.wikipedia.org/wiki/International_System_of_Units#Units_and_prefixes
If we're going to abbreviate, I think it's reasonable to go with their standard. But I'm willing to be overridden.