plistlib can't decode date from year 0

[shields-fn]

BPO	41083
Nosy	@ronaldoussoren, @tiran, @ned-deily, @shields-fn

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-06-22.22:42:45.413>
labels = ['OS-mac', '3.8', 'type-bug', 'library']
title = "plistlib can't decode date from year 0"
updated_at = <Date 2020-06-23.17:24:51.579>
user = 'https://github.com/shields-fn'

bugs.python.org fields:

activity = <Date 2020-06-23.17:24:51.579>
actor = 'shields-fn'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'macOS']
creation = <Date 2020-06-22.22:42:45.413>
creator = 'shields-fn'
dependencies = []
files = []
hgrepos = []
issue_num = 41083
keywords = []
message_count = 5.0
messages = ['372131', '372151', '372152', '372159', '372185']
nosy_count = 4.0
nosy_names = ['ronaldoussoren', 'christian.heimes', 'ned.deily', 'shields-fn']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue41083'
versions = ['Python 3.8']

[shields-fn]

On macOS 10.5.5:

/tmp $ defaults export com.apple.security.KCN -
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"\>
<plist version="1.0">
<dict>
<key>absentCircleWithNoReason</key>
<false/>
<key>applicationDate</key>
<date>0000-12-30T00:00:00Z</date>
<key>lastCircleStatus</key>
<integer>-1</integer>
<key>lastWritten</key>
<date>2019-10-15T17:23:33Z</date>
<key>pendingApplicationReminder</key>
<date>4001-01-01T00:00:00Z</date>
<key>pendingApplicationReminderInterval</key>
<integer>86400</integer>
</dict>
</plist>
/tmp $ cat plist_date_reduction.py
#!/usr/bin/env python3

import plistlib
import subprocess

if __name__ == "__main__":
    plist = subprocess.check_output(["defaults", "export", "com.apple.security.KCN", "-"])
    print(plistlib.loads(plist, fmt=plistlib.FMT_XML))
/tmp $ python3.8 plist_date_reduction.py 
Traceback (most recent call last):
  File "plist_date_reduction.py", line 8, in <module>
    print(plistlib.loads(plist, fmt=plistlib.FMT_XML))
  File "/usr/local/Cellar/python@3.8/3.8.3/Frameworks/Python.framework/Versions/3.8/lib/python3.8/plistlib.py", line 1000, in loads
    return load(
  File "/usr/local/Cellar/python@3.8/3.8.3/Frameworks/Python.framework/Versions/3.8/lib/python3.8/plistlib.py", line 992, in load
    return p.parse(fp)
  File "/usr/local/Cellar/python@3.8/3.8.3/Frameworks/Python.framework/Versions/3.8/lib/python3.8/plistlib.py", line 288, in parse
    self.parser.ParseFile(fileobj)
  File "/private/tmp/python@3.8-20200527-50093-16hak5w/Python-3.8.3/Modules/pyexpat.c", line 461, in EndElement
  File "/usr/local/Cellar/python@3.8/3.8.3/Frameworks/Python.framework/Versions/3.8/lib/python3.8/plistlib.py", line 300, in handle_end_element
    handler()
  File "/usr/local/Cellar/python@3.8/3.8.3/Frameworks/Python.framework/Versions/3.8/lib/python3.8/plistlib.py", line 376, in end_date
    self.add_object(_date_from_string(self.get_data()))
  File "/usr/local/Cellar/python@3.8/3.8.3/Frameworks/Python.framework/Versions/3.8/lib/python3.8/plistlib.py", line 254, in _date_from_string
    return datetime.datetime(*lst)
ValueError: year 0 is out of range

[tiran]

There is no year 0. It does not exist. The year after 1 BC is the year 1 AD.

[tiran]

By the way most datetime libraries will give you incorrect values for dates before 1582, 1752, 1926, 1949 or any dates in that range depending on your country and the predominant religion of your country, county, state, or principality. Dates before 1970 are generally problematic unless the date format also references a calendar system.

It's ... messy.

[ronaldoussoren]

Year 0 does exist in ISO 8601 though, but that wouldn't help us here as year 0 in that standard is year 1 BCE which is not representable in Python's datetime module.

I'm not sure what we can do about this. The best we could do with plistlib is probably to add an option to either set unrepresentable dates to None or represent those dates as strings.

A more comprehensive fix is to change datetime to be able to represent these dates, but that's a much larger change that likely requires a PEP.

[shields-fn]

You're correct that there is no year 0, but as you see Apple does use <date>0000-12-30T00:00:00Z</date> in their plists. I did not set that in order to test plistlib; it's what I found on my system.

If it's a goal that plistlib be able to parse system-generated plists and round-trip them to an equivalent serialization -- and I think that should be a goal -- then using strings or None also won't work. Maybe there could be a plistlib.Datetime for dates which are outside what datetime can represent?

[zitterbewegung]

This hasn't been accepted and opened still for three years and also is likely a change that would require a PEP and no PEP addressing this hasn't been created. IMHO this should be closed.

[dgelessus]

Jumping into the discussion here... I looked into this a few months ago and subscribed to this issue, but never actually commented.

The short explanation is: according to Apple, this date is actually 01 January 0001 AD/CE. If you use Apple's plutil tool to print the contents of the plist, you'll see that the date written in the XML as 0000-12-30T00:00:00Z is printed as 0001-01-01 00:00:00:

plutil -p com.apple.security.KCN.plist
{
  "absentCircleWithNoReason" => 0
  "applicationDate" => 0001-01-01 00:00:00 +0000
  "lastCircleStatus" => -1
  "lastWritten" => 2019-10-15 17:23:33 +0000
  "pendingApplicationReminder" => 4001-01-01 00:00:00 +0000
  "pendingApplicationReminderInterval" => 86400
}

I don't remember all the messy details, but the Apple/macOS date representation (NSDate) works in an unusual way before 15 October 1582. I think it switches to the Julian calendar instead of using proleptic Gregorian like most other code, but I'm not sure.

Regardless of how it's written, this particular date is relevant because it's the value of NSDate.distantPast. It's basically the Apple equivalent of datetime.datetime.min. It would be helpful if this date value could be round-tripped somehow (especially considering that it appears in a standard macOS system plist), even if it can't be represented as a normal datetime object.

re. the last comment: extending the range of datetime would probably require a PEP, but that's not the only way to solve this issue. The original reporter suggested some realistic alternatives how round-tripping could be implemented.

[zitterbewegung]

Sorry didn't catch the part of changing behavior that would be modifying current behavior by changing round-tripping.

[ronaldoussoren]

This issue should IMHO not be closed.