plistlib fails to parse bplist with 0x80 UID values

[slosleuth]

BPO	26707
Nosy	@ronaldoussoren, @serhiy-storchaka, @bigfootjon
PRs	bpo-26707: make plistlib read UID Key-Archive data #5922 bpo-26707: enable plistlib to read UID keys #12153
Files	plistlib_uid.diff: diff file with proposed UID patch issue26707.diff plistlib_uid.diff: diff file with updated UID patch cat.plist plist_hack.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ronaldoussoren'
closed_at = <Date 2019-05-15.20:15:41.815>
created_at = <Date 2016-04-07.01:34:01.338>
labels = ['3.8', 'type-feature', 'library']
title = 'plistlib fails to parse bplist with 0x80 UID values'
updated_at = <Date 2019-05-15.20:15:41.814>
user = 'https://bugs.python.org/slosleuth'

bugs.python.org fields:

activity = <Date 2019-05-15.20:15:41.814>
actor = 'serhiy.storchaka'
assignee = 'ronaldoussoren'
closed = True
closed_date = <Date 2019-05-15.20:15:41.815>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2016-04-07.01:34:01.338>
creator = 'slo.sleuth'
dependencies = []
files = ['42388', '42390', '42395', '47468', '47483']
hgrepos = []
issue_num = 26707
keywords = ['patch']
message_count = 19.0
messages = ['262974', '262984', '262985', '262999', '263000', '263003', '263016', '263021', '312987', '313189', '313240', '313246', '313253', '313751', '313752', '313764', '314586', '337062', '342599']
nosy_count = 5.0
nosy_names = ['ronaldoussoren', 'SilentGhost', 'serhiy.storchaka', 'slo.sleuth', 'bigfootjon']
pr_nums = ['5922', '12153']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue26707'
versions = ['Python 3.8']

[slosleuth]

libplist raises an invalid file exception on loading properly formed binary plists containing UID (0x80) values. The binary files were tested for form with plutil.

Comments at line 706 state the value is defined but not in use in plists, and the object is not handled. However, I find them frequently in bplists, e.g., iOS Snapchat application files. I have attached a proposed patch that I have tested on these files and can now successfully parse them with the _read_object method in the _BinaryPlistParser class.

My proposed patch is pasted below for others consideration while waiting for the issue to be resolved.

706,707c706,708
< # tokenH == 0x80 is documented as 'UID' and appears to be used for
< # keyed-archiving, not in plists.
---

    elif tokenH == 0x80: #UID
        s = self.\_get_size(tokenL)
        return self.\_fp.read(s).decode('ascii')

Thanks for your consideration.

[SilentGhost]

Here is the version of the patch suitable for the Rietveld. John, could you perhaps provide an example file that uses UID values?

Also, the code is identical to handling of 0x50 token, perhaps it could be incorporated into it.

[serhiy-storchaka]

UID is rather int than bytes. And I would use a special UID type.

According to Apple's sources [1], the size of UID data is tokenL+1, not self._get_size(tokenL).

[1] http://www.opensource.apple.com/source/CF/CF-1153.18/CFBinaryPList.c

[slosleuth]

I’m sorry, but the files in which I detected the problem cannot be circulated. I will try to create a test account on Snapchat and generate some test data, but I can’t do this anytime soon.

On Apr 7, 2016, at 12:51 AM, SilentGhost <report@bugs.python.org> wrote:

SilentGhost added the comment:

Here is the version of the patch suitable for the Rietveld. John, could you perhaps provide an example file that uses UID values?

Also, the code is identical to handling of 0x50 token, perhaps it could be incorporated into it.

----------
nosy: +SilentGhost
stage: -> patch review
versions: +Python 3.6
Added file: http://bugs.python.org/file42390/issue26707.diff

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue26707\>

---

[slosleuth]

I’m glad you found it in the Apple specification. I looked, but missed it. I would absolutely defer to you on your assessment of the decoding.

On Apr 7, 2016, at 1:46 AM, Serhiy Storchaka <report@bugs.python.org> wrote:

Serhiy Storchaka added the comment:

UID is rather int than bytes. And I would use a special UID type.

According to Apple's sources [1], the size of UID data is tokenL+1, not self._get_size(tokenL).

[1] http://www.opensource.apple.com/source/CF/CF-1153.18/CFBinaryPList.c

----------

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue26707\>

---

[slosleuth]

Based on the format specification pointed to by Serhiy, perhaps this a better patch, correcting size from previous patch submission and treating:

706,707c706,708
< # tokenH == 0x80 is documented as 'UID' and appears to be used for
< # keyed-archiving, not in plists.
---

    elif tokenH == 0x80:  # UID
        s = self.\_get_size(tokenL + 1)
        return int.from_bytes(self.\_fp.read(s), 'big')

I have compared output with OS X plutil and plistlib.load() with this patch and the values are identical for UID fields.

[ronaldoussoren]

How can you create plist files that contain UID values using Apple's APIs?

The plist library is meant to be interoperable with files created using Apple's APIs for creating plist files, and I didn't find an API that created UID values at the time.

[serhiy-storchaka]

The size of UID data is just tokenL + 1, not self._get_size(tokenL + 1).

FYI, the code for support UIDs and other types was proposed in bpo-14455, but was excluded from the final patch.

[ronaldoussoren]

Note that I'm still -1 on merging this patch because it is unclear how to create such plist files using public Apple APIs.

P.S. The low-level code for creating and reading binary plist files appears to be used for more than juist plist archives.

[Bigfootjon]

Hello,

I have attached a file extracted from the database of the 2Do App for iOS and macOS. The file contains information about tags used in the app.

plistlib cannot currently parse this file because it lacks the ability to read byte 0x80 (UID).

I believe the documentation for generating these type of files can be found at: https://developer.apple.com/documentation/foundation/nskeyedarchiver