Unicode char 304 in lowercase has len = 2

[KirilDimitrov]

BPO	33108
Nosy	@vstinner, @ezio-melotti, @stevendaprano, @methane, @serhiy-storchaka, @animalize
Superseder	bpo-17252: Latin Capital Letter I with Dot Above

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-03-22.06:09:55.919>
created_at = <Date 2018-03-20.13:22:06.518>
labels = ['type-bug', 'expert-unicode']
title = 'Unicode char 304 in lowercase has len = 2'
updated_at = <Date 2018-03-22.06:09:55.891>
user = 'https://bugs.python.org/KirilDimitrov'

bugs.python.org fields:

activity = <Date 2018-03-22.06:09:55.891>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2018-03-22.06:09:55.919>
closer = 'serhiy.storchaka'
components = ['Unicode']
creation = <Date 2018-03-20.13:22:06.518>
creator = 'Kiril Dimitrov'
dependencies = []
files = []
hgrepos = []
issue_num = 33108
keywords = []
message_count = 7.0
messages = ['314142', '314143', '314146', '314150', '314230', '314232', '314234']
nosy_count = 7.0
nosy_names = ['vstinner', 'ezio.melotti', 'steven.daprano', 'methane', 'serhiy.storchaka', 'malin', 'Kiril Dimitrov']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '17252'
type = 'behavior'
url = 'https://bugs.python.org/issue33108'
versions = ['Python 3.6']

[KirilDimitrov]

>>> chr(304)
'İ'
>>> chr(304).lower()
'i̇'
>>> len(chr(304).lower())
2

This breaks unicode text matching. There is no other unicode character with the same behaviour (in 3.6.2 and 3.6.4).

[methane]

Another example:

>>> s = "ß"
>>> len(s)
1
>>> len(s.upper())
2
>>> s.upper()
'SS'
>>> ord(s)
223

This breaks unicode text matching.

What do you talking about? re module?

[KirilDimitrov]

This is roughly my use case:
zip( "ßx", [0.5, 0.3]) is [('ß', 0.5), ('x', 0.3)]
zip("ßx".upper(), [0.5, 0.3]) will be [('S', 0.5), ('S', 0.3)] in later
case you never get to see the value for 'x'.

At least my expectation was that lower and upper should preserve text
length. At least this seemed to be the case in python2.7

2018-03-20 15:28 GMT+02:00 INADA Naoki <report@bugs.python.org>:

INADA Naoki <songofacandy@gmail.com> added the comment:

Another example:

>>> s = "ß"
>>> len(s)
1
>>> len(s.upper())
2
>>> s.upper()
'SS'
>>> ord(s)
223

> This breaks unicode text matching.

What do you talking about? re module?

----------
nosy: +inada.naoki

---

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue33108\>

---

[stevendaprano]

It has never been the case that upper() or lower() are guaranteed to preserve string length in Unicode. For example, some characters decompose into a base plus combining characters. Ligatures are another example. See here for more details:

https://unicode.org/faq/casemap_charprop.html

However, this example surprises me. In Python 2, I get the result I expected:

py> c = unichr(304)
py> unicodedata.name(c)
'LATIN CAPITAL LETTER I WITH DOT ABOVE'
py> unicodedata.name(c.lower())
'LATIN SMALL LETTER I'

If I am reading the UnicodeData.txt file correctly, I think that the right behaviour is for LATIN CAPITAL LETTER I WITH DOT ABOVE to lowercase to LATIN SMALL LETTER I, as it did in Python 2.

ftp://ftp.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt

[methane]

Maybe, we should update UnicodeData?

[animalize]

There was a discussion about "Latin Capital Letter I with Dot Above"
https://bugs.python.org/issue17252

[serhiy-storchaka]

Thank you Ma Lin.

Closed as a duplicate of bpo-17252.