Unicode space character \u200b unrecognised a space

[pbnan]

BPO	10567
Nosy	@malemburg, @loewis, @abalkin

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2010-11-28.18:40:37.641>
created_at = <Date 2010-11-28.17:51:43.193>
labels = ['type-bug', 'invalid', 'expert-unicode']
title = 'Unicode space character \\u200b unrecognised a space'
updated_at = <Date 2010-11-28.19:42:19.332>
user = 'https://bugs.python.org/pbnan'

bugs.python.org fields:

activity = <Date 2010-11-28.19:42:19.332>
actor = 'belopolsky'
assignee = 'none'
closed = True
closed_date = <Date 2010-11-28.18:40:37.641>
closer = 'loewis'
components = ['Unicode']
creation = <Date 2010-11-28.17:51:43.193>
creator = 'pbnan'
dependencies = []
files = []
hgrepos = []
issue_num = 10567
keywords = []
message_count = 14.0
messages = ['122690', '122692', '122694', '122699', '122701', '122703', '122704', '122706', '122710', '122711', '122712', '122713', '122714', '122715']
nosy_count = 4.0
nosy_names = ['lemburg', 'loewis', 'belopolsky', 'pbnan']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue10567'
versions = ['Python 2.7', 'Python 3.2']

[pbnan]

Python:
Python 2.7 (r27:82500, Oct 20 2010, 03:21:03)
[GCC 4.5.1] on linux2

Code:
>>> c = u'\u200b'
>>> c.isspace()
False

In both 2.6, 3.1 it works.

http://www.cs.tut.fi/~jkorpela/chars/spaces.html

[SilentGhost]

It returns False on the latest py3k checkout as well.

[abalkin]

The category of U-200B was changed in Unicode 4.0.1:

"""
The main new features in Unicode 4.0.1 are the following:
...

• Changed: general category of U+200B ZERO WIDTH SPACE
  """ http://unicode.org/versions/Unicode4.0.1/

[loewis]

In 2.6, there was a manually maintained list, probably dating back to before Unicode 4.0. Python uses the following criterion for determining white space characters:

/* Returns 1 for Unicode characters having the bidirectional type
'WS', 'B' or 'S' or the category 'Zs', 0 otherwise. */

Since r75272, this is generated from the current Unicode database, and should thus be always correct.

Unless you can somehow prove that the criterion should be changed, or that Python computes it incorrectly, I'm closing this report as invalid.

[SilentGhost]

It's not just this character. isspace() is also False for \u200c and \u200d (from the same category). and \u2060, \u2800 and \ufeff

[loewis]

It's not just this character. isspace() is also False for \u200c and \u200d (from the same category). and \u2060, \u2800 and \ufeff

What reason do you have to believe that they should be classified as
whitespace, other than the web page you are quoting (which is apparently
out of date, and the fact that Python 2.6 was classifying
them this way, which are also out of date for the very same reason)?

[SilentGhost]

I'm not quoting anything. Thank you very much.

[malemburg]

Martin v. Löwis wrote:

Martin v. Löwis <martin@v.loewis.de> added the comment:

In 2.6, there was a manually maintained list, probably dating back to before Unicode 4.0.

That's not quite correct: Python 1.6.x - 2.5.x used tables for the
PyUnicode_ISSPACE() function that were created from the Unicode database.
Python 2.6.x introduced a short-cut table for ASCII whitespace, but still
reverted back to the generated tables for non-ASCII code points.

The tables were never manually maintained, but we also did not update
Python for each new Unicode version:

Python 1.6: Unicode 3.0
Python 2.0: Unicode 3.0
Python 2.1: Unicode 3.0
Python 2.2: Unicode 3.0
Python 2.3: Unicode 3.2
Python 2.4: Unicode 3.2
Python 2.5: Unicode 4.1
Python 2.6: Unicode 5.1
Python 2.7: Unicode 5.2

Python uses the following criterion for determining white space characters:

/* Returns 1 for Unicode characters having the bidirectional type
'WS', 'B' or 'S' or the category 'Zs', 0 otherwise. */

This definition has been used since Python 1.6.x.

[loewis]

I'm not quoting anything. Thank you very much.

Oops, sorry - I confused you with the OP.