Some character disappear on mac os x

[sportica]

std::isspace() of mac os x recognize 0xa0 as whitespace.
I think that it's a bug.
http://en.cppreference.com/w/cpp/string/byte/isspace

'죠' is 0xec, 0xa3, 0xa0
'고' 0xea, 0xb3, 0xa0
Both '죠' and '고' have 0xa0.

Perhaps more character will be affected.

[jmarshallnz]

A better bet might be to just hit the ones we know we want?

@Karlson2k ?

[Karlson2k]

Don't break things that already broken. :)
Actually it's incorrect to use any single-byte char function with utf-8 bytes.
You can fix it by:

• detecting UTF-8 char length and checking only first byte of each UTF-8 char by ::isspace() in TrimX functions
• convert string to wide character and use ::iswspace() in TrimX functions
• use our CRegExp class in UTF-8 mode in TrimX funtions

[sportica]

Only single byte within UTF-8 characters could be whitespace.
Because UTF-8 was designed for backward compatibility with ASCII.
http://en.wikipedia.org/wiki/UTF-8

How about this commit, @Karlson2k ?

[Karlson2k]

@sportica UTF-8 is compatible backward with US-ASCII only, not ASCII in general. Unicode has much more whitespace characters than US-ASCII.
As temporal quick fix, I'd remove this helper and rewrite TrimX functions with direct tests like if ((c & 0x80) == 0 && ::isspace(c)) with comment like //TODO: support all UNICODE whitespace chars

[sportica]

Okay. Thank you Karlson2k.

[sportica]

Now use std::ctype functions instead of isspace(). std::ctype functions determine white spaces by system locale. How about this way?

[sportica]

How about writing own function, @Karlson2k ?

[sportica]

@Karlson2k It is based on your second suggestion. But only convert necessary amount, not whole string. Convert one character at once, and compare it.

[jmarshallnz]

IMO I'd just implement using if ((c & 0x80) == 0 && ::isspace(c)) and keep the find_if as it is. We really only want to trim space, tab, linefeeds and carriage returns in these functions, not every possible whitespace character under the sun.

[sportica]

Please let me know if you know suitable STL. It seems that there is no template for our case. Or we should keep helper function.

[jmarshallnz]

The original? With the simple conditional on a char you have a function that takes a char and returns a bool. Exactly what the isspace_c() function does.

[sportica]

Please check again.

[jmarshallnz]

@Karlson2k mind taking a look?

[Karlson2k]

Looks fine.
(unsigned char) is not required with high bit off.
OK to get in.

[jmarshallnz]

jenkins build this please

[Karlson2k]

jenkins build this please again, without strange fails this time