Reworked unicode support

[lynxlynxlynx]

Originally reported by: Gennady Trafimenkov (Bitbucket: gennady, GitHub: gennady)

---

At the moment there is a number of bugs related to unicode string support: #4, #5, #6

Most of them come from differences in wchar_t on various platforms (2 bytes on Windows, 4 bytes on Linux, ? bytes on MinGW on Linux, ? bytes on MinGW on Linux).

There is also an issue with Android NDK, which doesn't properly support wchar_t.

It might be a good idea to:

• don't use wchar_t at all;
• keep unicode strings incoded in utf-8;
• transform to utf-16 where it is necessary (displaying, writing into files (saves, for example).

The ISO/IEC 10646:2003 Unicode standard 4.0 says: "The width of wchar_t is compiler-specific and can be as small as 8 bits. Consequently, _programs that need to be portable across any C or C++ compiler should not use wchar_t for storing Unicode text. The wchar_t type is intended for storing compiler-defined wide characters, which may be Unicode characters in some compilers."_

Here is a seemingly good library for the job: UTF8-CPP: UTF-8 with C++ in a Portable Way
Another option is QString, but it is a huge dependency for the project.

---

• Bitbucket: https://bitbucket.org/gennady/ja2-stracciatella/issue/17

[derSteFfi]

afaik we are using utf8-cpp since b668ee9

close?

[lynxlynxlynx]

That commit only introduced it to the packaging dump and there's still plenty of use of wchar_t.

[derSteFfi]

~/ja2-strat (git)-[master] % grep -r wchar_t * | wc -l
3648

OK, that's a lot.

edit:
a bit better if we do not count binary matches:

~/ja2-strat (git)-[master] % grep -rI wchar_t * | wc -l
3260

[lynxlynxlynx]

Rechecking a year later: it indeed appears not to be needed any more (as in extra changes), especially since the new android test saw no issues (#228 ). Perhaps something from Omnigraphical's Android patchset (#435) would require it, but we can handle it then. This one is too vague, while wchar_t and original bugs are not a problem any more. Closing ...