Compiling Liblouis with 32-bit Unicode support

[AAClause]

In my opinion, the Liblouis DLL included in NVDA should be compiled with 32-bit Unicode.
Currently, we improve the French table and we'd like to add any 32-bit Unicode such as \y1D4D0, \y1D49C, \y1D4D2, \y1D49E, etc.
What do you think?

[josephsl]

Hi, have you contacted Liblouis folks about this possibility? Thanks. From: André-Abush Clause [mailto:notifications@github.com] Sent: Wednesday, January 4, 2017 8:29 AM To: nvaccess/nvda <nvda@noreply.github.com> Cc: Subscribed <subscribed@noreply.github.com> Subject: [nvaccess/nvda] Compiling Liblouis with 32-bit Unicode support (#6695) In my opinion, the Liblouis DLL included in NVDA should be compiled with 32-bit Unicode. Currently, we improve the French table and we'd like to add any 32-bit Unicode such as \y1D4D0, \y1D49C, \y1D4D2, \y1D49E, etc. What do you think? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#6695> , or mute the thread <https://github.com/notifications/unsubscribe-auth/AHgLkACqe_CV2gtBzv8GH_1rJZgTcW5Wks5rO8jZgaJpZM4Lazbp> .

[AAClause]

@josephsl yes, it is possible. See https://github.com/liblouis/liblouis/blob/master/README.windows

Edit the file configure.mk If you want 32-bit unicode change the 2 in the line UCS=2 to a 4.

[josephsl]

CC @jcsteh, @dkager, @LeonarddeR From: André-Abush Clause [mailto:notifications@github.com] Sent: Wednesday, January 4, 2017 8:43 AM To: nvaccess/nvda <nvda@noreply.github.com> Cc: Joseph Lee <joseph.lee22590@gmail.com>; Mention <mention@noreply.github.com> Subject: Re: [nvaccess/nvda] Compiling Liblouis with 32-bit Unicode support (#6695) @josephsl <https://github.com/josephsl> yes, it is possible. See https://github.com/liblouis/liblouis/blob/master/README.windows Edit the file configure.mk If you want 32-bit unicode change the 2 in the line UCS=2 to a 4. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6695 (comment)> , or mute the thread <https://github.com/notifications/unsubscribe-auth/AHgLkN28Hz7C6gLCwbHTqmhFR70RvS61ks5rO8wQgaJpZM4Lazbp> .

[dkager]

AFAIK that needs an update to the Python wrapper similar to what is done in Java.
Another use case for this feature is support for emoji, e.g. U+1F170.

[jcsteh]

1. We don't build using the liblouis build stuff for Windows; it's integrated into NVDA's own build system. However, it's easy enough to switch that to 32 bit Unicode.
2. As @dkager notes, the bigger issue is that the Python bindings are currently tied to the OS Python Unicode width, which is 16 bits on Windows. We'd need to somehow make this optional in the upstream Python bindings based on how liblouis was compiled. That's going to be a bit tricky because on Windows, Python Unicode still uses 16 bits, so the bindings will have to figure out when and what to convert. We also need to make sure it doesn't break Unix. :)

[jcsteh]

@Andre9642, how important is this for French? Does the table fail to work with NVDA if you include these characters or do they just get ignored? Are these characters frequently used in French?

[AAClause]

@jcsteh:

Yes, the table fail to work with NVDA if I include these characters. Moreover Lou_checktable indicate that the table is invalid:

<table>:<line>: error: liblouis has not been compiled for 32-bit Unicode

No, these characters aren't frequently used in French. They are used in mathematics. In fact, our additions relate in part to Unicode Math Symbols. Extract:

uplow \x0391\x03b1 46-45-1,45-1 Α,α alpha
uplow \x0392\x03b2 46-45-12,45-12 Β,β beta
uplow \x0393\x03b3 46-45-1245,45-1245 Γ,γ gamma
...
#sign \y1D4D2 46-5-14 # MATHEMATICAL BOLD SCRIPT CAPITAL C
...
#sign \y1D49F 46-5-145 # MATHEMATICAL SCRIPT CAPITAL D
...

[jcsteh]

P2 because this prevents tables from even working in NVDA if they contain 32 bit characters.

[bhavyashah]

Just a gentle ping to bring the attention of NVDA code contributors to this P2 issue in case anyone desires to work on this.

[dkager]

As noted in #6695 (comment) this needs some upstream work first. (In general, I'm not too fond of how liblouis deals with Unicode.)

[josephsl]

Hi, we need this for Python 3 (#7105(. CC @jcsteh From: bhavyashah [mailto:notifications@github.com] Sent: Sunday, August 6, 2017 4:52 AM To: nvaccess/nvda <nvda@noreply.github.com> Cc: Joseph Lee <joseph.lee22590@gmail.com>; Mention <mention@noreply.github.com> Subject: Re: [nvaccess/nvda] Compiling Liblouis with 32-bit Unicode support (#6695) Just a gentle ping to bring the attention of NVDA code contributors to this P2 issue in case anyone desires to work on this. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6695 (comment)> , or mute the thread <https://github.com/notifications/unsubscribe-auth/AHgLkDQJgjP_feXsQSQw-ySnBWT7NP21ks5sVajNgaJpZM4Lazbp> .

[jcsteh]

@josephsl commented on 7 Aug 2017, 03:31 GMT+10:

Hi, we need this for Python 3 (#7105(.

No, we don't. Python 3 Unicode does handle 32 bit characters (actually, internally, i believe it's variable width now). However, Windows is still UTF-16 and thus ctypes still treats Unicode in C functions as UTF-16, since we're running on Windows. So, this is not required for Python 3.

[jcsteh]

For anyone that does want to try to implement this into the liblouis Python bindings, here are some rough thoughts:

1. The Python bindings need to know the number of bits used for widechar (16 or 32) by the compiled liblouis. That'll need to be an autotools macro expansion, just as is done for ###LIBLOUIS_SONAME###. (NVDA's build system will also need to handle this expansion as it does for ###LIBLOUIS_SONAME###.)
2. This determines the encoding to use when passing strings to and from liblouis. If it's 16, we must use UTF-16. If it's 32, we must use UTF-32.
3. It's probably simplest to tell ctypes to use c_char_p instead of c_wchar_p so that we get raw bytes.
4. Wherever ctypes.sizeof(ctypes.c_wchar) is used, instead use the number of bits from 1) divided by 8 for bytes.
5. When passing unicode strings to liblouis, first encode them using the encoding from 2); e.g. inbuf.encode("UTF-32")
6. After obtaining unicode strings from liblouis, decode them using the encoding from 2) before returning them; e.g. outbuf.value.decode("UTF-32")

Note that this proposed implementation eliminates the need to consider the ctypes unicode width altogether. However, it may be slightly less optimal when the ctypes unicode width does match the liblouis unicode width because it always uses a Python byte string as an intermediary, where perhaps ctypes is more efficient internally; I'm not sure. You could get around this by checking the ctypes unicode width with ctypes.sizeof(ctypes.c_wchar) and only using the proposed new behaviour if it differs from the liblouis width. However, this will make the code somewhat more complicated (especially with regard to testing on different configurations) and is probably premature optimisation unless a significant speed difference can be proved.

[dkager]

There is also a function to get the widechar size, though that would require a library call from the wrapper:

/**
 * Return the size of widechar
 */
LIBLOUIS_API
int EXPORT_CALL lou_charSize(void);

A much more ambitious solution would be to make liblouis work with UTF-16 as opposed to UCS-2, i.e. add support for surrogate characters.