Fix utf8 issue #2
Conversation
As sqlite3_column_text() returns an unsigned char*, checking each byte as signed int could result in incorrect bitmask and therefore malformatted unicode signs. Getting each byte into an unsigned char instead of an int seems to fix this issue.
…tbuild-free, the matching SQLite build flags have to be added.
|
Also as a sidenote to this: cutting out all chars below 32 could be problematic too sometimes (look at One should be able to get the complete UTF-8 string out of SQLite as it is stored, regardless of code entry point values in it.. |
|
Thanks for the contribution, please accept my apologies for the delay in integrating this one as well as storesafe/cordova-sqlite-storage#709. I just saw the issue in case of 2-octet and 3-octet UTF-8 characters on Android x86_64 emulator, hoping to make this fix in the near future for which you will absolutely get credit.
Sounds right, will test further to be absolutely sure. P.S. Also reproduced on x86 (32-bit) emulator |
|
Nice! |
(unwanted sign extension in conversion) THANKS to @spacepope (Hannes Petersen <jhpetersen@gmx.de>) for pointing this one out. Closes #2 Fixes #1 (marked as duplicate) (Also reported in storesafe/cordova-sqlite-evcore-extbuild-free#19)
(unwanted sign extension in conversion) THANKS to @spacepope (Hannes Petersen <jhpetersen@gmx.de>) for pointing this one out. - #2 - fixes #1 (marked as duplicate) - storesafe/cordova-sqlite-evcore-extbuild-free#19
also using sqlite3_column_bytes to avoid possible trunction issue address sidenote by @spacepope (Hannes Petersen <jhpetersen@gmx.de>) about character values below 32 (U+0020) in #2
also using sqlite3_column_bytes to avoid possible trunction issue address sidenote by @spacepope (Hannes Petersen <jhpetersen@gmx.de>) about character values below 32 (U+0020) in #2
(cordova-sqlite-evcore-free-dependencies 0.8.3) ref: - storesafe/android-sqlite-evcore-ndk-driver-free#2 - storesafe/android-sqlite-evcore-ndk-driver-free#1 - storesafe/cordova-sqlite-evcore-extbuild-free#19
(cordova-sqlite-evcore-legacy-ext-common-free 0.0.6) using cordova-sqlite-evcore-free-dependencies 0.8.2 ref: - storesafe/cordova-sqlite-evcore-extbuild-free#19 - storesafe/android-sqlite-evcore-ndk-driver-free#1 - storesafe/android-sqlite-evcore-ndk-driver-free#2
(cordova-sqlite-evcore-extbuild-free 0.9.7-pre1) in included evcore-native-driver.jar for Android ref: - #19 - storesafe/android-sqlite-evcore-ndk-driver-free#1 - storesafe/android-sqlite-evcore-ndk-driver-free#2
Merge https://github.com/litehelpers/cordova-sqlite-evcore-legacy-ext-common-free (Fix for multi-byte UTF-8 characters on Android, using cordova-sqlite-evcore-free-dependencies 0.8.2) ref: - storesafe/cordova-sqlite-evcore-extbuild-free#19 - storesafe/android-sqlite-evcore-ndk-driver-free#1 - storesafe/android-sqlite-evcore-ndk-driver-free#2
|
The issue you reported in #1 is now resolved, solution is very similar to what @spacepope proposed here to resolve a conversion warning (a second, similar warning was also resolved for x86 & x86_64). The issue with character values below 32 will be handled in storesafe/cordova-sqlite-evcore-extbuild-free#28, should be resolved in the near future. My apologies to anyone who may have been affected by the delay. |
Hi Chris,
i found the cause of #1.
As sqlite3_column_text() returns an unsigned char*, checking each byte as signed int could result in misinterpreted bitmask and therefore malformatted unicode signs.
Getting each byte into an unsigned char instead of an int seems to fix this issue. 🎉
Can you please confirm this and merge the fix or the fixed libraries also in your https://github.com/litehelpers/Cordova-sqlite-evcore-extbuild-free repository?
(i had to modify the jni/*.mk files to be able to successfully compile and use this library under Windows 10 bash. Not sure if you need this, but i included it in the PR anyway..)
Thanks and best regards,
Hannes