Unrecoverable error. Failure when converting to UTF-8

[AKlimashevskyCedon]

I get this error when try to save an object.
io.realm.exceptions.RealmError: Unrecoverable error. Failure when converting to UTF-8; error_code = 7; 0xd83d 0xd83d in io_realm_internal_UncheckedRow.cpp line 298

[kneth]

Can you say what values of the stings are in that object? Moreover, which version of Realm? Was the Realm created under the same version of Realm? Do you use encryption?

[AKlimashevskyCedon]

1. the string is 0xd83d 0xd83d (I gues this is a iOs iOS Emoji, which is UTF-16 base encoded)
2. 0.84
3. yes
4. no

[zaki50]

0xd83d 0xd83d is an invalid sequence in UTF-16 because both are high surrogates.

High surrogates(d800-dbff) must be followed by low surrogates(dc00-dfff).

[kneth]

As @zaki50 points out, it is not a valid UTF-16 sequence. Can you tell us which emoji it is (http://punchdrunker.github.io/iOSEmoji/table_html/)?

[kneth]

@AKlimashevskyCedon Was your Realm created under iOS?

[ghost]

Hey,

We see a similar error in our crash reporting:

Caused by io.realm.exceptions.RealmError: Unrecoverable error. Failure when converting to UTF-8; error_code = 6; 0x05e7 0x05d5 0x05e8 0x05df 0x0020 0x05d4 0x05de 0x05e2 0x05e6 0x05d1 0x05df 0xd83d in io_realm_internal_UncheckedRow.cpp line 298

I currently cannot tell what the Java string value is, but we will can add a tracking for that.

Concerning the RealmError, does this make the Realm instance unusable? As this basically is a type error is it not possible to throw and Exception from which the app can recover?

Edit:
I tried to reproduce it with the iOS emojis mentioned, but no success
The value we try to set are names of friends of the user and can be freely changed by the user.

[kneth]

It is safest to close the app when a RealmError occur, and you shouldn't use the instance. The Realm file should be fine. If you can deduce the Java string, it could turn into very valuable information!

[ghost]

I will try to get that information, but this might take some time, as we need to release a new version which includes this information in the crash tracking.

My point was, when I looked where the error occurs in native code you see that it is before any changes are made to the Realm (

realm-java/realm/realm-jni/src/io_realm_internal_UncheckedRow.cpp

Line 296 in 05fa8b0

JStringAccessor value2(env, value); // throws

and

realm-java/realm/realm-jni/src/util.cpp

Line 409 in 05fa8b0

throw runtime_error(string_to_hex("Failure when converting to UTF-8", chars.data(), chars.size(), error_code));

). If it is only a type mismatch or an invalid value it should not be an error that stops Realm (and therefore the whole app) from working. (Since I do not know the whole code that might be a wrong assumption 😉 )

Currently the only way to prevent the error in our app, would be to validate the String beforehand on Java side, which I don't know how.

[kneth]

No, you shouldn't validate all strings. My interest in the Java string is to use it to see if I can reproduce it more directly with our string conversion (and see if there is a bug triggered by very rare cases).

[ghost]

I will send you the string as soon as I get my hands on one, but as mentioned I will take about a week 😕

[ghost]

Update: it seems, that this is an Android issue. As surrogate pair can be separated by methods like substring it can happen that the application handles invalid strings. We are doing some cleanup on the Strings so that they can only contain valid UTF-8 now and the error is gone (the problem where incomplete surrogate pairs).

I still would expect that to be an Exception and not an Error on Realm side. For me it is kind of confusing. I try to insert an invalid string (which is basically an InvalidArgumentException) and Realm reports back a RealmError (which indicates that it is an unrecoverable error and the app needs to be restarted)

[kneth]

@rimeissner Thanks for the update.

The question of Exception vs. Error boils down to whether an app can recover for the situation and "do the right thing". And maybe we at Realm are a bit pessimistic - we chose Error to be sure that the Realm file wasn't corrupted. It might be a good time to reconsider!

[ghost]

In general I agree to that. But as for as I understand the code in this case, this happens before any call to the realm-core is made. So the realm file should not be corrupted (even so that transaction should probably be canceled and nothing should be written to the Realm).

Either way, this thread was a great help figuring out the problem 😃.

Should I send you some "cut-off" strings for testing?

[kneth]

If you have some interesting strings, I would love to include them in our tests! You're welcome to send them to help@realm.io.

[ghost]

I can post them here. These are the one we use in our tests:
"Invalid high surrogate \uD83C\uD83C\uDF51"
"Invalid low surrogate \uD83C\uDF51\uDF51"

[kneth]

@rimeissner I have merged a pull request which implements your suggestion. You can try it using the 0.88 snapshot release in a short while or wait a bit longer for 0.88 to be released.

[ghost]

Cool, I will try out the new version 👍

[zhangwenjia616]

Caused by: io.realm.exceptions.RealmError: Unrecoverable error. Failure when computing UTF-16 size error_code = 5; retcode = 56; StringData.size = 74; StringData.data = http://img4.bitautoimg.com/autoalbum/files/20160AAAA in /Users/cm/Realm/realm-java/realm/realm-library/src/main/cpp/io_realm_internal_UncheckedRow.cpp line 136

version 2.0.2

[kneth]

@zhangwenjia616 Do you have access to the Realm file or the complete stack trace?

[grobarko]

I've had a similar crash:
Fatal Exception: io.realm.exceptions.RealmError: Unrecoverable error. Failure when converting short string to UTF-16 error_code = 8; retcode = 0; StringData.size = 36; StringData.data = AAAAe in /Users/cm/Realm/realm-java/realm/realm-library/src/main/cpp/io_realm_internal_UncheckedRow.cpp line 136 at io.realm.internal.UncheckedRow.nativeGetString(Unknown Source) at io.realm.internal.UncheckedRow.getString(Unknown Source:153) at io.realm.SessionRealmProxy.realmGet$mAccessToken(Unknown Source:209) at omitted_on_purpose(Unknown Source:50) at omitted_on_purpose(Unknown Source:65) at omitted_on_purpose(Unknown Source:30) at omitted_on_purpose(Unknown Source:1118) at android.app.Activity.performCreate(Activity.java:6372) at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1110) at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2432) at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2539) at android.app.ActivityThread.access$900(ActivityThread.java:168) at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1378) at android.os.Handler.dispatchMessage(Handler.java:102) at android.os.Looper.loop(Looper.java:150) at android.app.ActivityThread.main(ActivityThread.java:5665) at java.lang.reflect.Method.invoke(Method.java) at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:799) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:689)

The devices where I've noticed this so far are: HUAWEI VNS-L31 and HUAWEI G6-L11
The realm version being used is: 2.1.0
Unfortunately, I don't have access to those devices and can not get the database file that reproduces this.

[kneth]

@grobarko Do you have an idea what the string should be? Your crash says that the string should be 36 characters long but is only AAAAe. Btw, do you know the locale of the device?

[eddieberklee]

I got a similar issue, does this copy pasta from my crash logs help?

Exception java.lang.IllegalArgumentException: Illegal Argument: Failure when converting to UTF-8; error_code = 6; 0x0023 0x004f 0x006c 0x0069 0x0076 0x0065 0x0072 0x0043 0x0068 0x0072 0x0069 0x0073 0x0053 0x0069 0x006c 0x0076 0x0061 0x0020 0xd83d in /Users/cm/Realm/realm-java/realm/realm-library/src/main/cpp/io_realm_internal_UncheckedRow.cpp line 271
io.realm.internal.UncheckedRow.nativeSetString (UncheckedRow.java)
io.realm.internal.UncheckedRow.a (UncheckedRow.java:228)
io.realm.EntryRealmProxy.realmSet$title (EntryRealmProxy.java:260)
com.compscieddy.writeaday.models.Entry.setTitle (Entry.java:132)

[kneth]

@eddieberklee I would like to know Realm version, and info about the device.

[eddieberklee]

realm verson classpath "io.realm:realm-gradle-plugin:2.2.1" [image: Inline image 1]

…

On Mon, May 22, 2017 at 1:33 PM, Kenneth Geisshirt ***@***.*** > wrote: @eddieberklee <https://github.com/eddieberklee> I would like to know Realm version, and info about the device. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1756 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABwGbd_76tAdiJgf1N0wkGoA_H41qbUZks5r8fE0gaJpZM4GgKw-> .

[Zhuinden]

@eddieberklee do you use encryption?

[eddieberklee]

Nope

…

On Tue, May 23, 2017 at 3:26 PM, Gabor Varadi ***@***.***> wrote: @eddieberklee <https://github.com/eddieberklee> do you use encryption? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1756 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABwGbRAb8wOXe6Wmh8sr-OkAXyeO2Rr2ks5r810TgaJpZM4GgKw-> .

[kneth]

@eddieberklee Error code 6 indicates Incomplete surrogate pair. Character 0xd83d can be part of an iOS smiley. Does that sounds something which is related to your app?

[eddieberklee]

Yeah that's definitely a possibility. I run a journaling app on top of Realm. So Realm doesn't support emoji unicodes?? Ed

…

On Mon, May 29, 2017 at 4:06 AM, Kenneth Geisshirt ***@***.*** > wrote: @eddieberklee <https://github.com/eddieberklee> Error code 6 indicates Incomplete surrogate pair. Character 0xd83d can be part of an iOS smiley. Does that sounds something which is related to your app? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1756 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABwGbT2CxS_2Vkwqy9XKKdW3IAlEkoCQks5r-qaqgaJpZM4GgKw-> .

[zaki50]

@eddieberklee Realm supports emoji. The problem is that your String is ending with 0xd83d.
0xd83d is a high surrogate character and it must be followed by low surrogate (0xdc00-0xdfff) to be a valid UTF-16 string.

https://en.wikipedia.org/wiki/UTF-16#U.2B10000_to_U.2B10FFFF

[eddieberklee]

Any ideas on how this might have happened? It's not possible to just type this high surrogate character in and of itself, from what it sounds like

…

On Mon, May 29, 2017 at 11:34 AM, Makoto Yamazaki ***@***.***> wrote: @eddieberklee <https://github.com/eddieberklee> Realm supports emoji. The problem is that your String is ending with 0xd83d. 0xd83d is a high surrogate character and it must be followed by low surrogate (0xdc00-0xdfff) to be valid UTF-16 string. https://en.wikipedia.org/wiki/UTF-16#U.2B10000_to_U.2B10FFFF — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1756 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABwGbV7u5amNAcqXdfKCCzYJKY7WIY5Aks5r-w-lgaJpZM4GgKw-> .

[cmelchior]

Any method in Java that operates on chars instead of unicode code points have the potential to corrupt strings with emojii. Most (if not all) methods on String only operate on chars, so things like substring() and reverse() could both cause this.

[eddieberklee]

Hmm ok, must be incorrect string manipulation on my end then. Seems like something Realm could safeguard against as well though?

…

On Mon, May 29, 2017 at 12:13 PM, Christian Melchior < ***@***.***> wrote: Any method in Java that operates on chars instead of unicode code points have the potential to corrupt strings with emojii. Most methods on String only operate on chars, so things like substring() and reverse() could both cause this. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1756 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABwGbbLXFJRgDZ0IXKzT-ayDK-rJpFPwks5r-xjngaJpZM4GgKw-> .

[kneth]

@eddieberklee We do have checks (unlike SQLite) during insertions, and I am a bit surprised to see that the error message when reading from the Realm file. But in version 3.1.0 we fixed a unreported bug which could lead to corruption of Realm files. I suggest that you upgrade if possible.

[Zhuinden]

Yeah, 3.1.4 and 3.3.1 are good versions

[eddieberklee]

Thanks a lot guys - I've updated my Realm version and I'll let you know if I see this error again.

…

On Tue, May 30, 2017 at 1:39 AM, Gabor Varadi ***@***.***> wrote: Yeah, 3.1.4 and 3.3.1 are good versions — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1756 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABwGbZu1ADjldXSqKI1CUgCkFBzuxTE-ks5r-9WvgaJpZM4GgKw-> .

[jpmcosta]

Fatal Exception: java.lang.IllegalArgumentException: Illegal Argument: in_begin != in_end when converting to UTF-8; error_code = 4;  0x0041 […] 0xd83d in /home/cc/repo/realm/release/realm/realm-library/src/main/cpp/io_realm_internal_UncheckedRow.cpp line 308
       at io.realm.internal.UncheckedRow.nativeSetString(Unknown Source)
       at io.realm.internal.UncheckedRow.getColumnCount(Unknown Source:42)
       at io.realm.NatificationRealmProxy.realmSet$text(Unknown Source:94)
       at io.realm.NatificationRealmProxy.copy(Unknown Source:64)
       at io.realm.NatificationRealmProxy.realmSet$_largeIconByteArray(Unknown Source:190)
       at io.realm.DefaultRealmModuleMediator.createRealmObjectSchema(Unknown Source:111)
[…]

The property in question is a ByteArray (I'm using Kotlin), but it is not used. I never read or write it. The previous property is a String.

Realm version: 3.3.1
Android version: 8

[cmelchior]

@jpmcosta That is interesting. It looks like the setter is being delegated to the wrong underlying property. Can you create a new issue with the model class in question?

[TimothyGCY]

I get this error due to substring(), I might have accidentally split the emoji