String based parser: Non-latin characters are not supported in field names.

[sync-by-unito]

The new query parser crashes when field names with non-latin characters are used.

Example

델타 = $0

Stacktrace

2021-02-26 16:48:59.651 21485-21485/? A/DEBUG: *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
2021-02-26 16:48:59.651 21485-21485/? A/DEBUG: Build fingerprint: 'google/sdk_gphone_x86_64/generic_x86_64:10/QSR1.200715.002/6695061:userdebug/dev-keys'
2021-02-26 16:48:59.651 21485-21485/? A/DEBUG: Revision: '0'
2021-02-26 16:48:59.651 21485-21485/? A/DEBUG: ABI: 'x86_64'
2021-02-26 16:48:59.651 21485-21485/? A/DEBUG: Timestamp: 2021-02-26 16:48:59+0100
2021-02-26 16:48:59.651 21485-21485/? A/DEBUG: pid: 21448, tid: 21479, name: roidJUnitRunner  >>> io.realm.test <<<
2021-02-26 16:48:59.651 21485-21485/? A/DEBUG: uid: 10216
2021-02-26 16:48:59.651 21485-21485/? A/DEBUG: signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr --------
2021-02-26 16:48:59.651 21485-21485/? A/DEBUG: Abort message: 'JNI DETECTED ERROR IN APPLICATION: input is not valid Modified UTF-8: illegal continuation byte 0x20
        string: 'Illegal Argument: Invalid predicate: '델타 = $0': invalid character: � in /Users/clemente.tort/realm-java/realm/realm-library/src/main/cpp/io_realm_internal_TableQuery.cpp line 472'
        input: '0x49 0x6c 0x6c 0x65 0x67 0x61 0x6c 0x20 0x41 0x72 0x67 0x75 0x6d 0x65 0x6e 0x74 0x3a 0x20 0x49 0x6e 0x76 0x61 0x6c 0x69 0x64 0x20 0x70 0x72 0x65 0x64 0x69 0x63 0x61 0x74 0x65 0x3a 0x20 0x27 0xeb 0x8d 0xb8 0xed 0x83 0x80 0x20 0x3d 0x20 0x24 0x30 0x27 0x3a 0x20 0x69 0x6e 0x76 0x61 0x6c 0x69 0x64 0x20 0x63 0x68 0x61 0x72 0x61 0x63 0x74 0x65 0x72 0x3a 0x20 0xeb <0x20> 0x69 0x6e 0x20 0x2f 0x55 0x73 0x65 0x72 0x73 0x2f 0x63 0x6c 0x65 0x6d 0x65 0x6e 0x74 0x65 0x2e 0x74 0x6f 0x72 0x74 0x2f 0x72 0x65 0x61 0x6c 0x6d 0x2d 0x6a 0x61 0x76 0x61 0x2f 0x72 0x65 0x61 0x6c 0x6d 0x2f 0x72 0x65 0x61 0x6c 0x6d 0x2d 0x6c 0x69 0x62 0x72 0x61 0x72 0x79 0x2f 0x73 0x72 0x63 0x2f 0x6d 0x61 0x69 0x6e 0x2f 0x63 0x70 0x70 0x2f 0x69 0x6f 0x5f 0x72 0x65 0x61 0x6c 0x6d 0x5f 0x69 0x6e 0x74 0x65 0x72 0x6e 0x61 0x6c 0x5f 0x54 0x61 0x62 0x6c 0x65 0x51 0x75 0x65 0x72 0x79 0x2e 0x63 0x70 0x70 0x20 0x6c 0x69 0x6e 0x65 0x20 0x34 0x37 0x32'
        in call to ThrowNew
        from void io.realm.internal.TableQuery.nativeRawPredicate(long, boolean, java.lang.String, long, long)'
2021-02-26 16:48:59.651 21485-21485/? A/DEBUG:     rax 0000000000000000  rbx 00000000000053c8  rcx 000078fc9415c3f8  rdx 0000000000000006
2021-02-26 16:48:59.651 21485-21485/? A/DEBUG:     r8  000078fc05ec9d00  r9  0000000000000000  r10 000078fbbb075e10  r11 0000000000000246
2021-02-26 16:48:59.652 21485-21485/? A/DEBUG:     r12 000078fbbb07c008  r13 000078fbbb07c008  r14 000078fbbb075e98  r15 00000000000053e7
2021-02-26 16:48:59.652 21485-21485/? A/DEBUG:     rdi 00000000000053c8  rsi 00000000000053e7
2021-02-26 16:48:59.652 21485-21485/? A/DEBUG:     rbp 000078fc05ec9d00  rsp 000078fbbb075e08  rip 000078fc9415c3f8
2021-02-26 16:48:59.736 21485-21485/? A/DEBUG: backtrace:
2021-02-26 16:48:59.736 21485-21485/? A/DEBUG:       #00 pc 00000000000943f8  /apex/com.android.runtime/lib64/bionic/libc.so (syscall+24) (BuildId: b5c6019a3b4ea61b5e9a2f56319b584e)
2021-02-26 16:48:59.736 21485-21485/? A/DEBUG:       #01 pc 0000000000097146  /apex/com.android.runtime/lib64/bionic/libc.so (abort+182) (BuildId: b5c6019a3b4ea61b5e9a2f56319b584e)
2021-02-26 16:48:59.736 21485-21485/? A/DEBUG:       #02 pc 00000000005531ff  /apex/com.android.runtime/lib64/libart.so (art::Runtime::Abort(char const*)+2399) (BuildId: bc83816dbd83f3fb4b84b85b9be82621)
2021-02-26 16:48:59.736 21485-21485/? A/DEBUG:       #03 pc 000000000000c873  /system/lib64/libbase.so (android::base::LogMessage::~LogMessage()+611) (BuildId: 40d2b536dbf0730fdc31abd2b469f94f)
2021-02-26 16:48:59.736 21485-21485/? A/DEBUG:       #04 pc 00000000003ede64  /apex/com.android.runtime/lib64/libart.so (art::JavaVMExt::JniAbort(char const*, char const*)+1604) (BuildId: bc83816dbd83f3fb4b84b85b9be82621)
2021-02-26 16:48:59.736 21485-21485/? A/DEBUG:       #05 pc 00000000003ee045  /apex/com.android.runtime/lib64/libart.so (art::JavaVMExt::JniAbortV(char const*, char const*, __va_list_tag*)+85) (BuildId: bc83816dbd83f3fb4b84b85b9be82621)
2021-02-26 16:48:59.736 21485-21485/? A/DEBUG:       #06 pc 00000000003dfb34  /apex/com.android.runtime/lib64/libart.so (art::(anonymous namespace)::ScopedCheck::AbortF(char const*, ...)+180) (BuildId: bc83816dbd83f3fb4b84b85b9be82621)
2021-02-26 16:48:59.736 21485-21485/? A/DEBUG:       #07 pc 00000000003df9f5  /apex/com.android.runtime/lib64/libart.so (art::(anonymous namespace)::ScopedCheck::CheckNonHeapValue(char, art::(anonymous namespace)::JniValueType)+1189) (BuildId: bc83816dbd83f3fb4b84b85b9be82621)
2021-02-26 16:48:59.736 21485-21485/? A/DEBUG:       #08 pc 00000000003de0ae  /apex/com.android.runtime/lib64/libart.so (art::(anonymous namespace)::ScopedCheck::CheckPossibleHeapValue(art::ScopedObjectAccess&, char, art::(anonymous namespace)::JniValueType)+174) (BuildId: bc83816dbd83f3fb4b84b85b9be82621)
2021-02-26 16:48:59.736 21485-21485/? A/DEBUG:       #09 pc 00000000003dd712  /apex/com.android.runtime/lib64/libart.so (art::(anonymous namespace)::ScopedCheck::Check(art::ScopedObjectAccess&, bool, char const*, art::(anonymous namespace)::JniValueType*)+690) (BuildId: bc83816dbd83f3fb4b84b85b9be82621)
2021-02-26 16:48:59.736 21485-21485/? A/DEBUG:       #10 pc 00000000003c87dd  /apex/com.android.runtime/lib64/libart.so (art::(anonymous namespace)::CheckJNI::ThrowNew(_JNIEnv*, _jclass*, char const*)+861) (BuildId: bc83816dbd83f3fb4b84b85b9be82621)
2021-02-26 16:48:59.736 21485-21485/? A/DEBUG:       #11 pc 0000000000812778  /data/app/io.realm.test-u1C0hpbFPdNMjpnCcQ_VNg==/lib/x86_64/librealm-jni.so (_JNIEnv::ThrowNew(_jclass*, char const*)+40) (BuildId: b48f6b21f5fef639312f588f472cf79de83e63cb)
2021-02-26 16:48:59.736 21485-21485/? A/DEBUG:       #12 pc 000000000086ffbc  /data/app/io.realm.test-u1C0hpbFPdNMjpnCcQ_VNg==/lib/x86_64/librealm-jni.so (ThrowException(_JNIEnv*, ExceptionKind, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>> const&, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>> const&)+860) (BuildId: b48f6b21f5fef639312f588f472cf79de83e63cb)
2021-02-26 16:48:59.736 21485-21485/? A/DEBUG:       #13 pc 000000000086ef49  /data/app/io.realm.test-u1C0hpbFPdNMjpnCcQ_VNg==/lib/x86_64/librealm-jni.so (ConvertException(_JNIEnv*, char const*, int)+4601) (BuildId: b48f6b21f5fef639312f588f472cf79de83e63cb)
2021-02-26 16:48:59.736 21485-21485/? A/DEBUG:       #14 pc 0000000000806b53  /data/app/io.realm.test-u1C0hpbFPdNMjpnCcQ_VNg==/lib/x86_64/librealm-jni.so (Java_io_realm_internal_TableQuery_nativeRawPredicate+915) (BuildId: b48f6b21f5fef639312f588f472cf79de83e63cb)

[sync-by-unito]

➤ Jørgen Edelbo commented:

[~james.stone] I don't think this was ever supported?

[sync-by-unito]

➤ James Stone commented:

This was not supported by the PEGTL implementation, so it is a new feature request. [~clemente.tort] does that information impact the priority of this ticket?

[bmunkholm]

There is a very clear and easy workaround, and as this hasn't been supported for other SDK's I would consider this very low priority.

[cmelchior]

I strongly disagree. This is a pretty severe breaking change for Java. We have many users which use local languages in their model classes like Chinese.

This worked out of the box when using the Core API's directly which was what we did in the old query API, but we replaced all of the internal representation with the String parser.

[bmunkholm]

Ahh sorry. I had missed that you migrated the existing typed queries to the new parser as well! Yeah, then I agree.