Can't json decode Chinese character on arm platform

[scuzhanglei]

Bug Report

Describe the bug

on arm platform, regex parser with Decode_Field_As can't work correctly.
this is fluent-bit error log:

[2021/03/01 19:48:34] [error] Crossing over string boundary
[2021/03/01 19:48:34] [error] Not at boundary but still NULL terminating : 9 - '�呀"}'

To Reproduce

• Rubular link if applicable: https://rubular.com/r/p3ryl22EbVRU00

• Steps to reproduce the problem:
  with above parser and the example log with Chinese character or other special character like '《》' will reproduce it.

Expected behavior

Screenshots

Your Environment

• Version used:

from v1.2.0 to v1.7.1 which is build on my arm machine with default configuration: cmake3 ../ && make would suffer this issue

v1.1.x fluent-bit cant start at all, it exit directly

v1.0.x will work correctly

• Configuration:

parser.conf

[PARSER]
    Name db_center
    Format regex
    Regex ^\[(?<time>.+): (?<levelname>.+)\/(?<processName>.+)\] \[(?<name>.+):(?<result>.+)\] \[(?<source>.+):(?<userrole>.+):(?<username>.+)\] (?<resources>{.*}) (?<data>{.+}) (?<messages>{.+})
    Time_Key time
    Decode_Field_As json resources
    Decode_Field_As json data
    Decode_Field_As json messages

• Environment name and version (e.g. Kubernetes? What version?):
• Server type and version: aarch64
• Operating System and version: CentOS Linux release 7.8.2003 (AltArch)
• Filters and plugins:

Additional context

and I tested it on x86 machine, it works ok, this issue only occurs on a arm machine.

[tlamr]

Hello!

we are facing same issue. This blocks our migration to arm64. It works ok with latest version and amd64 version thought.

Cheers,
Tomas

[PettitWesley]

@edsiper Is this a known limitation of the arm64 build?

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[tlamr]

ping

[agup006]

@nokute78 Would this be something you would be able to help with? I recall you helping with some localization issues before

[nokute78]

I excerpted from flb_unescape.c and try to print japanese.
https://gist.github.com/nokute78/9fbf670d657277d8645ad11ad0413ce8

I can reproduce it on Ubuntu 20.04 (x86_64) and qemu which emulates aarch64.

$ sudo apt install g++-aarch64-linux-gnu qemu-user-binfmt 
$ aarch64-linux-gnu-gcc -o aaa --static arm_string.c
$ ./aaa
Crossing over string boundaryNot at boundary but still NULL terminating : 6 - '��'あぃ

ã

On x86_64, it outputs successfully.
Hmm.

$ gcc --static arm_string.c
$ ./a.out 
あぃ

あぃ

[nokute78]

Casting result is different. (signed char -> unsigned uint32_t)
It affects

fluent-bit/src/flb_unescape.c

Line 187 in 8b8d82a

ch = (uint32_t) *in_buf;

example code

#include <stdio.h>
#include <stdint.h>

int main() {
  char a = -1;
  printf("a           = %0x\n", a);
  printf("a(uint32_t) = %0x\n", (uint32_t)a);
}

On x86_64

a           = ffffffff
a(uint32_t) = ffffffff

On aarch64

a           = ff
a(uint32_t) = ff

[nokute78]

Hmm,
printf("%x", (char)-1);
is different on x86_64 / aarch64.

[nokute78]

%x expects variable is unsigned int. So (char) -1 will be converted to unsigned int.
https://linux.die.net/man/3/printf

o, u, x, X
The unsigned int argument is converted to

Refer to Draft of ISO/IEC 9899:201x, (char) -1 will be converted to (uint32_t) UINT32_MAX in my understanding.

6.3.1.3 Signed and unsigned integers
(snip)
2 Otherwise, if the new type is unsigned, the value is converted by repeatedly adding or
subtracting one more than the maximum value that can be represented in the new type
until the value is in the range of the new type.60)

[YukkiKaze]

I excerpted flb_unescape.c from v1.0.4 ,which fluent-bit works good both on arm and x86, but I didn't get expected output，

root in localhost in ~/test 
❯ gcc -o flb_104 flb_104.c 

root in localhost in ~/test 
❯ ./flb_104 
被拔

so I add some conditional statements taken from v1.7.4
https://gist.github.com/YukkiKaze/1f0210b207ab92146dcb830fbc3c6e7a
now I can get expected result on x86,

root in localhost in ~/test 
❯ gcc -o flb_104 flb_104.c 

root in localhost in ~/test 
❯ ./flb_104 
被拔

被拔

but still face the same problem on arm

[root@nestedcluster 18:54:36 test]$gcc -o flb_104 flb_104.c
[root@nestedcluster 18:54:44 test]$./flb_104
Not at boundary but still NULL terminating : 6 - '��'
被拔

被

emmm...
and I compare flb_unescape_string_utf8 between v1.0.4 and v1.7.4, overall logic seems same.
If casting issue is the culprit, why v1.0.4 didn't expose it?

[YukkiKaze]

@nokute78 @edsiper Hi~ I seem found a solution, ch = (uint32_t) (signed char) *in_buf; will solve the problem.
Because char type on arm is unsigned char by default, but signed char on x86. So when casting char to uint32_t, C will do Integer Promotion.
For example

char a = 0xe8
(uint32_t) a == 4294967272 // 0xffffffe8 on x86
(uint32_t) a == 232 // 0x000000e8 on arm
(uint32_t) (signed char) a == 4294967272 // 0xffffffe8 on both x86 and arm

[edsiper]

@nokute78 ^

[nokute78]

@YukkiKaze Thank you for pointing it!
I'll check your patch #3522

https://developer.arm.com/documentation/den0013/d/Porting/Miscellaneous-C-porting-issues/unsigned-char-and-signed-char

[nokute78]

The patch #3522 is merged.
Close this issue.