CData numbers are compared incorrectly

[Gerold103]

box.cfg{listen = 3313}
netbox = require('net.box')
box.schema.user.grant('guest', 'read,write,execute', 'universe')
c = netbox.connect(box.cfg.listen)
res = {}
for i = 1, 10 do table.insert(res, c:execute('SELECT random()').rows[1]:totable()[1]) end
table.sort(res)
res

tarantool> res
---
- - 1064442337705904389
  - 5430535088656468933
  - 8084626668617668138
  - -9114667304427008884
  - -8060774415037890208
  - -6794628713547132026
  - -3278541528434270828
  - -2746644752953370174
  - -2467935731651350913
  - -1266204910185345866
...

All the values are cdata.

[Gerold103]

A test appeared to be much simpler:

8084626668617668138LLU < -9114667304427008884LL
---
- true
...

Just compare unsigned number and a signed negative number.

[Totktonada]

It seems it can be fixed like so (less-then only):

diff --git a/src/lj_carith.c b/src/lj_carith.c
index 218abd2..d32f680 100644
--- a/src/lj_carith.c
+++ b/src/lj_carith.c
@@ -159,22 +159,31 @@ static int carith_int64(lua_State *L, CTState *cts, CDArith *ca, MMS mm)
 {
   if (ctype_isnum(ca->ct[0]->info) && ca->ct[0]->size <= 8 &&
       ctype_isnum(ca->ct[1]->info) && ca->ct[1]->size <= 8) {
-    CTypeID id = (((ca->ct[0]->info & CTF_UNSIGNED) && ca->ct[0]->size == 8) ||
-                 ((ca->ct[1]->info & CTF_UNSIGNED) && ca->ct[1]->size == 8)) ?
+    CTypeID id0 = ((ca->ct[0]->info & CTF_UNSIGNED) && ca->ct[0]->size == 8) ?
                 CTID_UINT64 : CTID_INT64;
-    CType *ct = ctype_get(cts, id);
+    CTypeID id1 = ((ca->ct[1]->info & CTF_UNSIGNED) && ca->ct[1]->size == 8) ?
+                CTID_UINT64 : CTID_INT64;
+    CTypeID id = id0 == CTID_UINT64 || id1 == CTID_UINT64 ?
+                CTID_UINT64 : CTID_INT64;
+    CType *ct0 = ctype_get(cts, id0);
+    CType *ct1 = ctype_get(cts, id1);
     GCcdata *cd;
     uint64_t u0, u1, *up;
-    lj_cconv_ct_ct(cts, ct, ca->ct[0], (uint8_t *)&u0, ca->p[0], 0);
+    lj_cconv_ct_ct(cts, ct0, ca->ct[0], (uint8_t *)&u0, ca->p[0], 0);
     if (mm != MM_unm)
-      lj_cconv_ct_ct(cts, ct, ca->ct[1], (uint8_t *)&u1, ca->p[1], 0);
+      lj_cconv_ct_ct(cts, ct1, ca->ct[1], (uint8_t *)&u1, ca->p[1], 0);
     switch (mm) {
     case MM_eq:
       setboolV(L->top-1, (u0 == u1));
       return 1;
     case MM_lt:
-      setboolV(L->top-1,
-              id == CTID_INT64 ? ((int64_t)u0 < (int64_t)u1) : (u0 < u1));
+      if (id0 != id1 && id0 == CTID_INT64 && (int64_t)u0 < 0)
+        setboolV(L->top-1, 1);
+      else if (id0 != id1 && id1 == CTID_INT64 && (int64_t)u1 < 0)
+        setboolV(L->top-1, 0);
+      else
+        setboolV(L->top-1,
+                id == CTID_INT64 ? ((int64_t)u0 < (int64_t)u1) : (u0 < u1));
       return 1;
     case MM_le:
       setboolV(L->top-1,

Not sure about behaviour of jitted code. Not sure whether it is right thing to do.

[igormunkin]

Several LuaJIT implementations seem to be affected with it. I'll take a look at @Totktonada patch and check whether compiler also need to be patched.

[igormunkin]

TL;DR:

Considering the described semantics about 64-bit arithmetics here, the comparison result is fine (see the cite below).

64 bit integer comparison: two cdata numbers, or a cdata number and a Lua number can be compared with each other. If one of them is an uint64_t, the other side is converted to an uint64_t and an unsigned comparison is performed. Otherwise both sides are converted to an int64_t and a signed comparison is performed.

---

I took another look at the problem and after some time knee deep into FFI internals I can provide the following rationale for such behaviour.

According to the FFI page on LJ wiki: "The FFI library allows <snipped> using C data structures from pure Lua code". Thereby, all operations with such types should be interpreted considering its C specifics. The issue relates only to 64-bit integers which are represented with cdata objects instead of native Lua numbers. All other integer types successfully maps into Lua numeric type (see here for more).

According to 6.3.1.8 and 6.5.8 C99 standard paragraphs (the C standard FFI implementation complies to) whether "the operand that has unsigned integer type has rank greater or equal to the rank of the type of the other operand, then the operand with signed integer type is converted to the type of the operand with unsigned integer type".

Many thanks to @Totktonada for a minimal oneliner with PoC that I modified a bit for the comment:

$ gcc -Wextra -x c <(echo 'int main() { return !(8084626668617668138ULL
< -9114667304427008884LL); }') -o cmp && ./cmp; echo $?
/proc/self/fd/11: In function 'main':
/proc/self/fd/11:1:46: warning: comparison of integer expressions of
different signedness: 'long long unsigned int' and 'long long int'
[-Wsign-compare]
0
$ gcc -Wextra -x c <(echo 'int main() { return !(8084626668617668138LL
< -9114667304427008884LL); }') -o cmp && ./cmp; echo $?
1
$

You can see the first result is the same one provided by LuaJIT. Considering the warning produced by gcc we will obtain the proper comparison result after changing the suffix of the first literal:

tarantool> 8084626668617668138LL < -9114667304427008884LL
---
- false
...

Long story short:

The issue is not related to FFI and I suggest replacing luajit label with the corresponding one. As far as I'm able to judge the root problem is about representing some MP data with native Lua types. In this way I propose to proceed the following work in this direction.

[kyukhin]

This looks like a correct behavior. @igormunkin please prepare update request to documentation with details.

[Totktonada]

See also #1555.

[igormunkin]

@kyukhin, created the corresponding issue in tarantool/doc queue.

[kyukhin]

Closing as not a bug.

[no1seman]

@olegrok It's not an error, it's documented behaviour: http://www.lua.org/pil/2.3.html