ESP32: 64-bit integer in python variable parsed into 32-bit in dynamic module written in C

[shuki25]

Issue

I'm writing a chess engine wrapper which uses a 64-bit bitboard for board positions, so it is important that it works on ESP32.

While writing a native dynamic module in C, I ran into two issues: (1) It appears the code is not being executed properly on a 32-bit architecture (ESP32 for example) when using 64-bit integer; (2) generating a 64-bit mask using left/right shift operator 1ULL << n compiles successfully on ESP32 platform but fails to link to a .mpy file. The module works as expected when running on 64-bit linux.

I ran into some issues passing a 64-bit integer to a native module function and passed values being parsed into 32-bit. However, ESP32 does support 64-bit integer as evident shown below. Looks like the Micropython native module functions are not converting numbers correctly.

If i assign a large value (e.g. 0xf0c00c000030c0f0) to a python variable as integer, it is stored correctly and output correctly. But if I pass that value into a function, mp_obj_get_int() converted it to a 32-bit integer instead of 64-bit integer. Even casting it to uint64_t doesn't do a thing.

Example testing output:

MPY: soft reboot
MicroPython 699477d12 on 2022-12-28; ESP32 module (spiram) with ESP32
Type "help()" for more information.
>>> import uchess
>>> uchess.test_64bit()
default hex value: 0xf0c00c000030c0f0
num bits: 64
Using left shift with '& 0x8000000000000000 bitwise' mask
1111000011000000000011000000000000000000001100001100000011110000
Correct expected bit output:
1111000011000000000011000000000000000000001100001100000011110000

Using right shift with '& 0x0000000000000001 bitwise' mask (expected output should be flipped from above)
0000111100000011000011000000000000000000001100000000001100001111
Correct expected bit output:
0000111100000011000011000000000000000000001100000000001100001111
>>> a = 0xf0c00c000030c0f0
>>> a
17347878958773879024
>>> hex(a)
'0xf0c00c000030c0f0'
>>> uchess.test_64bit(a)
parameter passed: 3195120
num bits: 64
Using left shift with '& 0x8000000000000000 bitwise' mask
0000000000000000000000000000000000000000001100001100000011110000

Using right shift with '& 0x0000000000000001 bitwise' mask (expected output should be flipped from above)
0000111100000011000011000000000000000000000000000000000000000000
>>> uchess.test_64bit_lshift()
default hex value: 0xf0c00c000030c0f0
num bits: 64
Using left shift to generate '& bitwise' mask
0000000000110000110000001111000000000000001100001100000011110000
Correct expected bit output:
1111000011000000000011000000000000000000001100001100000011110000
>>> uchess.test_64bit_lshift(a)
parameter passed: 3195120
num bits: 64
Using left shift to generate '& bitwise' mask
0000000000110000110000001111000000000000001100001100000011110000
>>> 

The above output is produced with the following code snippet below. For the uchess.test_64bit() function to test 64-bit integer functionality in ESP32 as a native module function. Without a parameter, it uses a default value, otherwise it is parsed with mp_obj_get_int function. It uses a fixed bitwise mask to test a bit while shifting the 64-bit integer to either left or right.

STATIC mp_obj_t uchess_test_64bit(size_t n_args, const mp_obj_t *args_in) {

    uint64_t test_int = 0;

    if (n_args == 0) {
        test_int = 0xf0c00c000030c0f0;
        mp_printf(&mp_plat_print, "default hex value: 0xf0c00c000030c0f0\n");
    } else if(n_args == 1) {
        test_int = (uint64_t)mp_obj_get_int(args_in[0]);
        mp_printf(&mp_plat_print, "parameter passed: %lu\n", test_int);
    }

    int bits = get_num_bits();
    mp_printf(&mp_plat_print, "num bits: %d\n", bits);

    uint64_t test_bits = test_int;
    
    mp_printf(&mp_plat_print, "Using left shift with '& 0x8000000000000000 bitwise' mask\n", bits);

    for (int i = 0; i < bits; i++) {    
        if (test_bits & 0x8000000000000000) {
            mp_printf(&mp_plat_print, "1");
        } else {
            mp_printf(&mp_plat_print, "0");
        }
        test_bits <<= 1;
    }
    mp_printf(&mp_plat_print, "\n");
    if (n_args == 0) {
        mp_printf(&mp_plat_print, "Correct expected bit output:\n");
        mp_printf(&mp_plat_print, "1111000011000000000011000000000000000000001100001100000011110000\n", bits);
    }

    mp_printf(&mp_plat_print, "\nUsing right shift with '& 0x0000000000000001 bitwise' mask (expected output should be flipped from above)\n", bits);

    test_bits = test_int;
    for (int i = 0; i < bits; i++) {    
        if (test_bits & 0x0000000000000001) {
            mp_printf(&mp_plat_print, "1");
        } else {
            mp_printf(&mp_plat_print, "0");
        }
        test_bits >>= 1;
    }
    mp_printf(&mp_plat_print, "\n");
    if (n_args == 0) {
        mp_printf(&mp_plat_print, "Correct expected bit output:\n");
        mp_printf(&mp_plat_print, "0000111100000011000011000000000000000000001100000000001100001111\n", bits);
    }
    return mp_const_none;
}

STATIC MP_DEFINE_CONST_FUN_OBJ_VAR_BETWEEN(uchess_test_64bit_obj, 0, 1, uchess_test_64bit);

In the following code snippet below, uchess.test_64bit_lshift() uses left shift << bit operation to generate a mask for a & bitwise operation test. It would function properly if 1ULL << i is used, however, when compiled, it throws a linking error. It is unable to link __ashldi3 symbol. Which I think this particular function handles 64-bit integer left/right shift operation. I also have seen the same link error with the __lshldi3 symbol as well. If using bit test operator (& , |, ^, etc) only (as evident in the first test above), it works as expected and does not require these symbols.

Compile error when using 1ULL

josh@upython-dev:~/uchess/src$ make ARCH=xtensawin V=1
GEN build/uchess.config.h
python3 ../micropython/tools/mpy_ld.py '-vvv' --arch xtensawin --preprocess -o build/uchess.config.h helper.c uchess.c
CC helper.c
xtensa-esp32-elf-gcc -I. -I../micropython -std=c99 -Os -Wall -Werror -DNDEBUG -DNO_QSTR -DMICROPY_ENABLE_DYNRUNTIME -DMP_CONFIGFILE='<build/uchess.config.h>' -fpic -fno-common -U _FORTIFY_SOURCE  -DMICROPY_FLOAT_IMPL=MICROPY_FLOAT_IMPL_FLOAT  -o build/helper.o -c helper.c
CC uchess.c
xtensa-esp32-elf-gcc -I. -I../micropython -std=c99 -Os -Wall -Werror -DNDEBUG -DNO_QSTR -DMICROPY_ENABLE_DYNRUNTIME -DMP_CONFIGFILE='<build/uchess.config.h>' -fpic -fno-common -U _FORTIFY_SOURCE  -DMICROPY_FLOAT_IMPL=MICROPY_FLOAT_IMPL_FLOAT  -o build/uchess.o -c uchess.c
LINK build/helper.o
python3 ../micropython/tools/mpy_ld.py '-vvv' --arch xtensawin --qstrs build/uchess.config.h -o build/uchess.native.mpy build/helper.o build/uchess.o
qstr vals: get_depth, init, print_bitboard, set_depth, test_64bit, test_64bit_lshift, test_64bit_rshift
qstr objs: uchess
LinkError: build/uchess.o: undefined symbol: __ashldi3
make: *** [../micropython/py/dynruntime.mk:150: build/uchess.native.mpy] Error 1
josh@upython-dev:~/uchess/src$ 

So I used 1UL instead and it compiled properly but gives wrong results as seen above because it is 32-bit integer, so it couldn't do proper bit test operation.

Code using shift operator to generate bitwise operation mask

STATIC mp_obj_t uchess_test_64bit_lshift(size_t n_args, const mp_obj_t *args_in) {

    uint64_t test_int = 0;

    if (n_args == 0) {
        test_int = 0xf0c00c000030c0f0;
        mp_printf(&mp_plat_print, "default hex value: 0xf0c00c000030c0f0\n");
    } else if(n_args == 1) {
        test_int = (uint64_t)mp_obj_get_int(args_in[0]);
        mp_printf(&mp_plat_print, "parameter passed: %lu\n", test_int);
    }

    int bits = get_num_bits();
    uint64_t mask;

    mp_printf(&mp_plat_print, "num bits: %d\n", bits);

    uint64_t test_bits = test_int;
    mp_printf(&mp_plat_print, "Using left shift to generate '& bitwise' mask\n", bits);
    for (int i = bits-1; i >= 0; i--) {   
        mask = 1UL << i; 
        if (test_bits & mask) {
            mp_printf(&mp_plat_print, "1");
        } else {
            mp_printf(&mp_plat_print, "0");
        }
    }
    mp_printf(&mp_plat_print, "\n");
    if (n_args == 0) {
        mp_printf(&mp_plat_print, "Correct expected bit output:\n");
        mp_printf(&mp_plat_print, "1111000011000000000011000000000000000000001100001100000011110000\n", bits);
    }

    return mp_const_none;
}

STATIC MP_DEFINE_CONST_FUN_OBJ_VAR_BETWEEN(uchess_test_64bit_lshift_obj, 0, 1, uchess_test_64bit_lshift);

Expected outcome

I would expect Micropython to properly parse 64-bit integer on ESP32 as it is evident it can handle 64-bit integer as shown in the REPL output, but when using with custom written native module, it becomes 32-bit. And I would expect it to be able to do shift operation on a 64-bit integer.

The native module works properly when executed on 64-bit platform like Linux with 1ULL.

Output on 64-bit architecture (Linux)

MicroPython 699477d12 on 2022-12-28; linux [GCC 11.3.0] version
Use Ctrl-D to exit, Ctrl-E for paste mode
>>> import uchess
>>> uchess.test_64bit()
default hex value: 0xf0c00c000030c0f0
num bits: 64
Using left shift with '& 0x8000000000000000 bitwise' mask
1111000011000000000011000000000000000000001100001100000011110000
Correct expected bit output:
1111000011000000000011000000000000000000001100001100000011110000

Using right shift with '& 0x0000000000000001 bitwise' mask (expected output should be flipped from above)
0000111100000011000011000000000000000000001100000000001100001111
Correct expected bit output:
0000111100000011000011000000000000000000001100000000001100001111
>>> a = 0xf0c00c000030c0f0
>>> a
17347878958773879024
>>> uchess.test_64bit(a)
parameter passed: 17347878958773879024
num bits: 64
Using left shift with '& 0x8000000000000000 bitwise' mask
1111000011000000000011000000000000000000001100001100000011110000

Using right shift with '& 0x0000000000000001 bitwise' mask (expected output should be flipped from above)
0000111100000011000011000000000000000000001100000000001100001111
>>> uchess.test_64bit_lshift()
default hex value: 0xf0c00c000030c0f0
num bits: 64
Using left shift to generate '& bitwise' mask
1111000011000000000011000000000000000000001100001100000011110000
Correct expected bit output:
1111000011000000000011000000000000000000001100001100000011110000
>>> uchess.test_64bit_lshift(a)
parameter passed: 17347878958773879024
num bits: 64
Using left shift to generate '& bitwise' mask
1111000011000000000011000000000000000000001100001100000011110000
>>> 

Proposed Fix

I think this issue can be fixed if it is able to link to __ashldi3 or __lashldi3 symbol at compile time.

Build Environment

Using version 4.4 ESP-IDF toolchain.

josh@upython-dev:~/uchess/src$ env
SHELL=/bin/bash
IDF_PYTHON_ENV_PATH=/home/josh/.espressif/python_env/idf4.4_py3.10_env
PWD=/home/josh/uchess/src
LOGNAME=josh
XDG_SESSION_TYPE=tty
IDF_PATH=/home/josh/esp/esp-idf-4.4
OPENOCD_SCRIPTS=/home/josh/.espressif/tools/openocd-esp32/v0.11.0-esp32-20221026/openocd-esp32/share/openocd/scripts
MOTD_SHOWN=pam
HOME=/home/josh
LANG=en_US.UTF-8
LESSCLOSE=/usr/bin/lesspipe %s %s
XDG_SESSION_CLASS=user
TERM=xterm-256color
LESSOPEN=| /usr/bin/lesspipe %s
USER=josh
SHLVL=1
PATH=/home/josh/esp/esp-idf-4.4/components/esptool_py/esptool:/home/josh/esp/esp-idf-4.4/components/espcoredump:/home/josh/esp/esp-idf-4.4/components/partition_table:/home/josh/esp/esp-idf-4.4/components/app_update:/home/josh/.espressif/tools/xtensa-esp-elf-gdb/11.2_20220823/xtensa-esp-elf-gdb/bin:/home/josh/.espressif/tools/riscv32-esp-elf-gdb/11.2_20220823/riscv32-esp-elf-gdb/bin:/home/josh/.espressif/tools/xtensa-esp32-elf/esp-2021r2-patch5-8.4.0/xtensa-esp32-elf/bin:/home/josh/.espressif/tools/xtensa-esp32s2-elf/esp-2021r2-patch5-8.4.0/xtensa-esp32s2-elf/bin:/home/josh/.espressif/tools/xtensa-esp32s3-elf/esp-2021r2-patch5-8.4.0/xtensa-esp32s3-elf/bin:/home/josh/.espressif/tools/riscv32-esp-elf/esp-2021r2-patch5-8.4.0/riscv32-esp-elf/bin:/home/josh/.espressif/tools/esp32ulp-elf/2.35_20220830/esp32ulp-elf/bin:/home/josh/.espressif/tools/openocd-esp32/v0.11.0-esp32-20221026/openocd-esp32/bin:/home/josh/.espressif/python_env/idf4.4_py3.10_env/bin:/home/josh/esp/esp-idf-4.4/tools:/home/josh/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
IDF_TOOLS_EXPORT_CMD=/home/josh/esp/esp-idf-4.4/export.sh
IDF_TOOLS_INSTALL_CMD=/home/josh/esp/esp-idf-4.4/install.sh
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
SSH_TTY=/dev/pts/1
OLDPWD=/home/josh/uchess
_=/usr/bin/env

• firmware: custom compiled used standard config for esp32-spiram port
• git commit hash and port/board: hash 699477d, port: esp32-spiram
• version information shown in the REPL (hit Ctrl-B to see the startup message)

MPY: soft reboot
MicroPython 699477d12 on 2022-12-28; ESP32 module (spiram) with ESP32
Type "help()" for more information.

[stinos]

The return value of mp_obj_get_int is of type mp_int_t. If you are on a platform where mp_int_t is defined as a 32bit integer type, as seems to be the case for esp32 seeing it has typedef int32_t mp_int_t; in its mpconfigport.h, it is impossible to get a non-truncated 64bit number out of that function. That's not really a bug. You should use a function appropriate for your platform; IIRC there is no builtin one: see rather old #2783 for background and e.g. #8089 for another implementation.

[IhorNehrutsa]

py\obj.c: Get 64-bit integer arg. #8089