chore: change position and size parameters to size_t to avoid 64-bit system compiler warnings

[glueckm]

Change the type of the position and size parameters from uint32_t to size_t to ignore compiler warnings on 64bit systems

[XuNeo]

compiler warnings on 64bit systems

May I ask which platform are you using? I'm unable to get the warning on my ubuntu22.04.

[glueckm]

I'm getting these warnings using Microsoft Visual Studio 2022.

[kisvegabor]

Thank you! I've opened #5904 to have a Windows based CI too.

[XuNeo]

Do we need to change cache->start, cache->end to size_t?

[kisvegabor]

I see no issue with changing them to size_t.

[FASTSHIFT]

@glueckm Do you have time to complete this modification? I believe the merger will be completed soon :)

[glueckm]

Sure, I can do it, but I'm not 100% sure how to proceed?
Should I change the start(end back to uint32_t and add the cast or do we keep the start/end at size_t and >I just make sure that the pull request merges with the current master?

[FASTSHIFT]

It is recommended to unify to size_t:

typedef struct {
    size_t start;
    size_t end;
    size_t file_position;
    void * buffer;
} lv_fs_file_cache_t;

[lvgl-bot]

We need some feedback on this pull request.

Now we mark this as "stale" because there was no activity here for 14 days.

Remove the "stale" label or comment else this will be closed in 7 days.

[kisvegabor]

Looks good to me. Is there anything else to add here?

[FASTSHIFT]

@glueckm
The size member of lv_fs_path_ex_t also needs to be modified to size_t type.

[FASTSHIFT]

Use %zu to print size_t type without casting.
See: https://stackoverflow.com/questions/2524611/how-can-one-print-a-size-t-variable-portably-using-the-printf-family

[glueckm]

Use %zu to print size_t type without casting. See: https://stackoverflow.com/questions/2524611/how-can-one-print-a-size-t-variable-portably-using-the-printf-family

Did not know that this exist. Should have tried tried to google it..

Should I add a new LV_PRIsize_t define for it, oder just use the the raw %zu format string

[FASTSHIFT]

You can use %zu directly. It directly matches size_t and does not require adding macros for conversion.

[FASTSHIFT]

Please consider removing any unnecessary casts.

[glueckm]

I have no idea why the microypthon test failed. Has nothing to do with my changes

[FASTSHIFT]

%zu

[FASTSHIFT]

use %zu

[FASTSHIFT]

%zu

[FASTSHIFT]

%zu

[FASTSHIFT]

%zu

[FASTSHIFT]

file_p->cache->file_position += *br;

[FASTSHIFT]

I have no idea why the microypthon test failed. Has nothing to do with my changes

It seems that memory allocation failed. Try pushing again to see if re-triggering CI can solve the problem.

AssertionError: b'Traceback (most recent call last):\n  File "<stdin>", line 37, in _wrapper\n  File "<stdin>", line 442, in main\nMemoryError: memory allocation failed, allocating 1873 bytes\n'

[liamHowatt]

I reproduced it.

[liamHowatt]

I can make the micropython test pass by adding this cast back. But it wasn't here before. The problem is from something else in this PR. Still looking.

[liamHowatt]

I see the problem.

@kdschlosser can we update the mpy binding to handle the case when size_t is 64 bits? Or would it be better to change the unix port test so that it builds in 32 bit mode? MICROPY_FORCE_32BIT = 1

Edit: Or for some APIs like FS we could just keep the u32.

Here is what I did to get it to work. Obviously this isn't the fix, though.

diff --git a/gen/gen_mpy.py b/gen/gen_mpy.py
index b8ec6e2..ef2e2ed 100644
--- a/gen/gen_mpy.py
+++ b/gen/gen_mpy.py
@@ -693,8 +693,7 @@ register_int_ptr_type('u32ptr',
         'unsigned *',
         'unsigned int *',
         'unsigned long *',
-        'unsigned long int *',
-        'size_t *')
+        'unsigned long int *')
 
 register_int_ptr_type('i32ptr',
         'int32_t *',
@@ -710,7 +709,8 @@ register_int_ptr_type('u64ptr',
         'int64_t *',
         'signed long long *',
         'long long *',
-        'long long int *')
+        'long long int *',
+        'size_t *')
 
 register_int_ptr_type('i64ptr',
         'uint64_t *',
diff --git a/lib/fs_driver.py b/lib/fs_driver.py
index bac84b3..e5a8b1a 100644
--- a/lib/fs_driver.py
+++ b/lib/fs_driver.py
@@ -38,7 +38,7 @@ def fs_read_cb(drv, fs_file, buf, btr, br):
     try:
         tmp_data = fs_file.__cast__()['file'].read(btr)
         buf.__dereference__(btr)[0:len(tmp_data)] = tmp_data
-        br.__dereference__(4)[0:4] = struct.pack("<L", len(tmp_data))
+        br.__dereference__(8)[0:8] = struct.pack("<Q", len(tmp_data))
     except OSError as e:
         raise RuntimeError("fs_read_callback(%s) exception %s" % (fs_file.__cast__()['path'], e))
 
@@ -57,7 +57,7 @@ def fs_seek_cb(drv, fs_file, pos, whence):
 def fs_tell_cb(drv, fs_file, pos):
     try:
         tpos = fs_file.__cast__()['file'].tell()
-        pos.__dereference__(4)[0:4] = struct.pack("<L", tpos)
+        pos.__dereference__(8)[0:8] = struct.pack("<Q", tpos)
     except OSError as e:
         raise RuntimeError("fs_tell_callback(%s) exception %s" % (fs_file.__cast__()['path'], e))
 
@@ -67,7 +67,7 @@ def fs_tell_cb(drv, fs_file, pos):
 def fs_write_cb(drv, fs_file, buf, btw, bw):
     try:
         wr = fs_file.__cast__()['file'].write(buf.__dereference__(btw)[0:btw])
-        bw.__dereference__(4)[0:4] = struct.pack("<L", wr)
+        bw.__dereference__(8)[0:8] = struct.pack("<Q", wr)
     except OSError as e:
         raise RuntimeError("fs_write_callback(%s) exception %s" % (fs_file.__cast__()['path'], e))
 

[kdschlosser]

I am failing to understand why there is a need to access a file that has a size of 18446744073709551615 bytes from LVGL. I don't even see the need to access a file that is 4294967295bytes in size. That's 3.99GB which is what a 32but unsigned integer is able to hold.

In the end you are still only dealing with files on a byte by byte basis. so reading and writing to files should only be calculated using 8 bits at a time anywho.

The compiler using 64 bit or 32 bit doesn't change anything when using any type that isn't a long type. so a uint32_t is 4 bytes no matter what. memory addresses change size from 4 bytes to 8 bytes but that has zero effect on the size of the underlying data.

If the underlying file system driver is using say a uint32_t for telling the driver what the position is to be in the file and that structure is expecting to get a data type that is 8 bits in size I recommend using unsigned long as the data type. and cast whatever is being passed to that data type. The long data type changes from 4 bytes to 8 bytes when compiling in 64bit.

The datatype that is used when calling the callback should be casting to a uint32_t for the position. If that is done then there is no python code that needs to change at all.

[kdschlosser]

the issue is in LVGL not in the binding. That is where the problem needs to be corrected.

[kdschlosser]

look here.

Line 290 in lv_fs.c you have this code.

uint32_t bw_tmp = 0;
res = file_p->drv->write_cb(file_p->drv, file_p->file_d, buf, btw, &bw_tmp);

the issue is passing a pointer that has to be filled for the bytes that have been written is bonkers. the treturn type from the callback should be an in32_t. if the number is zero or greater that is the number of bytes written and if it is less than zero that indicates an error. you should not be passing a pointer that needs to be filled for an integer when the mechanics are in place to return the value that would be put into that pointer anyhow. It causes excess complications that shouldn't have to be dealt with.

I think that LVGL having the capacity to handle a file that is 2147483647 bytes in size is more than sufficient. That would be the maximum that could be returned from the callback function. and again errors would be anything less than zero.

[kdschlosser]

lets use the seek function from the posix driver

static lv_fs_res_t fs_seek(lv_fs_drv_t * drv, void * file_p, uint32_t pos, lv_fs_whence_t whence)
{
    LV_UNUSED(drv);
    int w;
    switch(whence) {
        case LV_FS_SEEK_SET:
            w = SEEK_SET;
            break;
        case LV_FS_SEEK_CUR:
            w = SEEK_CUR;
            break;
        case LV_FS_SEEK_END:
            w = SEEK_END;
            break;
        default:
            return LV_FS_RES_INV_PARAM;
    }

    int fd = FILEP2FD(file_p);
    off_t offset = lseek(fd, pos, w);
    if(offset < 0) {
        LV_LOG_WARN("Could not seek file: %d, errno: %d", fd, errno);
        return LV_FS_RES_FS_ERR;
    }

    return LV_FS_RES_OK;
}

This code can have the potential to have the compiler complain. This is because of the type that is set for the pos parameter for the function. It is being passed uncasted to a function that is not under the control of LVGL. If that functions type changes based on 32 or 64 bit some complaining will take place. there needs to be a specific cast made to that parameters data type. If the data type is a size_t then this would correct the compiler complaining about it.

static lv_fs_res_t fs_seek(lv_fs_drv_t * drv, void * file_p, uint32_t pos, lv_fs_whence_t whence)
{
    LV_UNUSED(drv);
    int w;
    switch(whence) {
        case LV_FS_SEEK_SET:
            w = SEEK_SET;
            break;
        case LV_FS_SEEK_CUR:
            w = SEEK_CUR;
            break;
        case LV_FS_SEEK_END:
            w = SEEK_END;
            break;
        default:
            return LV_FS_RES_INV_PARAM;
    }

    int fd = FILEP2FD(file_p);
    off_t offset = lseek(fd, (size_t)pos, w);
    if(offset < 0) {
        LV_LOG_WARN("Could not seek file: %d, errno: %d", fd, errno);
        return LV_FS_RES_FS_ERR;
    }

    return LV_FS_RES_OK;
}

in ubuntu this is the declaration for lseek

       off_t lseek(int fd, off_t offset, int whence);

off_t is a signed long where as size_t is an unsigned long.
so the correct type cast for the pos parameter would be to signed long and not size_t

[liamHowatt]

Maybe the best thing is to remove size_t from the API. Like you said, it's rare to approach the 32nd bit for file offsets, for sure. It would be a pretty hard sell to use uint32 in lv_string.h. But lv_string.h isn't in the micropython binding from what I can see. So maybe we can stop using size_t in any part of the API that's available in the mpy binding? And then for Win and POSIX filesystem drivers, we could compare and log the incoming size_t if it's too large before it gets cast down to a u32.

[kdschlosser]

Where the problem is stemming from with the file system drivers is the pointer that gets passed to the callback function so it can be filled with the bytes that are written or read. How you do that on the Python side of things ends up being dependent on the memory address size which changes if compiled 32bit vs 64bit. what I am saying is instead of passing that pointer that gets filled why not just return the value from the callback instead. Set any file system errors to negative values and set the return type to int32_t. Granted you will be only to return a written or read size of 2GB but who on earth is going to be loading a 2GB file using LVGL? No one would.

[kdschlosser]

I could write wrappers around the file system drivers to handle this but that is going to make the binding even more complicated and prone to having errors. It will also need to have code in it that ends up being LVGL specific which is something that has been avoided so far. You don't want to have code in it that looks for specific function names to wrap code to modify the behavior.

[kisvegabor]

What about detecting if MP is running on a 32 or 64 bit system, and add an if like

if on_32bit:
  bw.__dereference__(4)[0:4] = struct.pack("<L", wr)
else: 
  bw.__dereference__(8)[0:8] = struct.pack("<Q", wr)

[kdschlosser]

What about detecting if MP is running on a 32 or 64 bit system, and add an if like

You might be able to do that IF micropython has the bits added to it to be able to>>

import sys

is_64bit = sys.maxsize > 2 ** 32

import ctypes

is_64bit = ctypes.sizeof(ctypes.c_void_p) == 8

There is another one that you can do using the struct module as well. I don't remember what that one is off the top of my head tho.

Or you will need to specifically add an integer constant in the LVGL module that is able to provide the bitness that things were compiled in.

The issue with doing this Is you are only solving an issue in a single place. There are may other places in LVGL where a pointer gets passed as a parameter to be filled and that is where the problem stems from. If it is a primitive type like an int and not an array or a struct being able to access the memory location properly is what ends up causing problems. The simple solution for this is to return the value instead of putting it into a pointer. In the majority of cases where something like this needs to be done it is done when reading or writing some kind of a buffer or file like object this is also the why errors those kinds of operations are negative numbers. Well originally that was the intention was to be be able to return the position or size as a positive number and the errors as a negative. Once memory and file sizes get up into the Gigabytes that's when that was done away with and replaced by using the parameter because you cannot hold an integer that is > 2GB in an int unless it is unsigned and then you loose the ability to return negative numbers.

For some crazy reason I cannot foresee the need to have the ability to deal with buffers or file objects that have a size > 2GB. That's a very large amount of data. a 2GB image that is RGBA using RAW pixel data is going to have 536,870,912 pixels. That is the maximum that a signed integer can handle . That's a HUGE image. 536 megapixel of RGBA data.

[kisvegabor]

I see, however passing a pointer to do something with it is quite normal. E.g. lv_draw_rect_dsc_init(&dsc). So guess it's a solved problem in general even if the size of lv_draw_rect_dsc_t is different on 32/64 bit.

So how does it work for lv_draw_rect_dsc_init? 🙂

[kdschlosser]

while I know that is normal for C operations it becomes a problem because LVGL is being bound to for higher level programming languages that do not know how to deal with pointers properly. You have to remember that operations in higher level languages are going to be about 200 times slower so each new line of code that has to be added to handle these kinds of things is going to weigh it down. Simple data types are not an issue but when dealing with the mechanics that have to be added to handle pointers it really causes problems.

I do not know why the micropython binding was written to return a pointer as a structure of some kind instead of returning a memoryview object. A memoryview object is how python deals with pointers. With the memoryview object there should not need to be any knowledge of the bit depth for the memory address that is handled internally.

As an example say you have a pointer that points to address 0xAC015632 this address is pointing to a single location and that location is able to hold 8 bits of data. That is where the size of what is being stored comes into play to determine what the end point is. When a uint32_t type gets passed into python code as a pointer the memoryview object that gets created to represent it is going to point to the start address and it is going to have a length of 4 bytes. so now we know the start and stop addresses

start address = 0xAC015632
stop address = 0xAC015632 + 4

what needs to be done is the data being stored has to be split into the 4 bytes of data and that gets stored as you would in a C array. There is no knowledge of the side of the pointer address. What is causing the problem is the mechanics in the binding when dealing with pointers and having the pointer wrapped in a structure which masks the underlying memory location.

In Python there is the ability to pass a pointer to a python object as user data using the "id" of that object. The ID is the memory location. If that ID gets passed from C code back into Python it can be converted back into the Python object the ID points to. Currently this ability doesn't exist in the binding. I have no way of passing the instance of a class into C code and then back again. That is why None is always set as the user_data for callback functions. This is a limitation of the LVGL binding code and nothing more. It doesn't handle pointers properly.

[kisvegabor]

I understand the issue (mostly), but in this particular PR we need to solve only the issue of size_t which is either 32 or 64 bit and it seems like just a few ifs as @liamHowatt pointed out here.

I would like to keep the use of size_t in C because it's standard and lot of stdlib functions return that type. If we move to uint32_t we will have a lot of warnings.

[lvgl-bot]

We need some feedback on this pull request.

Now we mark this as "stale" because there was no activity here for 14 days.

Remove the "stale" label or comment else this will be closed in 7 days.

[liamHowatt]

Not stale

[lvgl-bot]

We need some feedback on this pull request.

Now we mark this as "stale" because there was no activity here for 14 days.

Remove the "stale" label or comment else this will be closed in 7 days.