External flash usage

[Avamander]

It would be good to map out where LittleFS would potentially be used in InfiniTime, right now and in the future. What would have to be migrated and what can't be migrated.

Current usages of flash contain:

• Bootloader scratch sections
• Settings storage
• Datetime backup (Littlefs (or similar) datetime backup #87)

Future usages might contain:

• Resource storage (for LVGL)
• Buffer for HR/step data (Bug/Feature: Persist Step Count Between Resets #493)
• ??? Please leave your comment below

[joaquimorg]

I managed to do a first performance test between RAWFS versus LITTLEFS.
The use of LITTLEFS adds another 20Kb in the firmware.
As we can see in the video the performance between the two is evident.

The LITTLEFS introduces some advantages in the management of the files, but I think that the loss of performance and increase of the firmware does not justify the gain of functionalities.

In my case, I will not invest more time in the littlefs solution, I will try to improve RAWFS and add some features to allow partial updates or subdivide between resources and watch faces, so that I can have customizable watch faces without having to always send everything.

[Avamander]

What's the LittleFS configuration you used?

[joaquimorg]

What's the LittleFS configuration you used?

https://github.com/joaquimorg/PinetimeLite/blob/1db8712ecd49ecb2ee37147eebe2e5b6dac521ac/src/components/fs/FS.cpp#L81

[Avamander]

I will try to optimize the settings, because I think better read performance is doable and worth the investment.

The advantages we'd get from LittleFS, such as reduced maintenance and development burden, full-flash wear leveling, better failure tolerance (CoW), thought-out and documented design, higher interoperability and alternative implementations are more than some.

[joaquimorg]

In my tests I had already identified that LVGL is always opening files to read, and did not keep the files open while refreshing the screen.
After all, it was my mistake, the code allow that there could be several files open at the same time, but LVGL does not take advantage of this if it is not indicated in LV_IMG_CACHE_DEF_SIZE how many images we want to keep open.
I will make this change and carry out further tests.

I would also prefer to use littlefs, but I don't want its use to translate into having something that makes a point of losing the little performance we have.

When I indicated that I would not invest more time in littlefs it was in the continuation of developing the mechanisms so that we can have a management of the files, delete files, create folders etc...

So whenever I discover something that can improve the performance I will test it.

[JF002]

Thanks @joaquimorg and @Avamander , great work!

From your video, the performance penality looks... surprisingly huge! Maybe @Avamander will be able to fine-tune/optimise the integration of littlefs with lvgl?

Now... Maybe we are trying to solve many problems with a single solution?
The use-case tested by @joaquimorg on his video is a "read-only" use case : we store files in the external memory once and only read them from the firmware. Those data are (over)written only when upgrading the graphical assets.
The use-case where a more advanced file system like littlefs would bring a lot of advantages are "read/write" use-cases like storing 24h or heart rate data, user settings, save the master piece the user's just drawn using InfiniPaint,...

Maybe we can explore the possibility to split the external flash space available in 2 parts : one read only to store fonts, icons, bitmaps, static assets. And the other one to store dynamic data that is often read/written.

What's your opinion?

[Avamander]

Thanks joaquimorg and Avamander , great work!
Maybe Avamander will be able to fine-tune/optimise the integration of littlefs with lvgl?

joaquimorg deserves more credit here and he also said that LV_IMG_CACHE_DEF_SIZE might provide a quick and easy way of regaining the lost performance, if that doesn't work out I'll investigate as well.

Maybe we can explore the possibility to split the external flash space available in 2 parts : one read only to store fonts, icons, bitmaps, static assets. And the other one to store dynamic data that is often read/written.

Yes, that's a theoretical possibility. But the returns are somewhat smaller. If it still turns out that LFS is not worth it, we just have to go with an alternative solution.

[joaquimorg]

Maybe we can explore the possibility to split the external flash space available in 2 parts : one read only to store fonts, icons, bitmaps, static assets. And the other one to store dynamic data that is often read/written.

I think it may be a viable solution, a few days ago we were discussing this on the discord.

Really the part of the resources for the firmware can be read-only and we don't have the need to have littlefs for that, and we can have an r/w zone with littlefs, in total we have about 3.4 Mb still free if we share between the two we have space for lots of applications to save data.

In the end we have to remember that we are talking about a device that has its limitations and we do not want to have a very complex system, my goal is always to look for a way to try to do things to get the maximum performance from the smartwatch which it often leads to doing things in a less generic way, more oriented towards the optimization of the few resources we have.

This weekend I will try to do some more tests to try to see if the changes have influence.

[ObiKeahloa]

Maybe we can explore the possibility to split the external flash space available in 2 parts : one read only to store fonts, icons, bitmaps, static assets. And the other one to store dynamic data that is often read/written.

We can also make a proper filesystem with folders (Directory's) , Example:
A fonts folder for the font files , Assets that have all assets for the firmware (Resources) , AppData (Contains appdata that is stored to the spi flash) , Userdata (User Settings , Time Settings etc,)

If possible we could also try to make the Assets r/w , so that the user can add or modify the assets without having to make an entirely new firmware....

This could be a vulnerability but a hierarchy system can be put in place: System > User > Apps.
This might need a lot of code so might be out of reach.

[joaquimorg]

I did some more tests changing the LVGL configuration (LV_IMG_CACHE_DEF_SIZE)
and it really made all the difference, of course there is some loss of performance, but I think it is possible to live with it.
Now the hardest work begins, turning everything into a PR ...

[Avamander]

I'm amazed, truly incredibly cool. Thanks @joaquimorg

[JF002]

This is really impressive! Thanks for your experimentation!

So, LVGL could be the unique answer for all our "external" storage use-cases?

For the PR, what would be the easiest way for you? A single PR of split the changes accross multiple PR to ease the review process?
Do you have a branch on github so we can have a look at how it works?

[joaquimorg]

Do you have a branch on github so we can have a look at how it works?

The branch with the test code is here
https://github.com/joaquimorg/PinetimeLite/tree/UsingLITTLEFS

[joaquimorg]

For the PR, what would be the easiest way for you? A single PR of split the changes accross multiple PR to ease the review process?

I will try to split as much as possible to be easier to include without causing breaks in what already exists, and thus also be easier to review, however it will only work in full when everything is together.

[JF002]

I will try to split as much as possible to be easier to include without causing breaks in what already exists, and thus also be easier to review, however it will only work in full when everything is together.

Awesome! Feel free to comment here or to ping us in the chat if you need help!

[ObiKeahloa]

I did some more tests changing the LVGL configuration (LV_IMG_CACHE_DEF_SIZE)
and it really made all the difference, of course there is some loss of performance, but I think it is possible to live with it.
Now the hardest work begins, turning everything into a PR ...

Awesome! Really nice that it has possibilities to be speedy.

[JF002]

@Avamander provided this link in the chat room, it could be interesting to have a closer look at this lib/protocol : https://github.com/adafruit/adafruit_circuitpython_ble_file_transfer

[JF002]

@joaquimorg Any news on this topic?

I was thinking : how will we use this filesystem? I mean, we have multiple types of data :

• read only data for the firmware (I think you call them resources in your FW): logo, icons, pictures,...
• User settings
• "run time data" like HR and steps values

I have many questions about this, and I think you've already solved most of them in PineTimeLite :)

User settings and runtime data will be created and read/written by the firmware.

I'm more concerned about the resources : The user will need to manually flash (OTA) these data for the FW to work correctly. What happens if the user does not send the file? Is there an error? Or a message asking the user to flash the data? Or the FW works in "degraded" mode without the nice pictures?

Also, what protocol do we use to send the resources? DFU? A custom one? Another one (like the one from adafruit mentioned by Avamander) ?

Next : how do we generate these data? I guess you have a script that converts pictures into that binary file?
Do you also handle versioning so that the firmware can check if the correct resources file is installed ?

These questions are mostly implementations details, but I think there are many many ways to do all of this, and I would like to have your opinions about this :)

[ObiKeahloa]

@joaquimorg Any news on this topic?

I was thinking : how will we use this filesystem? I mean, we have multiple types of data :

• read only data for the firmware (I think you call them resources in your FW): logo, icons, pictures,...
• User settings
• "run time data" like HR and steps values

I have many questions about this, and I think you've already solved most of them in PineTimeLite :)

User settings and runtime data will be created and read/written by the firmware.

I'm more concerned about the resources : The user will need to manually flash (OTA) these data for the FW to work correctly. What happens if the user does not send the file? Is there an error? Or a message asking the user to flash the data? Or the FW works in "degraded" mode without the nice pictures?

Also, what protocol do we use to send the resources? DFU? A custom one? Another one (like the one from adafruit mentioned by Avamander) ?

Next : how do we generate these data? I guess you have a script that converts pictures into that binary file?
Do you also handle versioning so that the firmware can check if the correct resources file is installed ?

These questions are mostly implementations details, but I think there are many many ways to do all of this, and I would like to have your opinions about this :)

It could show an error saying "Resources not found" similar to the mi band that says "Please connect to mi fit"

We could use the same method we are using now for this , the resources can be a compressed file that is uncompressed once sent to the watch.

[joaquimorg]

@joaquimorg Any news on this topic?

Hi, unfortunately I haven't been able to dedicate much time to the project, my work has taken up a lot of time.

I was thinking : how will we use this filesystem? I mean, we have multiple types of data :

• read only data for the firmware (I think you call them resources in your FW): logo, icons, pictures,...
• User settings
• "run time data" like HR and steps values

I have many questions about this, and I think you've already solved most of them in PineTimeLite :)

User settings and runtime data will be created and read/written by the firmware.

I'm more concerned about the resources : The user will need to manually flash (OTA) these data for the FW to work correctly. What happens if the user does not send the file? Is there an error? Or a message asking the user to flash the data? Or the FW works in "degraded" mode without the nice pictures?

In my fork I do not present any information, what happens is that the images do not appear, however it would be good to present some information about the lack of updating the resources, to avoid that the user does not think that the update was not done correctly.

Also, what protocol do we use to send the resources? DFU? A custom one? Another one (like the one from adafruit mentioned by Avamander) ?

I do not think that DFU can be used, since it only allows to have the Bootloader, Softdevice and Application in the payload, so I was able to investigate.
So we would have to have something to be able to upload the resources.
It would be useful to be able to upload together or separately.

The adafruit example is very similar to the one I have, mine is a little simpler but it works.
If we want to have a way to be able to control the files that are in the flash we will have to have an additional service to the DFU, in order to be able to list, create and delete files.

Next : how do we generate these data? I guess you have a script that converts pictures into that binary file?
Do you also handle versioning so that the firmware can check if the correct resources file is installed ?

Yes I have a script that converts PNG to LVGL images,
then another script creates the RAWFS file, it is this file that will be sent to the watch.
And yes the RAWFS has the version in the header, to be able to validate that it is the right one when the watch starts.

However with littlefs I think it cannot be done the way I have it, in the tests I did the files were all separate, and this is the way that makes the most sense to me, for example having a "resources" folder and inside the necessary images, there will also be a metadata file in that folder with the version information and other information that is necessary to identify that the resources are valid for the version that the watch is running.

The generation of resources can be done by generating a zip with all the necessary files, and it is on the side of the companion application to open that zip and send each file to the watch inside the "resources" folder.

[ObiKeahloa]

The generation of resources can be done by generating a zip with all the necessary files, and it is on the side of the companion application to open that zip and send each file to the watch inside the "resources" folder.

Much better than my idea of senting a compressed file , will be less taxing on us and the watch.

[JF002]

@joaquimorg Thanks for all these informations, and I fully you prioritize your job on this project :)

I don't think we have a use-case for a "file explorer" API right now but we might need a way to remove older resources when we upgrade to a new version to avoid filling the memory with data that are not needed anymore.

I do not think that DFU can be used, since it only allows to have the Bootloader, Softdevice and Application in the payload

Technically, we might be able to use the DFU protocol, as the field for the data type is 1 byte long and only 3 values are defined by NRF (other values are "reserved"), so we could use those reserved values to add the resource type, but that wouldn't be "nrf" compliant. But, as you said, dfu does not provide any way to control the files (to delete older/unused files, for example).

So, if I understand correctly, we have to implement

• the filesystem
• the protocol to send data to this filesystem
• the file format of the resources + versioning
• using those resources in the firmware and handling the case when the resources are not available.

[joaquimorg]

Base support for littlefs added in #438

• the filesystem
• the protocol to send data to this filesystem
• the file format of the resources + versioning
• using those resources in the firmware and handling the case when the resources are not available.

[ialokim]

Base support for littlefs added in #438

• the filesystem
• the protocol to send data to this filesystem
• the file format of the resources + versioning
• using those resources in the firmware and handling the case when the resources are not available.

If I investigated correctly, #756 should be the second bullet point, right? I would like to help working on the next step to get the external memory usable for resources, but I just wanted to ensure I didn't overlook some work that has already been put into specifying the file format / directory structure for the resources. If no one objects, I would open a new issue with some proposals.

Just as a recap, what kind of resources do you want to support? I can think of:

• fonts, potentially somehow subdivided to support different special characters (see Umlauts (UTF-8) in notifications #212 or Long notifications don't display properly #388)
• images (to support watch faces similar to the ones in https://github.com/joaquimorg/PinetimeLite)
• strings for internationalization (see Translation (i18n) support #363)

[JF002]

You are right, #756 is the 2nd bullet! So now, we have the filesystem and an API to send data to this file system from a companion app.
We currently haven't specified any file format or directory structure so your analysis and proposals are more than welcome!

The list of resources you mentioned looks quite right. Note that we don't need to support all of them in one shot. We can start with only one type of data, the one we'll find the easiest to implement, and add the other ones later one.

For example, for the December Pine64 community update, I (with the help of @geekbozu) wrote a very simple change in the Digital watch face to display a background image if it's available on the file system. I think that would be a nice first step to implement.

Note that we'll probably want to add a bullet point : "improve the performance of the SpiNorFlash driver and/or fileesystem integration", as reading a full picture from the filesystem slows down the whole UI.

[geekbozu]

So something to note, A lot of the slowdown has to do with how often LVGL opens the image and how much of it it can cache. Since LVGL is "blitting" the image it has to read the image and seek in it every line...

A large speedup will be using properly "compressed" (Smaller Bit per pixel with pallet data) images on the SPI flash instead of the demo we put together which loaded uncompressed images.

That needs a LVGL Image Decoder To handle this "compressed" format.

[JF002]

I've finally decided to focus on this feature. I've created a new project dedicated to this topic.

First, a did a few benchmark of a first use-case : read a picture and use it as a full screen background for the digital watchface.

First, using InfiniTime without any change to the FS layer. Vertical scroll looks good, and similar to @joaqimorg results on this video. However, left/right animation are veeeery slow.

infinitime-lfs2.mp4

I compared my results with PineTimeLite, @joaquimorg 's fork. Both animations are very fast. This implementation uses a "raw" FS layer instead of LittleFS.

pinetimelite.mp4

I integrated this raw fs layer in InfiniTime and observed similar results:

fslayer-from-pinetimelite2.mp4

So, my results are consistent with the ones from @joaquimorg, great!

The current implementation with LittleFS (which is still our preferred one) is quite efficient for linear readings, but way too slow for random read accesses needed by the left/right animation.
During the left/right animation, LFS seeks in the file in the correct positions and then reads 8 bytes. This translates to 2 to 8 read accesses on the SPI bus. I think this is LittleFS that is looking for the block that contains the data and this is probably what's causing this huge slowdown...

Next, I'll try other use-cases : read smaller pictures and icons, and also fonts. The goal is to check which use-cases could already be implemented and which ones need more work!

[JF002]

Here's an update on the font test : loading font from the external flash memory works out of the box. The code is pretty simple :

lv_font_t* font = lv_font_load("F:/font1.bin");
...
if(font != nullptr)
    lv_obj_set_style_local_text_font(label_time, LV_LABEL_PART_MAIN, LV_STATE_DEFAULT, font);
...
lv_font_free(font);

I grabbed 2 random fonts, converted them using lv_font_conv tool and sent them to my PineTime using ITD. Here's an example with a big font for the digital watchface:

And a smaller font in the SystemInfo app:

There's no visible impact on the performance of the display : everything is as fast as with the built-in fonts. Great!
However, this is not so great regarding the RAM memory usage : when calling lv_font_load(), LVGL loads the whole font (header + font data) in memory. Which means that it allocates (lv_mem_alloc()) a few big buffers in memory.

Here are the memory usages of those fonts:

• big font with only characters from 1 to 9 (digital watchface) : 6202 bytes
• small font with "ascii characters" (systeminfo) : 5161 bytes

The memory currently available to LVGL is 14KB in total, so this leaves enough room to load 1 or 2 fonts at a time, depending on the current app running. However.. this is not always true due to memory fragmentation... If memory is fragmented and if it cannot find a memory block big enough to load the font, font loading will fail. I've also noticed that the digital watchface would crash because further allocations needed by the watchface would fail too...

The way LVGL loads the font in RAM is very good for performances, but it'll be a bit challenging in our memory constraint device :)

[JF002]

New experiment : implement a new font engine that reads the data from the file in the SPI flash instead of reading the data (previously copied from flash to ram) from RAM memory.

This is a incomplete font engine. It does not dynamically read the font and glyph descriptions from the font data to read the glyph sat the correct position in the file. It just reads a fixed amount of bytes approximately at the position of the glyph. So the result is not as good looking as it should.

In this video, the big font for the time is read from the file:

font_flash_no_bg2.mp4

In this one, both the background and the font for the time are read from the file:

font_flash_bg2.mp4

As you can see, reading the font from the file slows down the scroll animation a little bit.

As I wrote above : this font engine is not complete and does not read all the data dynamically from the file. It also does not handle glyph buffers that are not aligned on 1 byte (if the buffer is not aligned on 1 bytes, LVGL reads data 1 byte at the time, see this). So these demo are probably a bit more optimistic than what we would get with a complete font engine...

[nische]

Hi,

I know that the ESP32 can load elf (binary-files) as libs from the SPI-Flash in RAM.
Do you think that is possible too?

That could be a very nice way to Store Games and Features that the user dont need much.

BR, NiSche

[JF002]

@nische Yes, it's definitely possible to do that with the PineTime too. However the ESP32 has 320KB or RAM and its SPI bus runs at 80Mhz, while the NRF52832 has only 64KB or RAM and a SPI bus at 8Mhz. Those limited resources makes the implementation of loadable apps a bit more challenging ;-)

[nische]

"a Bit" :😅

Maybe Like the Apps in Waspos. You get a List of all Apps and choose witch one will able to Run. If you uncheck a App the memory will released etc.

Maybe i find some Paper for the nrf5 Chips to get a easy entry in a soldution

[JF002]

Waspos is written in Python. The interpreted language allows to easily load and unload modules. Such flexibility is more difficult to reach with compiled languages like C and C++.

[JF002]

Here's a new "real life" example based on the G7710 watchface. In this video, the 3 fonts needed by the watchface are loaded in RAM at runtime when initializing the watchface.

g7710-external-fonts.mp4

As expected, the refresh of the display is quite fast as the fonts are effectively read from RAM. However, you can see that there's a bit of latency between the swipes and the beginning of the transition to the watchface. This is caused by the loading of the fonts from the external memory to the internal RAM.

Those fonts use ~8KB of RAM from the heap allocated to LVGL. Memory available in this heap goes from ~10-13KB to 2-4KB at run time.

[JF002]

In the next video, I tried to workaround the very slow loading of watchface that use a full-screen bitmap as background (see here) : the loading of the background is deferred until the 1st frame, so that it's not done in the ctor() of the watchface.

defer-bg-load.mp4

This ensures that the background will be read from the external memory in 1 chunk instead of several small chunks when it's read during the transition.

What do you think of the result?

[Avamander]

Looks great. I guess if the image is not rendered under the changing text, it wouldn't even be very noticeable.

[JF002]

New test with Infineat watchface (logo and fonts loaded from external memory) :

infineat-extmemory2.mp4

Fonts and logo from this comment, code based on the branch infineat-color by @dmlls.

[JF002]

Let's put everything together : https://video.codingfield.com/w/swqopgt9p561ZBiTtxbTAf

I'm honestly quite impressed by the result!

Infineat by @dmlls
G7710 by @ITCactus

[JF002]

I pushed the code of the above demo in this PR. Feel free to test it and provide feedback (but read the warning first :)).

I tested it for a few days and enjoyed those new watchfaces ! The user experience seems quite good, and the loading of the big pictures could be improved by using more compressed (less colors) pictures.

At this point, I think that the performances are good enough to be used in InfiniTime. This new feature will allow to add a few more watchfaces, which will be more that appreciated by a lot of users (me included!).

But before rolling out this feature, we first need to figure out how to integrate it nicely in InfiniTime so that it's easy to use and error proof :

• Resource files should be converted/generated at build time (similar to the font generation feature) : Package resources + a metadata file into a single .zip file. #1299
• It must be easy for the user to upload and update the resources : we don't want the user to directly interact with the BLE FS API to send all resource files one by one : Resources (fonts & pictures) packaging #1285 - Package resources + a metadata file into a single .zip file. #1299
  • How to package the resources in a single file (.zip file with resources + manifest/header?) Package resources + a metadata file into a single .zip file. #1299
  • How to upload it (we'll need the collaboration of companion app developers). Package resources + a metadata file into a single .zip file. #1299
• InfiniTime must handle the case where an app/watchface needs a resource from the FS and the corresponding file does not exist
• How to handle RAM memory usage for the fonts (when a font from the ext. memory is needed, it is entirely copied from flash to RAM). What if there is not enough memory available at run-time (due to RAM usage and/or fragmentation) ?

[Zandengoff]

How to upload it (we'll need the collaboration of companion app developers).

Do you know if anything has been added to the gadgetbridge issues list for this?

[JF002]

@Zandengoff I've already contacted developers from Gadgetbridge and Amazfish, and they say that adding the feature should not be an issue. Now, I think it's up to us (InfiniTime developers) to describe the upload procedure with more details (how will the data be packaged, how to handle update of the resources, how to remove older versions,...) so companion app developers have all the necessary info to implement it.

[Zandengoff]

@JF002 Excellent, wanted to help with the request if needed, but seems you are already ahead of the curve.

[JF002]

I'll close this issue/feature request as the external resource feature will be release in InfiniTime 1.11.Other topics (buffer for HR/Steps and other usage of the external flash memory) will be discussed in their dedicated issues.

Thanks everyone for your help on this huge topic!