Reduce memory footprint

[justcallmekoko]

ESP32 Marauder is running out of space. Pretty soon we won't have enough space for new features and this is a problem. An easy solution would be to just replace the ESP32-WROOM with an ESP32-WROVER, but that would make the current hardware obsolete.

[tobozo]

are you still loading the BLE stack ?

just take a couple of hours and try to adapt Nimble-Arduino instead of the original BLE library, it's really worth the effort and you'll get the space you need (and probably more)

[justcallmekoko]

Alright I will give that a try when I have some free time. Thanks for the recommendation.

The biggest hog is the BLE

[justcallmekoko]

@tobozo It worked

What is actually pretty interesting is that not only do you get some extra free memory right off the bat when you turn marauder on, you get an additional 15K when you run a bluetooth function then go back to the menu.

I suspect this is probably because of my bluetooth cleanup after exiting bluetooth functions.

Next thing to do is to figure out what to do with all of the free memory. Maybe make the SD Writing buffer bigger for less writes? More functions? The world is our oyster!

[tobozo]

yay! no more need to put a wrover then ?

[justcallmekoko]

I don't think so. The WROOM should suffice. Another thing about this change that was very interesting was that during the bluetooth scans with the old BLE stack, the memory would go down from 140K to 30K.

Now when you run a scan, it will go from 150K to 120K. That is just awesome.

[tobozo]

Have you tried with a smaller partition ? Some example sketches from Nimble-Arduino can compile with WiFi using the default partition size, so who knows ?

Also I know I'll sound redundant, but switching from TFT_eSPI to ESP32-Chimera-Core (or at least LovyanGFX) can provide an additional relief.

The M5Stack/Odroid-Go proof of concept I've made available for you on the private fork was using a TFT_eSPI based version of the ESP32-Chimera-Core along with some partial integration of the buttons for M5Stack and Odroid-Go.

Now I'll need to find a better way than using the default boards.txt from the Arduino Boards Manager for TFT/SD/Touch profile detection, so there's a high probability I'll have to add an "ESPMarauder" profile to the ESP32-Chimera-Core supported devices list and share some Arduino IDE boards.txt customizations with you to make those changes usable.

All in all, I can do this integration again with the LovyanGFX version while keeping the current logic for Touch unchanged.

Please let me know if you want to give a try in that direction.

[justcallmekoko]

Well you certainly have much more experience than I do with graphics libraries so I will trust your judgment on this one. I could definitely use some help on it for sure.

We can go with your recommendation to change gfx libraries on a new branch

[tobozo]

I'm still trying to get the Touch library out of TFT_eSPI but the #define mess is so huge I'll probably go with a custom implementation.

Also from what I've seen while doing my research is that there are several ways to implement touch, here's the one I'm using:

Spi setting:

• Sharing SPI with the TFT (will require spi transactions depending on the TFT setup)
• Using dedicated HSPI (requires extra wiring)
• Using dedicated VSPI (requires extra wiring)

Touch Setting:

• Using CS only (requires polling, creates false positives)
• Using CS with IRQ (requires extra wiring)

I'm thinking about taking PaulStoffregen/XPT2046_Touchscreen code and making its API compatible with TFT_eSPI to minimize the code migration effort, but I'll probably have to import some of its code, at least to deal with calibration and false positive readings.

@Joshinken's does the Touch interface of your custom build share the SPI with the TFT and are you using the T_IRQ pin or just the T_CS ?

also @justcallmekoko from your schematics can I safely assume you're also sharing SPI and using only CS for the touch interface ?

[justcallmekoko]

@tobozo I need to update the schematic because the SD interface is also involved now, but yes I am sharing SPI.

But this is exactly why I am happy you're helping with this. You have much more experience and knowledge on this subject.
You know what you're doing and you're good at reading my trash documentation and trash code.

[tobozo]

youpi I have succeeded into dropping TFT_eSPI from ESP32Marauder, now it's compiling and running fine on my custom LoLinD32Pro + ILI9341 build

The sketch now compiles using the default partition size, more SPIFFS space \o/

there was a tradeoff though, probably not an issue on other builds, but I had to remove the SD Card from the slot in order to get the display correctly detected, while this may be isolated to my situation there's still a solution for that, so let me know if this happens to you too (symptom : LoLinD32Pro is detected as M5Stack)

SD support, battery/neopixel and other stuff I don't have on my build have been disabled in the setup(), feel free to re-enable them, but only after testing :-)

tobozo@a5d9317

requirements :

1. use the very latest version of LovyanGFX you can find in the Arduino Library Manager
2. use the "Touch" branch of ESP32-Chimera-Core
3. select the "LoLinD32Pro" from the boards menu when compiling

[justcallmekoko]

Another consideration. With this extra space, we might be able to fit LVGL in there for some keyboards.

[justcallmekoko]

Check out this commit f86b5a4

I was able to only use SD when needed. When not needed, memory is release.
Now when just on the menu, you have as much RAM as you would if you didn't have the SD card inserted (aside from buffer_obj).

[justcallmekoko]

Never mind. I had to revert. There is some sort of memory leak associated with stopping and starting SD over and over.

[tobozo]

progress!!

I finally got the SD Card to work with the generic Touch interface but I'm still experiencing minor issues.

My ESP32Marauder SD Card interface doesn't like reboots or crashes, I can't tell if this is caused by my build or just clumsy code.

Occasional symptoms are : undetected SD and/or broken touch support, all fixed by a full power cycle or SD Card reinsertion + reset.

I haven't implemented your last changes though, I'll need to know if you're experimenting the same problems with this version before going forward in the git history, also I've removed some brutal statements from the code like the ceisure-inductive use of tft.init() and added some cosmetic changes such as menu transitions and smooth scrolling, obviously those will need to be reviewed before adaptation, but the dynamic memory use is down to 17% while fitting in a default partition.

Sketch uses 1283646 bytes (97%) of program storage space. Maximum is 1310720 bytes.
Global variables use 58052 bytes (17%) of dynamic memory, leaving 269628 bytes for local variables. Maximum is 327680 bytes.

https://github.com/tobozo/ESP32Marauder-1

[justcallmekoko]

Can't wait until all the kinks are worked out.

Hey @tobozo quick question. I was looking at the capabilities of the Flipper Zero and it has something that allows the user to install firmware plugins on top of the base firmware for if you want to run some peripheral devices or something like that.

Do you know of a way to do that on the ESP32? Maybe place some circuit python files or other arduino sketches in a directory on an SD card so that the marauder can import them as a plugin or an app to run on top of the base firmware.

[tobozo]

Can't wait until all the kinks are worked out.

can't wait to have your feedback on that SD Card problem so I can tell the kinks are only on my build :-)

allows the user to install firmware plugins on top of the base firmware

you can't mix platforms binaries (micropython, arduino, platformio, nodemcu, bare C) on flash space unless they all use the exact same partition scheme, which is incredibly difficult to achieve in a maintainable fashion

however, by sticking with Arduino you only need a few modifications to your actual code to make your existing SDUpdate methods compatible with the Launcher example provided with my M5Stack-SD-Updater library, then maintain as many standalone micro applications as you need, provided they will also implement a way to reload the launcher.

Typical use of this reload routine is : if button [x] is pressed at boot, it loads your "menu.bin" from the SD Card (or loads it from the other OTA partition if already copied there)

when "menu.bin" is loaded, it enumerates the other binaries on the SD Card, to let you choose which micro application to run (e.g. ESP32Marauder, setup the wifi ssid/pass, sync rtc to ntp server, run a web server, do other stuff that need memory)

when done with the app, just push [x] and [reset], release [reset] while keeping [x] pressed, and you're back to the launcher

rinse and repeat

[justcallmekoko]

The NimBLE was the perfect update