use RESIZABLE_JSVARS for other boards than linux ?

[espruino-discuss]

Posted at 2018-02-06 by JumJum

AFAIK, RESIZABLE_JSVARS are set for linux only.
Would it be possible to use this for other boards, like ESP32, too ?
I've some boards with additional psram, so memory is available like crazy.
Since psram needs a driver, its not available from start, means cannot be used for statically allocated block

Idea would be to check available heap during init and based on that malloc, memory for JsVars.
There would be a definitions in board.py for large heap and for small heap.

If something sounds as easy as this, usually there are some hidden problems,....
Any comment ?

[espruino-discuss]

Posted at 2018-02-06 by @gfwilliams

Yes, I think that could work. It's worth a try...

However:

• Resizable JsVars may default to using 32 bits for JsVarRef when actually 16 bit would do fine - it'd be worth checking
• It's a bit slower for JS execution, but maybe that doesn't matter too much?

---

Posted at 2018-02-06 by JumJum

hmm, we have 2 versions_

• esp32 wroom, here we fight for each byte
• esp32 wrover, here we have plenty of memory
  I would support one solution which supports all.

Would it be possible to replace static allocated block with malloc allocated block ?
In that case it wouldn't be resizable, between us, I would pay this price :-).
My understanding of JsVars is poor, would 32 bit reference be a problem ?

My expectations for speed is the same. Espruino supports an option to move all WIFI and BLE memory to psram. I'm pretty sure, they would not do this, if its much slower.

---

Posted at 2018-02-07 by @gfwilliams

Ahh, I think really there are 2 options...

• malloc the jsVars data at boot time. This should be totally fine to do with everything basically as-is - but it's not the same as RESIZABLE_JSVARS
• RESIZABLE_JSVARS is where extra jsVars are allocated as needed, and it's what is done on Linux. The issue here is that in order to do that there's not just one jsVars array, but different 'blocks' of jsVars - since you can't just move a JsVar around in memory if some bit of code has its address in a pointer. Personally, I'd avoid doing that since having the different blocks slows down everything since it's no longer a simple lookup to find a variable's address from its reference.

About 32 bits - yes, it's bad :) See http://www.espruino.com/Internals for the layout - 32 bits means vars are no longer 16 bytes each, but are 24 (IIRC) - so you now only get 2/3 of the amount of variables in the same RAM :(

---

Posted at 2018-02-07 by JumJum

Hmm, for ESP32 I would prefer simple malloc solution.
I checked this in a first simple test(see code at the end) and in a first test it works fine.
But doing board specific changes in jsvar.c is not my favourite solution.

Anyway I'm still confused about the impact of 32bits.
In internals link and in source of jsvar I only find 12 byte or 16 byte.
There is a note JsVars for 32 bit refs are similar, but what does this mean ?
Same for build_platform_config.py, which checks for less than 1024 only.
In which case would we need 24 bytes instead of 16 ?

BTW is there a limit for variables ? I can see ESPRUINOWIFI has 7135.
What would happen with 10000 or 50000 jsvars ?
It's a kind of theoretical question, but having 4MB heap, .....

#ifdef RESIZABLE_JSVARS JsVar
**jsVarBlocks = 0;
unsigned int jsVarsSize = 0;
#define JSVAR_BLOCK_SIZE 4096
#define JSVAR_BLOCK_SHIFT 12
#else
#ifdef ESP32
JsVar *jsVars = NULL;
unsigned int jsVarsSize = 0;
#else
JsVar jsVars[JSVAR_CACHE_SIZE];
unsigned int jsVarsSize = JSVAR_CACHE_SIZE;
#endif
#endif

and this:

void jsvInit() {
#ifdef RESIZABLE_JSVARS
jsVarsSize = JSVAR_BLOCK_SIZE;
jsVarBlocks = malloc(sizeof(JsVar*)); // just 1
jsVarBlocks[0] = malloc(sizeof(JsVar) * JSVAR_BLOCK_SIZE);
#endif
#ifdef ESP32
jsVars = (JsVar *)malloc(33600);
jsVarsSize = 2100;
#endif
jsVarFirstEmpty = jsvInitJsVars(1/first/, jsVarsSize);
jsvSoftInit();
}

---

Posted at 2018-02-08 by @gfwilliams

Normal JsVarRefs for most boards are 16 bits, so can store up to 65535 variables. But the Resizable vars build can do 4 billion because of 32 bit references (I believe). But adding 32 bit references really pushes the size of a JsVar up, so you don't want to do that.

Basically, what you've done is fine - although ultimately I guess you'll want to set jsVarsSize to something else before calling jsvInit?

I would do this instead in the second bit though - no point hard-coding numbers where it's not needed:

#ifdef ESP32
jsVarsSize = 2100;
jsVars = (JsVar *)malloc(sizeof(JsVar)*jsVarsSize);
#endif

---

Posted at 2018-02-08 by JumJum

Hardcoded value was for easy testing only :)

Hmmm, I could move the functionality to assign jsVarSize into function main of jshardware.c.
In there would be a switch based on heap size (or psram recognition) to assign "small memory VarSize" or "big memory VarSize"
Both values should be in ESP32.py.
Hmmm, have to check how these values will make it to platform_config.h. This should be no problem.
AFAIK Everything could be done without changes in jsvInit(jsvar.c), but I would prefer to make it more general. Means adding one more definition in ESP32.py, something like "board defined jsVarSize". This way we would add a more general functionality for other boards in the future.

What would you prefer ? Or is there another(better) option, which did not come to my mind ?
Anyway, I've a starting point now for more testing.

---

Posted at 2018-02-08 by Wilberforce

My be a bit hacky.... and need a suitable comment.

Set the jsvars to 0 in the board py to mean allocated at runtime? And then fix anything that depends on that?

---

Posted at 2018-02-08 by DrAzzy

If you've got 4mb of PSRAM to play with, why wouldn't one make the whole 64K jsvars permitted by the 16-bit addressing available?

---

Posted at 2018-02-09 by Wilberforce

If you've got 4mb of PSRAM to play with, why wouldn't one make the whole 64K jsvars permitted by the 16-bit addressing available?

Sure - for the psram case - @jumjum was looking to have one firmware that could be used on both psram and the non-psram boards as I understand it.

---

Posted at 2018-02-09 by JumJum

@wilberforce,
thats correct, I'm looking for one firmware running on all (known) ESP32 boards.
Right now, I need a change in ESP-IDF.
In actual version there is an option to use psram, but if no psram is found during startup, it aborts.
Better would be an option (choice) with abort or log missing psram only.
Just tested this, but its a change in ESP-IDF, so they need to accept.

---

Posted at 2018-02-09 by @gfwilliams

Set the jsvars to 0 in the board py to mean allocated at runtime?

That sounds like the best idea, and wouldn't require any board-specific defines in jsvar.c, which is a massive bonus.

---

Posted at 2018-02-09 by JumJum

Could we move definition for jsVarBlocks, jsVarsSize and jsVars into jsvars.h ?
Other option would be to add a function to jsvar.c to set jsVarsSize from main.c

As always, if you have any better idea...

---

Posted at 2018-02-09 by JumJum

I got it running this way.
Simple board gives me 2000 vars
Board with additional PSRAM gives me 20000 vars.
@drazzy, we could use 65500 here too

ESP32.py

 'variables'                : 2000,
 'variables_psram'          : 20000,
 'variables_mode'           : "malloc",             ESP32 uses malloc

jsvar.h

extern unsigned int jsVarsSize;

jsvar.c

#ifdef RESIZABLE_JSVARS
JsVar **jsVarBlocks = 0;
unsigned int jsVarsSize = 0;
#define JSVAR_BLOCK_SIZE 4096
#define JSVAR_BLOCK_SHIFT 12
#else
#ifdef VARIABLES_MODE_MALLOC
unsigned int jsVarsSize = 0;
JsVar *jsVars = NULL;
#else
JsVar jsVars[JSVAR_CACHE_SIZE];
unsigned int jsVarsSize = JSVAR_CACHE_SIZE;
#endif //end VARIABLES_MODE_ALLOC
#endif
......
void jsvInit() {
#ifdef RESIZABLE_JSVARS
  jsVarsSize = JSVAR_BLOCK_SIZE;
  jsVarBlocks = malloc(sizeof(JsVar*)); // just 1
  jsVarBlocks[0] = malloc(sizeof(JsVar) * JSVAR_BLOCK_SIZE);
#else
#ifdef VARIABLES_MODE_MALLOC
  jsVars = (JsVar *)malloc(sizeof(JsVar) * jsVarsSize);
#endif
#endif 
  jsVarFirstEmpty = jsvInitJsVars(1/*first*/, jsVarsSize);
  jsvSoftInit();
}

build_platform_config.py

if board.info["variables_mode"]:
  codeOut('#define VARIABLES_MODE_'+board.info["variables_mode"].upper() + ' // mode for allocating memory, standard is statically assigned');
  codeOut('');
...
  if board.chip["class"]=="ESP32":
    codeOut("#define JSVAR_CACHE_SIZE_PSRAM          "+str(board.info['variables_psram'])+" // Number of Javascript variables in RAM for boards with additional RAM (like ESP32 WROVER)");

last not least in main.c

  if(esp_get_free_heap_size() > 0x300000) jsVarsSize = JSVAR_CACHE_SIZE_PSRAM;  // looks like 4MB SPI_RAM is enabled for heap, pretty sure there is a better way, but for now ....
  else jsVarsSize = JSVAR_CACHE_SIZE;   // number of variables for boards without additional SPI RAM
  jsvInit();     // Initialize the variables

---

Posted at 2018-02-09 by @gfwilliams

That looks good - I guess you could change jsvInit to take the size as a parameter?

If someone's interested in it, there's the possibility of using this on some STM32 devices - for instance the original Espruino board has a chip that's only supposed to have 48kB of RAM, but which actually has 64kB (I believe). I just can't be sure every chip has it so I can't enable it in firmware.

Potentially some code could check for the RAM at boot and rearrange the stack to use what was available - although it's more difficult as Espruino for STM32 doesn't use malloc :)

---

Posted at 2018-02-09 by DrAzzy

So a build that would be like the bigram builds I used to do (at some point, the toolchain stopped working, and then I was forced to rebuild the system by amazon and forgot how to set it up anyway), only it would auto-detect whether that RAM was actually there? That would be really cool

An esp32 build that had 64K jsvars would be pretty cool. Would finally be hard to run out of ram on espruino.

---

Posted at 2018-02-09 by JumJum

@gfwilliams,
changing jsvInit to use size as a parameter would be easy.
But there are some other boards, that also use jsvInit, they would need to be changed too. Something that I would not like to do.
AFAIK, C does not support optional parameters. If somebody knows better, I'm still on first part of learning curve, please give me a hint.
@drazzy, I'm pretty sure, sooner or later there will be someone to run out of memory, even with 64K :-)

---

Posted at 2018-02-09 by @gfwilliams

@jumjum I can do that, but I've got a bunch of other stuff on my list at the moment. Maybe let me know when it makes it into ESP-IDF and I can do the changes then?

So a build that would be like the bigram builds I used to do

Yep, that's the idea.

@drazzy maybe you know something about this: https://github.com/espressif/ESP8266_MP3_DECODER

For that they needed more RAM on an ESP8266 so they piggy-backed an SPI RAM chip onto the SPI flash. Do you think that would end up getting exposed in the ESP8266's address space as well? Could be a really interesting way of boosting cheap ESP8266s as well?

---

Posted at 2025-01-10 by dponyatov

What should I add to board configs and compiler defines to make build of Espruino 2v22 for STM32F4 with RESIZABLE_JSVARS enabled?

I added -DRESIZABLE_JSVARS into compiler options, but got this error:

/usr/lib/gcc/arm-none-eabi/12.2.1/../../../arm-none-eabi/bin/ld: /usr/lib/gcc/arm-none-eabi/12.2.1/../../../arm-none-eabi/lib/thumb/v7e-m+fp/softfp/libg_nano.a(lib_a-sbrkr.o): in function `_sbrk_r':
/home/pere/src/newlib-salsa/build_nano/arm-none-eabi/thumb/v7e-m+fp/softfp/newlib/libc/reent/../../../../../../../../newlib/libc/reent/sbrkr.c:51: undefined reference to `_sbrk'
collect2: error: ld returned 1 exit status
make[1]: *** [make/targets/ARM.make:3: bin/horizon_2v22_dponyatov_250110.elf] Error 1
make[1]: Leaving directory '/home/dponyatov/Espruino'
make: *** [mk/all.mk:5: all] Error 2

What I need is enabling block allocation mode but build fw with three precompiled blocks:

• RAM 128K
• CCMRAM 48K (3/4 of CCMRAM)
• optional extRAM (up to 16 Mb)

PS: I take old 2v22 thus modern versions compiles into ELF file with broken LMA/VMA table but I don't have time to fix it (all source code debug info bounded to zero address etc)

---

Posted at 2025-01-10 by @fanoush

What I need is enabling block allocation mode but build fw with three precompiled blocks:

So you don't actually need resizable vars with its dynamic memory allocation, you need just 3 fixed separate blocks right from the start. sbrk is method to allocate memory (maybe malloc calls it internally?) - you don't need this to solve your problem?

I don't think there is a solution for this combination but it could be useful. I'd like to have something similar on nrf52 too, currently the most simple way used for nrf52 and stm32 is (one) big static jsVars array which is put randomly somewhere by the linker just like any static variable. What I used before with nrf52 is to put it into fixed memory area - not malloc'ed but also not static array variable. What you want is advanced variation of this with 3 blocks. it could be implemented like RESIZABLE_JSVARS blocks but still fixed and all 3 created at startup instead of allocating dynamically.

---

Posted at 2025-01-10 by dponyatov

@gfwilliams: Did you maybe tested some experimental version of JsVars allocation for multiple non-continues memory banks for some board with all tiny-bits optimizations enabled?

As I see in source & debugger, for STM32F4 series there is defult build used with a single preallocated array isn't it?

void jsvReset() {
    #ifdef RESIZABLE_JSVARS
    #else
        memset(jsVars, 0, sizeof(JsVar)*jsVarsSize);
    #endif

---

Posted at 2025-01-10 by @gfwilliams

So far I've only used RESIZABLE_JSVARS on Linux - otherwise you have to build in malloc/free/etc which we don't currently do (and it hurts performance).

If you want 3 distinct blocks I'd add jsVars2/jsVars3 arrays where you want them and then make your own jsvGetAddressOf function. I think the only other thing you need to worry about is in jsvNewFlatStringOfLength where there's currently an RESIZABLE_JSVARS ifdef - and you need to ensure that gets added.

---

Posted at 2025-01-10 by @fanoush

PS: I take old 2v22 thus modern versions compiles into ELF file with broken LMA/VMA table but I don't have time to fix it (all source code debug info bounded to zero address etc)

That looks related to your previous conversation https://forum.espruino.com/conversations/401187/
There is nothing wrong at having the code linked to address 0. It is not broken as already explained there. If for some reason you want to have it at 0x08000000 it is possible and was described there how to do that - the NUCLEOF401RE board uses such configuration.

I just compiled STM32F4DISCOVERY target which is linked to address zero too and in the lst file I see

bin/espruino_2v25.20_stm32f4discovery.elf:     file format elf32-littlearm
bin/espruino_2v25.20_stm32f4discovery.elf
architecture: armv7e-m, flags 0x00000112:
EXEC_P, HAS_SYMS, D_PAGED
start address 0x00000c01

Program Header:
    LOAD off    0x00001000 vaddr 0x00000000 paddr 0x00000000 align 2**12
         filesz 0x0006cf90 memsz 0x0006cf90 flags r-x
    LOAD off    0x0006e000 vaddr 0x20000000 paddr 0x0006cf90 align 2**12
         filesz 0x0000011c memsz 0x00012c5c flags rw-
private flags = 0x5000200: [Version5 EABI] [soft-float ABI]

Sections:
Idx Name          Size      VMA       LMA       File off  Algn
  0 .isr_vector   00000188  00000000  00000000  00001000  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  1 .ARM          00000008  00000188  00000188  00001188  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  2 .text         0006ce00  00000190  00000190  00001190  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  3 .data         0000011c  20000000  0006cf90  0006e000  2**3
                  CONTENTS, ALLOC, LOAD, DATA
  4 .bss          00012b3c  20000120  0006d0b0  0006e11c  2**3
                  ALLOC
  5 .ARM.attributes 0000002e  00000000  00000000  0006e11c  2**0
                  CONTENTS, READONLY

and it looks like both VMA and LMA are correct, they point to 0 where the code really is, so there is nothing to fix with VMA/LMA. If you still have some issue with that, then maybe continue in that previous conversation you created.