Compact vars branch now merged

[gfwilliams]

Posted at 2021-08-05 by @gfwilliams

Hi,

For year or two now there's been an experimental_compact_vars branch. This branch re-organises the contents of the 'JsVar' inside the interpreter to save a few bits that were otherwise not usable. If you have 1000 variables for instance you can use 10 bit addresses, rather than 16 bit - and this saving really adds up.

In Bangle.js, JsVars can use only 13 bytes, not 16, which gives you over 20% more variables available.

As far as I know, the only person that's actually used this is @MaBecker, but given the issues people seem to be having with Bangle.js memory it seemed worth merging this in now, as I don't actually see any regressions from using it (apart from maybe a minor slowdown).

I have just merged the branch into master so you should see new builds appearing in https://www.espruino.com/binaries/travis/master/ that you can try out. Builds for one or two devices are not appearing yet as it seems I need to do some work on reducing code size where it's a big tight, but otherwise I think this should represent a good improvement.

Please let me know how you get on!

[espruino-discuss3]

Posted at 2021-08-05 by @MaBecker

I have just merged the branch into master so you should see new builds appearing in

Great - many thanks!

---

Posted at 2021-08-05 by @MaBecker

Is there a formula to calc the new variables value?

---

Posted at 2021-08-05 by @fanoush

I think this should represent a good improvement.
Please let me know how you get on!

Wow, looks interesting. Could be useful to force old 16 byte way via some define in board file to compare otherwise same build version on two identical devices with same js code.

So there are more variables but smaller. Can it take more memory in some cases? Like something fitting two 16byte variables before (=32 bytes) that would need three variables now (=39 bytes)? Or even needing two now (=26) that would fit one variable before? That something being e.g array with one or two values, some typical string or byte array length, simple number variable with some typical variable name length ... just guessing, have no idea.

Also slightly related idea, can we have one or two more bits for array size so typed arrays over 64k are possible on 52840 (with 256k of ram)?

---

Posted at 2021-08-06 by @gfwilliams

Could be useful to force old 16 byte way via some define

Yes, good point...

Can it take more memory in some cases?

Yes... The main ones I can think of are:

• Arrays with numbers between -32768 and 32767 would be able to pack both the index and value into one variable. Now that number is lower - I guess between -2048 and 2047 on most boards.
• Non-flat strings aren't quite as efficient now. Each extra bit of string has an overhead of 4 bytes, so now the blocks are smaller it's roughly 30% overhead, not 25%.

However in reality I don't think much of that matters. Any String that you actually allocate in one chunk that's big enough gets turned into a flat string now, where the overhead is just one JsVar of 13 bytes (where previously it was 16 bytes).

And generally if you're storing a lot of numbers in an array you should probably have been using typed arrays anyway :)

can we have one or two more bits for array size so typed arrays over 64k are possible

Can you file an issue for this? I think worst case I should be able to add some bits from JsVarDataArrayBufferViewType to get us up to 256k.

---

Posted at 2021-08-06 by @allObjects

...good point... on the other hand, smaller chunks leave less unused, but small chunks need more chaining links... may be it ends up as an overall loss.

---

Posted at 2021-08-30 by @fanoush

Just to let you know that there is issue with native/InlineC code related to this. This was perhaps not found on ESP port as the native code feature is not there. Created issue espruino/Espruino#2040
Basically the 16 or 12 byte size variables had nice feature that they are long word aligned in memory so native code flat string produced by atob was always aligned. Now when it does not begin on 4 byte boundary it crashes/reboots when native code is run. Not sure there is easy/nice solution.

In fact I now wonder how it works with Uint16 or Uint32 arrays, maybe they are not aligned too? That would be a bit unfortunate as well, will try.

---

Posted at 2021-08-31 by @gfwilliams

maybe they are not aligned too?

Yes, I doubt they are aligned - probably best to discuss this on that issue