reading large files with readArrayBuffer

[espruino-discuss]

Posted at 2023-01-11 by wagnerf42

hi,

i'm trying to read a large file (200kb).
(i just tested on the emulator and not the watch.)

using : let a = require("Storage").readArrayBuffer("test.map");
it loads but a.length is equal to 22053.

if i try let a = require("Storage").read("test.map", 0, 200000);
it seems to work but i get a string and i'd rather like to turn my arraybuffer into various arrays.

is there a size limit on readArrayBuffer and is there a reason for it ? do you know any kind of workaround i could use ?

thanks.

[espruino-discuss]

Posted at 2023-01-11 by @fanoush

unfortunately there is 64KB limit for arrays, may be related to this issue espruino/Espruino#2036

[espruino-discuss]

Posted at 2023-01-11 by @fanoush

as for workaround there is E.toArrayBuffer so maybe it could work on <64K substring so you could access big string in 64KB chunks as array.

EDIT: I just tested it in RAM, hopefully it will work fine with flash strings too.

s="".padEnd(65536) // 64K of spaces
s=s.padEnd(65540,'a') // pad with few 'a' over 64KB

>s.substring(65535)
=" aaaa"
>E.toArrayBuffer(s.substring(65535))
=new Uint8Array([32, 97, 97, 97, 97]).buffer
>E.toArrayBuffer(s) // this fails due to 64KB limit
=new Uint8Array([32, 32, 32, 32]).buffer

[espruino-discuss]

Posted at 2023-01-11 by wagnerf42

ok, thanks for the quick answer. i think i actually may be able to have a workaround with inline c and strings

[espruino-discuss]

Posted at 2023-01-11 by wagnerf42

oh i didn't see your workaround. thanks again, i'll give it a try

[espruino-discuss]

Posted at 2023-01-11 by @fanoush

Instead of substrings the best workaround is probably to combine Storage.read with offset,length parameters and wrap the result by E.toArrayBuffer

As for Inline C it may not work as you expect with storage in SPI flash (Bangle watches), the 200K string is not directly in memory but read over SPI by JS interpreter on demand when iterating over it.
It might work with Storage in internal flash but there is not much space for big files there.

[gfwilliams]

Posted at 2023-01-11 by @gfwilliams

Instead of substrings the best worlaround is probably to combine Storage.read with offset,length parameters and wrap the result by E.toArrayBuffer

Yep, I'd say try this...

[espruino-discuss]

Posted at 2023-01-11 by wagnerf42

ok, i'll give it a try. thanks a lot

[espruino-discuss]

Posted at 2023-01-11 by wagnerf42

yeah. it's all good. thanks again.

Attachments:

• gps.png

[gfwilliams]

Posted at 2023-01-13 by @gfwilliams

Wow, very cool - a vector map?

[espruino-discuss]

Posted at 2023-01-13 by wagnerf42

i extracted my city's ways from openstreetmap. with some clever encoding (btw i laughed at Uint24Array but it's just what i needed) i'm actually below 200k. i'm amazed it's fast enough to display on the watch with little optimisations (1.5 secs to display). i still need to add the street names and port the low mem path algorithm to javascript and we might have a full gps. i'm not still 100% sure the path algorithm will fit the ram though, it's kind of a challenge. well, all the stuff i like.