reading parts of a large data file #1358
Replies: 18 comments
-
Posted at 2020-01-11 by Robin Sat 2020.01.11 Could it have been 'DataView' rather than 'DataSet'? > 'Somewhere I read that a statement such as' Will need the source link to better understand and respond. . . . |
Beta Was this translation helpful? Give feedback.
-
Posted at 2020-01-11 by AkosLukacs Storage.read docs agree with wath you say: "This function returns a String that points to the actual memory area in read-only memory, so it won't use up RAM."
You can force it to take up memory by slicing into it for example:
On each invocation memory decreases by You can also try
Oh, and you can get the size of things:
|
Beta Was this translation helpful? Give feedback.
-
Posted at 2020-01-12 by Andreas_Rozek @AkosLukacs, very well done! Thanks for your info! |
Beta Was this translation helpful? Give feedback.
-
Posted at 2020-01-12 by @fanoush
So you didn't use Bangle when trying this? There may be another issue that Bangle uses external SPI flash for storage (?) which is not directly mapped to cpu address space so data needs to be loaded to ram at some point. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2020-01-12 by AkosLukacs Good point, on the Bangle it does use more memory:
|
Beta Was this translation helpful? Give feedback.
-
Posted at 2020-01-13 by @gfwilliams Yes, it's because Bangle.js can't memory map the file. The plan has always been to modify Espruino to add a new String type that can read direct from external flash, so hopefully that'll fix the issue when it's done - as well as freeing up more RAM by allowing seldom-used code to reside directly in flash. But I guess |
Beta Was this translation helpful? Give feedback.
-
Posted at 2020-01-13 by @allObjects ...with this I will get my Having said so and working thru my storage options across all boards including bangle.js and after taking latter apart to look into modification options I could appreciate Did not take a look into the (file) name look-up, but I assume there is no caching for names since memory is in short supply anyway (and no disconnect / unmount needed). |
Beta Was this translation helpful? Give feedback.
-
Posted at 2020-01-14 by Andreas_Rozek I get
right after reading the large file
even without any other statements following. Thus, the docs do not seem to be correct - it's not just a pointer to a memory region in flash which is returned. Just as a remark: my lookup data is > 100kb long. I was able to successfully transfer it from PC to Bangle.js - but there it sits and cannot be used, not even in parts. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2020-01-14 by AkosLukacs Yes, that was confirmed by Gordon and me later. I think the docs should be updated for the Bangle. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2020-01-14 by Andreas_Rozek Well, I will then wait for the updated docs - because, with the current situation, I can't even determine the size of large files... |
Beta Was this translation helpful? Give feedback.
-
Posted at 2020-01-14 by @gfwilliams On latest builds there's now the ability to do The docs aren't updated yet - as I mentioned above the plan is to add the ability to read the file direct from flash at some point soon |
Beta Was this translation helpful? Give feedback.
-
Posted at 2020-01-14 by Andreas_Rozek Thanks a lot! So, there will no longer be the need to slice long binary files into shorter ones? Because accessing a certain part within such a file is performant enough? |
Beta Was this translation helpful? Give feedback.
-
Posted at 2020-01-15 by @gfwilliams Yes, that's correct :) And when the 'direct from flash' mode goes in you'll actually just be able to draw the image directly from flash too |
Beta Was this translation helpful? Give feedback.
-
Posted at 2020-01-15 by Andreas_Rozek Ok, you got me... When will we get that feature? |
Beta Was this translation helpful? Give feedback.
-
Posted at 2020-01-15 by @gfwilliams No idea - next few weeks I hope |
Beta Was this translation helpful? Give feedback.
-
Posted at 2020-01-16 by @gfwilliams Turns out it's today. Builds from http://www.espruino.com/binaries/travis/master now have it in. There are now 'Flash Strings' - they're special strings created only by That means you can use |
Beta Was this translation helpful? Give feedback.
-
Posted at 2020-01-16 by @allObjects does
mean that it is an either-or ? Since this creates dependencies between various things and mostly between the way code is done and built static and uploaded or dynamic at runtime, it is important have the setups and coding documented to not create confusion. I hope my paranoia of experiencing unpredictable / unexplainable effects as a plain js user will evaporate like fog in the sun the moment I understand better that constraint. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2020-01-17 by @gfwilliams Not sure I understand. Basically this is exactly what happens on all other Espruino devices. It's just that in Bangle.js the flash isn't memory-mapped, so some messing around has been required to 'fake' that. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2020-01-11 by Andreas_Rozek
Good morning!
Somewhere I read that a statement such as
would just copy a reference to the data in flash - and not transfer the whole file.
However, when doing so on my device, I get a LOW_MEMORY error. Am I wrong with my assumption?
If so, it would be nice to get an extended read operation with the following signature
in order to read just parts of a file (similar to Storage.write)
Using
StorageFile
instead is not an option as such files do not support binary data (because 0xFF is used internally)Beta Was this translation helpful? Give feedback.
All reactions