Trying to stringify an object to JSON, but not enough memory to fit the whole thing in memory #1274
Replies: 15 comments
-
Posted at 2018-12-30 by @allObjects @drazzy, I do not know how your HISTORY is organized. I assume it is time series... so I suggest to chunk by time slices and you manage those with minimal information by keeping two (2) time values: - not transmitted yet, in transmission (up to including this time), successfully transmitted (up to including this time). Also, it seems that you do not need all this pseudo-precision: limit to the number of digits you need at capturing time and stick these into the history. If this is not enough yet, apply some scaling - may be you get away with a constant for each of the types of measurement - and keep the actual value and integer. Those values you push into the history and then you take it from there. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2018-12-30 by DrAzzy I don't understand what you're suggesting? HISTORY object contains 8 typed 48 element arrays (representing data every 30 minutes), 4 Float32Arrays and 4 Uint16Arrays. The time data is not retained - I push onto one end and drop the oldest value every time I update it (by looping over the arrays since I can't do push() with a typed array). This works. The problem is that I need to stringify the whole thing and send it as JSON to the webpage that will display it as a nice neat graph. However, I get out of memory when I do this - in the above, I've commented out all except RH, Temp, and Clear - which works, but then I don't get to have all the data I want. It looks like the drain() method is made just for this kind of thing - but I don't know how to use it and there are no examples :-( |
Beta Was this translation helpful? Give feedback.
-
Posted at 2018-12-31 by @allObjects Ic. Sorry not being more understanding of the data structure as conveyed by the shared code. Based on above details I would make an initial request that just delivers the frame and code for subsequent - serialized - xhr requests for each of the 8 time series. The initial request delivers the list with the detail - type of the time series. The xhrs have to be serialized: after first returns data, data is put into a received history object, second is fired, and so fort. After last, you move on with with complete composed received History object to render it graphically, as you already do now. Before you though go thru this hassle, I'm not sure if there is not a streaming way... which you hint with the .drain() - because I recall that images can be sent, and they are a lot of data... Since everything is JS and thus single threaded - at least logically - you could enforce the writes to the result to be broken up by each of the history types and also throttle them the following way:
Currently, throttling is done by constant time... but it could be variable or even driven by available memory. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2018-12-31 by DrAzzy HUZZAH!!!!! Thank you!!
|
Beta Was this translation helpful? Give feedback.
-
Posted at 2018-12-31 by @allObjects @drazzy, URVVVW! ...I guess your uppercase allows me to use it too without being too loud... ;-) |
Beta Was this translation helpful? Give feedback.
-
Posted at 2018-12-31 by @allObjects Btw, if I would be more experienced in the use await, async function and Promise - the code could be a bit more in-line... The solution I sketched was the crowbar way and way before Promise existed. At the Chrome Dev Summit 2018 in SF mid Nov I noticed the At one time, the project used a Pixle in an enclosed, pressurized container controlling the compressor over BLE to drive the trumpet via latex lips... see youtube https://www.youtube.com/watch?time_continue=1&v=9hOYcU34KMs taken from We tried to build a Robot that plays the Trumpet and Happily Failed. But unfortunately, it was not reliable enough, and the solution as described in the link was used: focus a speaker to the mouth piece and use the tropet w/ correct fingering as a resonating transmission media/tube...) I do not know about the implementation of |
Beta Was this translation helpful? Give feedback.
-
Posted at 2019-01-06 by urish @allObjects there has been some discussion regarding Right now, you can use a build-step and a transpiler such as rollup to achieve the same. It also has some other advantages that help reduce bundled code size, I used it in my In-Real-Life Chrome T-Rex Project. You can find an example setup someone created here |
Beta Was this translation helpful? Give feedback.
-
Posted at 2019-01-07 by @gfwilliams There's actually an example on using
It should be pretty easy to tweak your example to use that rather than setTimeout @drazzy - also, if you wanted you could use Unfortunately async/await isn't in Espruino yet - its efficient implementation is particularly painful in Espruino. However honestly using Promises wouldn't be that much more typing and it's probably a lot more obvious what's happening. The current solution works fine without them though. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2019-01-10 by DrAzzy Thanks - adjusted code to use drain; will see how it works when I get home tomorrow. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2019-01-12 by DrAzzy
Works like a charm! I was thinking about the object.keys() approach - but I don't want to bite myself in the ass if I later add something to HISTORY object, since my code (as you can see) makes assumptions about which index number corresponds to which kind of data, since I have to do float arrays differently from integer arrays. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2019-01-12 by @allObjects ...line 49 confuses me... |
Beta Was this translation helpful? Give feedback.
-
Posted at 2019-01-14 by @gfwilliams
|
Beta Was this translation helpful? Give feedback.
-
Posted at 2019-01-14 by @allObjects ...I guess it is a left over, because it is sent after this draining loop completed and end has already been issued. I read this out of walking through the code: line 47 writes the closing and end after all sub objects have been written. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2019-01-14 by dave_irvine No, it is sent before the draining loop fires even though the code comes afterwards. The 'drain' event will only fire once the socket 'drains', which will happen once the |
Beta Was this translation helpful? Give feedback.
-
Posted at 2019-01-15 by @allObjects ...dddddooohhh... of course, it is the opening curly braces for the overall JSON object. Thanks, @dave_irvine to point this out... and it probably has to happen there, because if written before setting up the drain event handler, the drain even would already have passed... :/ |
Beta Was this translation helpful? Give feedback.
-
Posted at 2018-12-29 by DrAzzy
How do I do this?
I am currently doing this:
It doesn't seem any better than just doing res.write(JSON.stringify(HISTORY));
either way, it runs out of memory halfway through (the main reason being that some of the arrays being stringified are floats, so they get printed with a billion decimal places). I see that there's the .drain callback, but I can't find any example about how to use this. How do I keep track of where in the process I am so I know what to stringify next?
Thanks!
Beta Was this translation helpful? Give feedback.
All reactions