Large json

[espruino-discuss]

Posted at 2016-01-29 by cwilt

I am trying to download a json string from a weather service. Currently using the http client example and it always stops at the same spot. Seems that the json is larger than a single variable can hold. Looking for suggestions.

[espruino-discuss]

Posted at 2016-01-29 by @gfwilliams

It stops with an error?

If it's too big for memory, the best approach would be to just search for the tokens that you want. You'd probably want to check both the current and last bit of data together, in case the information you want spans multiple ones:

var last = "";
res.on('data',function(data) {
  var s = last+data;
  if (s.indexOf(...)) ...;
  last = data;
});

Potentially if you just want some of the data, you could bracket-count until you get to the right part?

---

Posted at 2016-01-29 by cwilt

No error, it just stops.

require("http").get("http://forecast.weather.gov/MapClick.php?lat=39.70&lon=-104.80&FcstType=json", function(res) {
  var contents = "";
  res.on('data', function(data) { contents += data; });
  res.on('close', function() { console.log(contents); });
});

After it stops I run process.memory() and it returns...
{ "free": 1349, "usage": 51, "total": 1400, "history": 28 }

Am I reading that correctly and its not used much memory?

---

Posted at 2016-01-29 by @gfwilliams

It definitely seems like it's not using much - however it could have used up loads of memory while loading the webpage, failed, and then freed it - but if it had a problem allocating memory it normally would have said something.

Just checked and that page does return 4845 bytes, which should fit into the available memory you have just fine.

Maybe check that on('data',... is actually getting called? It might be it can't make a connection, or I guess it's possible the connection is never getting closed?

... but if it's never getting closed then you'd expect more memory to be used.

---

Posted at 2016-01-29 by DrAzzy

Is it running out of free heap space? I remember @tve mentioning that the limited heap space for receiving data on the ESP8266 can run out, not just JSVars.