Memory leak in `JSON.parse` for "long" numbers

[raphael-bmec-co]

ISSUE

When calling JSON.parse on a JSON string containing a "large" number of characters, there is a memory leak for some cases. I have been unable to identify the exact cases as it seems to depend on the decimal position, sign and length of the number.

STEPS TO REPRODUCE

Running this sample code here or on a physical device:

#include "JSONVar.h"
#include <JSON.h>
#include <string>

std::vector<std::string> jsonStrings =  {
  "{\"18dp\": 1234567890.0987654}",
  "{\"18dn\": -1234567890.09876543}",
  "{\"18dp\": 12345678900987654.3}",
  "{\"18dn\": -12345678900987654.3}",
  "{\"18dp\": 0.123456789009876543}",
  "{\"18dn\": -0.123456789009876543}",
  "{\"18p\": 123456789009876543}",
  "{\"18n\": -123456789009876543}",
  "{\"17dp\": 1234567890.0987654}",
  "{\"17dn\": -1234567890.0987654}",
  "{\"17dp\": 1234567890098765.4}",
  "{\"17dn\": -1234567890098765.4}",
  "{\"17dp\": 0.12345678900987654}",
  "{\"17dn\": -0.12345678900987654}",
  "{\"17p\": 12345678900987654}",
  "{\"17n\": -12345678900987654}",
  "{\"16dp\": 1234567890.098765}",
  "{\"16dn\": -1234567890.098765}",
  "{\"16dp\": 123456789009876.5}",
  "{\"16dn\": -123456789009876.5}",
  "{\"16dp\": 0.1234567890098765}",
  "{\"16dn\": -0.1234567890098765}",
  "{\"16p\": 1234567890098765}",
  "{\"16n\": -1234567890098765}",
  "{\"15dp\": 1234567890.09876}",
  "{\"15dn\": -1234567890.09876}",
  "{\"15dp\": 12345678900987.6}",
  "{\"15dn\": -12345678900987.6}",
  "{\"15dp\": 0.123456789009876}",
  "{\"15dn\": -0.123456789009876}",
  "{\"15p\": 123456789009876}",
  "{\"15n\": -123456789009876}",
};

void dummyScope(int index) {
  auto jsonVar = JSON.parse(jsonStrings[index].c_str());
}

void setup() {
  Serial.begin(115200);

  for (int i = 0; i < jsonStrings.size(); i++) {

    uint32_t startFreeHeap = ESP.getFreeHeap();

    dummyScope(i);

    uint32_t endFreeHeap = ESP.getFreeHeap();

    Serial.print("startFreeHeap: " + String(startFreeHeap));

    Serial.print(" | endFreeHeap: " + String(endFreeHeap) + " | ");

    Serial.print("Result: " + (startFreeHeap == endFreeHeap ? String("PASS") : String("FAIL")) + " | ");

    Serial.println(String(jsonStrings[i].c_str()));

  }
}

void loop() {
  delay(10);
}

Gives the output:

startFreeHeap: 347644 | endFreeHeap: 347044 | Result: FAIL | {"18dp": 1234567890.0987654}
startFreeHeap: 347044 | endFreeHeap: 347000 | Result: FAIL | {"18dn": -1234567890.09876543}
startFreeHeap: 351980 | endFreeHeap: 351980 | Result: PASS | {"18dp": 12345678900987654.3}
startFreeHeap: 351980 | endFreeHeap: 351980 | Result: PASS | {"18dn": -12345678900987654.3}
startFreeHeap: 351980 | endFreeHeap: 351928 | Result: FAIL | {"18dp": 0.123456789009876543}
startFreeHeap: 351928 | endFreeHeap: 351928 | Result: PASS | {"18dn": -0.123456789009876543}
startFreeHeap: 351928 | endFreeHeap: 351928 | Result: PASS | {"18p": 123456789009876543}
startFreeHeap: 351928 | endFreeHeap: 351928 | Result: PASS | {"18n": -123456789009876543}
startFreeHeap: 351928 | endFreeHeap: 351884 | Result: FAIL | {"17dp": 1234567890.0987654}
startFreeHeap: 351884 | endFreeHeap: 351884 | Result: PASS | {"17dn": -1234567890.0987654}
startFreeHeap: 351884 | endFreeHeap: 351884 | Result: PASS | {"17dp": 1234567890098765.4}
startFreeHeap: 351884 | endFreeHeap: 351884 | Result: PASS | {"17dn": -1234567890098765.4}
startFreeHeap: 351884 | endFreeHeap: 351884 | Result: PASS | {"17dp": 0.12345678900987654}
startFreeHeap: 351884 | endFreeHeap: 351884 | Result: PASS | {"17dn": -0.12345678900987654}
startFreeHeap: 351884 | endFreeHeap: 351884 | Result: PASS | {"17p": 12345678900987654}
startFreeHeap: 351884 | endFreeHeap: 351884 | Result: PASS | {"17n": -12345678900987654}
startFreeHeap: 351884 | endFreeHeap: 351884 | Result: PASS | {"16dp": 1234567890.098765}
startFreeHeap: 351884 | endFreeHeap: 351884 | Result: PASS | {"16dn": -1234567890.098765}
startFreeHeap: 351884 | endFreeHeap: 351844 | Result: FAIL | {"16dp": 123456789009876.5}
startFreeHeap: 351844 | endFreeHeap: 351844 | Result: PASS | {"16dn": -123456789009876.5}
startFreeHeap: 351844 | endFreeHeap: 351844 | Result: PASS | {"16dp": 0.1234567890098765}
startFreeHeap: 351844 | endFreeHeap: 351844 | Result: PASS | {"16dn": -0.1234567890098765}
startFreeHeap: 351844 | endFreeHeap: 351844 | Result: PASS | {"16p": 1234567890098765}
startFreeHeap: 351844 | endFreeHeap: 351844 | Result: PASS | {"16n": -1234567890098765}
startFreeHeap: 351844 | endFreeHeap: 351844 | Result: PASS | {"15dp": 1234567890.09876}
startFreeHeap: 351844 | endFreeHeap: 351844 | Result: PASS | {"15dn": -1234567890.09876}
startFreeHeap: 351844 | endFreeHeap: 351844 | Result: PASS | {"15dp": 12345678900987.6}
startFreeHeap: 351844 | endFreeHeap: 351844 | Result: PASS | {"15dn": -12345678900987.6}
startFreeHeap: 351844 | endFreeHeap: 351844 | Result: PASS | {"15dp": 0.123456789009876}
startFreeHeap: 351844 | endFreeHeap: 351844 | Result: PASS | {"15dn": -0.123456789009876}
startFreeHeap: 351844 | endFreeHeap: 351844 | Result: PASS | {"15p": 123456789009876}
startFreeHeap: 351844 | endFreeHeap: 351844 | Result: PASS | {"15n": -123456789009876}

QUESTIONS

Is there a known limit on the number length, precision etc?

[raphael-bmec-co]

Seems related to #5 and #12. cJSON claims support to 64 characters, so I don't think the issue is there.

[raphael-bmec-co]

Update

The issue seems related to the underlying cJSON library.

This test code still give the issues:

#include "JSONVar.h"
#include <JSON.h>
#include <cJSON.h>
#include <string>

std::vector<std::string> jsonStrings =  {
  "{\"18dp\": 1234567890.0987654}",
  "{\"18dn\": -1234567890.09876543}",
  "{\"18dp\": 12345678900987654.3}",
  "{\"18dn\": -12345678900987654.3}",
  "{\"18dp\": 0.123456789009876543}",
  "{\"18dn\": -0.123456789009876543}",
  "{\"18p\": 123456789009876543}",
  "{\"18n\": -123456789009876543}",
  "{\"17dp\": 1234567890.0987654}",
  "{\"17dn\": -1234567890.0987654}",
  "{\"17dp\": 1234567890098765.4}",
  "{\"17dn\": -1234567890098765.4}",
  "{\"17dp\": 0.12345678900987654}",
  "{\"17dn\": -0.12345678900987654}",
  "{\"17p\": 12345678900987654}",
  "{\"17n\": -12345678900987654}",
  "{\"16dp\": 1234567890.098765}",
  "{\"16dn\": -1234567890.098765}",
  "{\"16dp\": 123456789009876.5}",
  "{\"16dn\": -123456789009876.5}",
  "{\"16dp\": 0.1234567890098765}",
  "{\"16dn\": -0.1234567890098765}",
  "{\"16p\": 1234567890098765}",
  "{\"16n\": -1234567890098765}",
  "{\"15dp\": 1234567890.09876}",
  "{\"15dn\": -1234567890.09876}",
  "{\"15dp\": 12345678900987.6}",
  "{\"15dn\": -12345678900987.6}",
  "{\"15dp\": 0.123456789009876}",
  "{\"15dn\": -0.123456789009876}",
  "{\"15p\": 123456789009876}",
  "{\"15n\": -123456789009876}",
};

void dummyScope(int index) {
  // auto jsonVar = JSON.parse(jsonStrings[index].c_str());

  cJSON* json = cJSON_Parse(jsonStrings[index].c_str());

  cJSON_Delete(json);

}

void setup() {
  Serial.begin(115200);

  for (int i = 0; i < jsonStrings.size(); i++) {

    uint32_t startFreeHeap = ESP.getFreeHeap();

    dummyScope(i);

    uint32_t endFreeHeap = ESP.getFreeHeap();

    Serial.print("startFreeHeap: " + String(startFreeHeap));

    Serial.print(" | endFreeHeap: " + String(endFreeHeap) + " | ");

    Serial.print("Result: " + (startFreeHeap == endFreeHeap ? String("PASS") : String("FAIL")) + " | ");

    Serial.println(String(jsonStrings[i].c_str()));

  }
}

void loop() {
  delay(10);
}

startFreeHeap: 347644 | endFreeHeap: 347044 | Result: FAIL | {"18dp": 1234567890.0987654}
startFreeHeap: 347044 | endFreeHeap: 347000 | Result: FAIL | {"18dn": -1234567890.09876543}
startFreeHeap: 351980 | endFreeHeap: 351980 | Result: PASS | {"18dp": 12345678900987654.3}
startFreeHeap: 351980 | endFreeHeap: 351980 | Result: PASS | {"18dn": -12345678900987654.3}
startFreeHeap: 351980 | endFreeHeap: 351928 | Result: FAIL | {"18dp": 0.123456789009876543}
startFreeHeap: 351928 | endFreeHeap: 351928 | Result: PASS | {"18dn": -0.123456789009876543}
startFreeHeap: 351928 | endFreeHeap: 351928 | Result: PASS | {"18p": 123456789009876543}
startFreeHeap: 351928 | endFreeHeap: 351928 | Result: PASS | {"18n": -123456789009876543}
startFreeHeap: 351928 | endFreeHeap: 351884 | Result: FAIL | {"17dp": 1234567890.0987654}
startFreeHeap: 351884 | endFreeHeap: 351884 | Result: PASS | {"17dn": -1234567890.0987654}
startFreeHeap: 351884 | endFreeHeap: 351884 | Result: PASS | {"17dp": 1234567890098765.4}
startFreeHeap: 351884 | endFreeHeap: 351884 | Result: PASS | {"17dn": -1234567890098765.4}
startFreeHeap: 351884 | endFreeHeap: 351884 | Result: PASS | {"17dp": 0.12345678900987654}
startFreeHeap: 351884 | endFreeHeap: 351884 | Result: PASS | {"17dn": -0.12345678900987654}
startFreeHeap: 351884 | endFreeHeap: 351884 | Result: PASS | {"17p": 12345678900987654}
startFreeHeap: 351884 | endFreeHeap: 351884 | Result: PASS | {"17n": -12345678900987654}
startFreeHeap: 351884 | endFreeHeap: 351884 | Result: PASS | {"16dp": 1234567890.098765}
startFreeHeap: 351884 | endFreeHeap: 351884 | Result: PASS | {"16dn": -1234567890.098765}
startFreeHeap: 351884 | endFreeHeap: 351844 | Result: FAIL | {"16dp": 123456789009876.5}
startFreeHeap: 351844 | endFreeHeap: 351844 | Result: PASS | {"16dn": -123456789009876.5}
startFreeHeap: 351844 | endFreeHeap: 351844 | Result: PASS | {"16dp": 0.1234567890098765}
startFreeHeap: 351844 | endFreeHeap: 351844 | Result: PASS | {"16dn": -0.1234567890098765}
startFreeHeap: 351844 | endFreeHeap: 351844 | Result: PASS | {"16p": 1234567890098765}
startFreeHeap: 351844 | endFreeHeap: 351844 | Result: PASS | {"16n": -1234567890098765}
startFreeHeap: 351844 | endFreeHeap: 351844 | Result: PASS | {"15dp": 1234567890.09876}
startFreeHeap: 351844 | endFreeHeap: 351844 | Result: PASS | {"15dn": -1234567890.09876}
startFreeHeap: 351844 | endFreeHeap: 351844 | Result: PASS | {"15dp": 12345678900987.6}
startFreeHeap: 351844 | endFreeHeap: 351844 | Result: PASS | {"15dn": -12345678900987.6}
startFreeHeap: 351844 | endFreeHeap: 351844 | Result: PASS | {"15dp": 0.123456789009876}
startFreeHeap: 351844 | endFreeHeap: 351844 | Result: PASS | {"15dn": -0.123456789009876}
startFreeHeap: 351844 | endFreeHeap: 351844 | Result: PASS | {"15p": 123456789009876}
startFreeHeap: 351844 | endFreeHeap: 351844 | Result: PASS | {"15n": -123456789009876}

[raphael-bmec-co]

The leak seems to stem all the way from strtod in the cJSON parse_number. So it is not specifically related to this library.

[raphael-bmec-co]

Okay I've been chasing ghosts here 🙈. I think there is a static allocation of memory in strtod on first call that was leading me astray. I am closing for now.