Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[TW#18645] NVS Blob and String operations leak memory #1642

Closed
tommybazar opened this issue Feb 19, 2018 · 12 comments
Closed

[TW#18645] NVS Blob and String operations leak memory #1642

tommybazar opened this issue Feb 19, 2018 · 12 comments
Assignees
@igrr

Comments

@tommybazar
Copy link

After spending some time looking for a slow, but persistent memory leakage problem in our code, I found that nvs_set_blob allocates some overhead memory that is not freed on nvs_erase_key, nor on nvs_erase_all nor on handle close. Other nvs_setXXX functions also allocate this memory, but erasing the given key does free it.

nvs_set_str is does something similar, but it does free about half of the memory it allocates...

Below is a small program demonstrating this.

We're using release v3.0 from January 16th (commit e7dc749). None of the commits from last month seem to have touched the NVS.

#include <iostream>
#include <freertos/FreeRTOS.h>
#include <freertos/task.h>
#include <nvs.h>
#include <sstream>
#include "esp_system.h"
#include <esp_log.h>
#include <nvs_flash.h>

#define MAIN_TAG "main"

std::string makeFileName(uint32_t index) {
  std::ostringstream outStr;
  outStr << "testname_" << index;
  return outStr.str();
}

extern "C" void app_main() {

  uint8_t buffer[255] = {0xFF};
  uint8_t buffer2[255] = {0x00};

  nvs_flash_init();

  esp_err_t error = 0;
  nvs_handle nvsHandle;
  nvs_open("test2", NVS_READWRITE, &nvsHandle);

  // Set blobs
  for (int i = 0; i < 100; i++) {
    error = nvs_set_blob(nvsHandle, makeFileName(i).c_str(), buffer, 255);
    nvs_commit(nvsHandle);
    ESP_LOGI(MAIN_TAG, "set blob @%d, error = %d, heap = %d", i, error, esp_get_free_heap_size());
    usleep(10);
  }

  // Get blobs to prove they're really there
  for (int i = 0; i < 100; i++) {
    uint32_t length = 256;
    error = nvs_get_blob(nvsHandle, makeFileName(i).c_str(), buffer2, &length);
    nvs_commit(nvsHandle);
    ESP_LOGI(MAIN_TAG, "get blob @%d, error = %d, heap = %d", i, error, esp_get_free_heap_size());
    usleep(10);
  }

  // Erase blobs to see them not giving back memory
  for (int i = 0; i < 100; i++) {
    error = nvs_erase_key(nvsHandle, makeFileName(i).c_str());
    nvs_commit(nvsHandle);
    ESP_LOGI(MAIN_TAG, "removed blob @%d, error = %d, heap = %d", i, error, esp_get_free_heap_size());
    usleep(10);
  }

  // Get blobs with error to prove they're really NOT there
  for (int i = 0; i < 100; i++) {
    uint32_t length = 0;
    error = nvs_get_blob(nvsHandle, makeFileName(i).c_str(), buffer2, &length);
    nvs_commit(nvsHandle);
    ESP_LOGI(MAIN_TAG, "get blob @%d, error = %d, heap = %d", i, error, esp_get_free_heap_size());
    usleep(10);
  }

  nvs_erase_all(nvsHandle);
  ESP_LOGI(MAIN_TAG, "2 Heap after delete all %d", esp_get_free_heap_size());
  nvs_close(nvsHandle);
  ESP_LOGI(MAIN_TAG, "2 Heap after close %d", esp_get_free_heap_size());
  nvs_open("test2", NVS_READWRITE, &nvsHandle);
  ESP_LOGI(MAIN_TAG, "2 Heap after open %d", esp_get_free_heap_size());

  // Add ints to see them consume heap.
  for (int i = 0; i < 100; i++) {
    error = nvs_set_i32(nvsHandle, makeFileName(i).c_str(), 1);
    nvs_commit(nvsHandle);
    ESP_LOGI(MAIN_TAG, "set int @%d, error = %d, heap = %d", i, error, esp_get_free_heap_size());
    usleep(10);
  }

  // Remove them to see them give it back.
  for (int i = 0; i < 100; i++) {
    error = nvs_erase_key(nvsHandle, makeFileName(i).c_str());
    nvs_commit(nvsHandle);
    ESP_LOGI(MAIN_TAG, "removed int @%d, error = %d, heap = %d", i, error, esp_get_free_heap_size());
    usleep(10);
  }

  nvs_close(nvsHandle);
  usleep(10000000);
}

Cheers,
Tomas.
@negativekelvin
Copy link
Contributor

Can you post the resulting output of your ESP_LOGI calls for reference

@agmuntianu
Copy link

agmuntianu commented Feb 20, 2018 •
edited
Loading

Because I am also hunding a memory leak somewhere, I ran @tommybazar code. My IDF is 8856cc0 , this is the output:

Long Logs 
I (533) main: set blob @0, error = 0, heap = 294476
I (533) main: set blob @1, error = 0, heap = 294308
I (533) main: set blob @2, error = 0, heap = 294308
I (543) main: set blob @3, error = 0, heap = 294308
I (543) main: set blob @4, error = 0, heap = 294308
I (553) main: set blob @5, error = 0, heap = 294308
I (553) main: set blob @6, error = 0, heap = 294308
I (563) main: set blob @7, error = 0, heap = 294308
I (563) main: set blob @8, error = 0, heap = 294308
I (573) main: set blob @9, error = 0, heap = 294308
I (583) main: set blob @10, error = 0, heap = 294308
I (583) main: set blob @11, error = 0, heap = 294308
I (593) main: set blob @12, error = 0, heap = 294308
I (593) main: set blob @13, error = 0, heap = 294176
I (603) main: set blob @14, error = 0, heap = 294176
I (603) main: set blob @15, error = 0, heap = 294176
I (613) main: set blob @16, error = 0, heap = 294176
I (623) main: set blob @17, error = 0, heap = 294176
I (623) main: set blob @18, error = 0, heap = 294176
I (633) main: set blob @19, error = 0, heap = 294176
I (633) main: set blob @20, error = 0, heap = 294176
I (643) main: set blob @21, error = 0, heap = 294176
I (643) main: set blob @22, error = 0, heap = 294176
I (653) main: set blob @23, error = 0, heap = 294176
I (653) main: set blob @24, error = 0, heap = 294176
I (663) main: set blob @25, error = 0, heap = 294176
I (673) main: set blob @26, error = 0, heap = 294176
I (673) main: set blob @27, error = 0, heap = 294044
I (683) main: set blob @28, error = 0, heap = 294044
I (683) main: set blob @29, error = 0, heap = 294044
I (693) main: set blob @30, error = 0, heap = 294044
I (693) main: set blob @31, error = 0, heap = 294044
I (703) main: set blob @32, error = 0, heap = 294044
I (713) main: set blob @33, error = 0, heap = 294044
I (713) main: set blob @34, error = 0, heap = 294044
I (723) main: set blob @35, error = 0, heap = 294044
I (723) main: set blob @36, error = 0, heap = 294044
I (733) main: set blob @37, error = 0, heap = 294044
I (733) main: set blob @38, error = 0, heap = 294044
I (743) main: set blob @39, error = 0, heap = 294044
I (743) main: set blob @40, error = 0, heap = 294044
I (753) main: set blob @41, error = 0, heap = 293912
I (763) main: set blob @42, error = 0, heap = 293912
I (763) main: set blob @43, error = 0, heap = 293912
I (773) main: set blob @44, error = 0, heap = 293912
I (773) main: set blob @45, error = 0, heap = 293912
I (783) main: set blob @46, error = 0, heap = 293912
I (783) main: set blob @47, error = 0, heap = 293912
I (793) main: set blob @48, error = 0, heap = 293912
I (803) main: set blob @49, error = 0, heap = 293912
I (803) main: set blob @50, error = 0, heap = 293912
I (813) main: set blob @51, error = 0, heap = 293912
I (813) main: set blob @52, error = 0, heap = 293912
I (823) main: set blob @53, error = 0, heap = 293912
I (823) main: set blob @54, error = 0, heap = 293912
I (833) main: set blob @55, error = 0, heap = 293780
I (843) main: set blob @56, error = 0, heap = 293780
I (843) main: set blob @57, error = 0, heap = 293780
I (853) main: set blob @58, error = 0, heap = 293780
I (853) main: set blob @59, error = 0, heap = 293780
I (863) main: set blob @60, error = 0, heap = 293780
I (863) main: set blob @61, error = 0, heap = 293780
I (873) main: set blob @62, error = 0, heap = 293780
I (873) main: set blob @63, error = 0, heap = 293780
I (883) main: set blob @64, error = 0, heap = 293780
I (893) main: set blob @65, error = 0, heap = 293780
I (893) main: set blob @66, error = 0, heap = 293780
I (903) main: set blob @67, error = 0, heap = 293780
I (903) main: set blob @68, error = 0, heap = 293780
I (913) main: set blob @69, error = 4357, heap = 293780
I (913) main: set blob @70, error = 4357, heap = 293780
I (923) main: set blob @71, error = 4357, heap = 293780
I (923) main: set blob @72, error = 4357, heap = 293780
I (933) main: set blob @73, error = 4357, heap = 293780
I (943) main: set blob @74, error = 4357, heap = 293780
I (943) main: set blob @75, error = 4357, heap = 293780
I (953) main: set blob @76, error = 4357, heap = 293780
I (953) main: set blob @77, error = 4357, heap = 293780
I (963) main: set blob @78, error = 4357, heap = 293780
I (973) main: set blob @79, error = 4357, heap = 293780
I (973) main: set blob @80, error = 4357, heap = 293780
I (983) main: set blob @81, error = 4357, heap = 293780
I (983) main: set blob @82, error = 4357, heap = 293780
I (993) main: set blob @83, error = 4357, heap = 293780
I (1003) main: set blob @84, error = 4357, heap = 293780
I (1003) main: set blob @85, error = 4357, heap = 293780
I (1013) main: set blob @86, error = 4357, heap = 293780
I (1013) main: set blob @87, error = 4357, heap = 293780
I (1023) main: set blob @88, error = 4357, heap = 293780
I (1023) main: set blob @89, error = 4357, heap = 293780
I (1033) main: set blob @90, error = 4357, heap = 293780
I (1043) main: set blob @91, error = 4357, heap = 293780
I (1043) main: set blob @92, error = 4357, heap = 293780
I (1053) main: set blob @93, error = 4357, heap = 293780
I (1053) main: set blob @94, error = 4357, heap = 293780
I (1063) main: set blob @95, error = 4357, heap = 293780
I (1073) main: set blob @96, error = 4357, heap = 293780
I (1073) main: set blob @97, error = 4357, heap = 293780
I (1083) main: set blob @98, error = 4357, heap = 293780
I (1083) main: set blob @99, error = 4357, heap = 293780
I (1093) main: get blob @0, error = 0, heap = 293780
I (1103) main: get blob @1, error = 0, heap = 293780
I (1103) main: get blob @2, error = 0, heap = 293780
I (1113) main: get blob @3, error = 0, heap = 293780
I (1113) main: get blob @4, error = 0, heap = 293780
I (1123) main: get blob @5, error = 0, heap = 293780
I (1133) main: get blob @6, error = 0, heap = 293780
I (1133) main: get blob @7, error = 0, heap = 293780
I (1143) main: get blob @8, error = 0, heap = 293780
I (1143) main: get blob @9, error = 0, heap = 293780
I (1153) main: get blob @10, error = 0, heap = 293780
I (1153) main: get blob @11, error = 0, heap = 293780
I (1163) main: get blob @12, error = 0, heap = 293780
I (1173) main: get blob @13, error = 0, heap = 293780
I (1173) main: get blob @14, error = 0, heap = 293780
I (1183) main: get blob @15, error = 0, heap = 293780
I (1183) main: get blob @16, error = 0, heap = 293780
I (1193) main: get blob @17, error = 0, heap = 293780
I (1193) main: get blob @18, error = 0, heap = 293780
I (1203) main: get blob @19, error = 0, heap = 293780
I (1213) main: get blob @20, error = 0, heap = 293780
I (1213) main: get blob @21, error = 0, heap = 293780
I (1223) main: get blob @22, error = 0, heap = 293780
I (1223) main: get blob @23, error = 0, heap = 293780
I (1233) main: get blob @24, error = 0, heap = 293780
I (1233) main: get blob @25, error = 0, heap = 293780
I (1243) main: get blob @26, error = 0, heap = 293780
I (1253) main: get blob @27, error = 0, heap = 293780
I (1253) main: get blob @28, error = 0, heap = 293780
I (1263) main: get blob @29, error = 0, heap = 293780
I (1263) main: get blob @30, error = 0, heap = 293780
I (1273) main: get blob @31, error = 0, heap = 293780
I (1273) main: get blob @32, error = 0, heap = 293780
I (1283) main: get blob @33, error = 0, heap = 293780
I (1293) main: get blob @34, error = 0, heap = 293780
I (1293) main: get blob @35, error = 0, heap = 293780
I (1303) main: get blob @36, error = 0, heap = 293780
I (1303) main: get blob @37, error = 0, heap = 293780
I (1313) main: get blob @38, error = 0, heap = 293780
I (1313) main: get blob @39, error = 0, heap = 293780
I (1323) main: get blob @40, error = 0, heap = 293780
I (1333) main: get blob @41, error = 0, heap = 293780
I (1333) main: get blob @42, error = 0, heap = 293780
I (1343) main: get blob @43, error = 0, heap = 293780
I (1343) main: get blob @44, error = 0, heap = 293780
I (1353) main: get blob @45, error = 0, heap = 293780
I (1363) main: get blob @46, error = 0, heap = 293780
I (1363) main: get blob @47, error = 0, heap = 293780
I (1373) main: get blob @48, error = 0, heap = 293780
I (1373) main: get blob @49, error = 0, heap = 293780
I (1383) main: get blob @50, error = 0, heap = 293780
I (1383) main: get blob @51, error = 0, heap = 293780
I (1393) main: get blob @52, error = 0, heap = 293780
I (1403) main: get blob @53, error = 0, heap = 293780
I (1403) main: get blob @54, error = 0, heap = 293780
I (1413) main: get blob @55, error = 0, heap = 293780
I (1413) main: get blob @56, error = 0, heap = 293780
I (1423) main: get blob @57, error = 0, heap = 293780
I (1423) main: get blob @58, error = 0, heap = 293780
I (1433) main: get blob @59, error = 0, heap = 293780
I (1443) main: get blob @60, error = 0, heap = 293780
I (1443) main: get blob @61, error = 0, heap = 293780
I (1453) main: get blob @62, error = 0, heap = 294908
I (1453) main: get blob @63, error = 0, heap = 294908
I (1463) main: get blob @64, error = 0, heap = 294908
I (1463) main: get blob @65, error = 0, heap = 294908
I (1473) main: get blob @66, error = 0, heap = 294908
I (1483) main: get blob @67, error = 0, heap = 294908
I (1483) main: get blob @68, error = 0, heap = 294908
I (1493) main: get blob @69, error = 4354, heap = 294908
I (1493) main: get blob @70, error = 4354, heap = 294908
I (1503) main: get blob @71, error = 4354, heap = 294908
I (1503) main: get blob @72, error = 4354, heap = 294908
I (1513) main: get blob @73, error = 4354, heap = 294908
I (1523) main: get blob @74, error = 4354, heap = 294908
I (1523) main: get blob @75, error = 4354, heap = 294908
I (1533) main: get blob @76, error = 4354, heap = 294908
I (1533) main: get blob @77, error = 4354, heap = 294908
I (1543) main: get blob @78, error = 4354, heap = 294908
I (1553) main: get blob @79, error = 4354, heap = 294908
I (1553) main: get blob @80, error = 4354, heap = 294908
I (1563) main: get blob @81, error = 4354, heap = 294908
I (1563) main: get blob @82, error = 4354, heap = 294908
I (1573) main: get blob @83, error = 4354, heap = 294908
I (1583) main: get blob @84, error = 4354, heap = 294908
I (1583) main: get blob @85, error = 4354, heap = 294908
I (1593) main: get blob @86, error = 4354, heap = 294908
I (1593) main: get blob @87, error = 4354, heap = 294908
I (1603) main: get blob @88, error = 4354, heap = 294908
I (1613) main: get blob @89, error = 4354, heap = 294908
I (1613) main: get blob @90, error = 4354, heap = 294908
I (1623) main: get blob @91, error = 4354, heap = 294908
I (1623) main: get blob @92, error = 4354, heap = 294908
I (1633) main: get blob @93, error = 4354, heap = 294908
I (1643) main: get blob @94, error = 4354, heap = 294908
I (1643) main: get blob @95, error = 4354, heap = 294908
I (1653) main: get blob @96, error = 4354, heap = 294908
I (1653) main: get blob @97, error = 4354, heap = 294908
I (1663) main: get blob @98, error = 4354, heap = 294908
I (1673) main: get blob @99, error = 4354, heap = 294908
I (1673) main: removed blob @0, error = 0, heap = 294908
I (1683) main: removed blob @1, error = 0, heap = 294908
I (1683) main: removed blob @2, error = 0, heap = 294908
I (1693) main: removed blob @3, error = 0, heap = 294908
I (1703) main: removed blob @4, error = 0, heap = 294908
I (1703) main: removed blob @5, error = 0, heap = 294908
I (1713) main: removed blob @6, error = 0, heap = 294908
I (1713) main: removed blob @7, error = 0, heap = 294908
I (1723) main: removed blob @8, error = 0, heap = 294908
I (1733) main: removed blob @9, error = 0, heap = 294908
I (1733) main: removed blob @10, error = 0, heap = 294908
I (1743) main: removed blob @11, error = 0, heap = 294908
I (1743) main: removed blob @12, error = 0, heap = 294908
I (1753) main: removed blob @13, error = 0, heap = 294908
I (1763) main: removed blob @14, error = 0, heap = 294908
I (1763) main: removed blob @15, error = 0, heap = 294908
I (1773) main: removed blob @16, error = 0, heap = 294908
I (1773) main: removed blob @17, error = 0, heap = 294908
I (1783) main: removed blob @18, error = 0, heap = 294908
I (1793) main: removed blob @19, error = 0, heap = 294908
I (1793) main: removed blob @20, error = 0, heap = 294908
I (1803) main: removed blob @21, error = 0, heap = 294908
I (1803) main: removed blob @22, error = 0, heap = 294908
I (1813) main: removed blob @23, error = 0, heap = 294908
I (1823) main: removed blob @24, error = 0, heap = 294908
I (1823) main: removed blob @25, error = 0, heap = 294908
I (1833) main: removed blob @26, error = 0, heap = 294908
I (1833) main: removed blob @27, error = 0, heap = 294908
I (1843) main: removed blob @28, error = 0, heap = 294908
I (1853) main: removed blob @29, error = 0, heap = 294908
I (1853) main: removed blob @30, error = 0, heap = 294908
I (1863) main: removed blob @31, error = 0, heap = 294908
I (1873) main: removed blob @32, error = 0, heap = 294908
I (1873) main: removed blob @33, error = 0, heap = 294908
I (1883) main: removed blob @34, error = 0, heap = 294908
I (1883) main: removed blob @35, error = 0, heap = 294908
I (1893) main: removed blob @36, error = 0, heap = 294908
I (1903) main: removed blob @37, error = 0, heap = 294908
I (1903) main: removed blob @38, error = 0, heap = 294908
I (1913) main: removed blob @39, error = 0, heap = 294908
I (1913) main: removed blob @40, error = 0, heap = 294908
I (1923) main: removed blob @41, error = 0, heap = 294908
I (1933) main: removed blob @42, error = 0, heap = 294908
I (1933) main: removed blob @43, error = 0, heap = 294908
I (1943) main: removed blob @44, error = 0, heap = 294908
I (1943) main: removed blob @45, error = 0, heap = 294908
I (1953) main: removed blob @46, error = 0, heap = 294908
I (1963) main: removed blob @47, error = 0, heap = 294908
I (1963) main: removed blob @48, error = 0, heap = 294908
I (1973) main: removed blob @49, error = 0, heap = 294908
I (1973) main: removed blob @50, error = 0, heap = 294908
I (1983) main: removed blob @51, error = 0, heap = 294908
I (1993) main: removed blob @52, error = 0, heap = 294908
I (1993) main: removed blob @53, error = 0, heap = 294908
I (2003) main: removed blob @54, error = 0, heap = 294908
I (2013) main: removed blob @55, error = 0, heap = 294908
I (2013) main: removed blob @56, error = 0, heap = 294908
I (2023) main: removed blob @57, error = 0, heap = 294908
I (2023) main: removed blob @58, error = 0, heap = 294908
I (2033) main: removed blob @59, error = 0, heap = 294908
I (2043) main: removed blob @60, error = 0, heap = 294908
I (2043) main: removed blob @61, error = 0, heap = 294908
I (2053) main: removed blob @62, error = 0, heap = 294908
I (2053) main: removed blob @63, error = 0, heap = 294908
I (2063) main: removed blob @64, error = 0, heap = 294908
I (2073) main: removed blob @65, error = 0, heap = 294908
I (2073) main: removed blob @66, error = 0, heap = 294908
I (2083) main: removed blob @67, error = 0, heap = 294908
I (2083) main: removed blob @68, error = 0, heap = 294908
I (2093) main: removed blob @69, error = 4354, heap = 294908
I (2103) main: removed blob @70, error = 4354, heap = 294908
I (2103) main: removed blob @71, error = 4354, heap = 294908
I (2113) main: removed blob @72, error = 4354, heap = 294908
I (2113) main: removed blob @73, error = 4354, heap = 294908
I (2123) main: removed blob @74, error = 4354, heap = 294908
I (2133) main: removed blob @75, error = 4354, heap = 294908
I (2133) main: removed blob @76, error = 4354, heap = 294908
I (2143) main: removed blob @77, error = 4354, heap = 294908
I (2153) main: removed blob @78, error = 4354, heap = 294908
I (2153) main: removed blob @79, error = 4354, heap = 294908
I (2163) main: removed blob @80, error = 4354, heap = 294908
I (2173) main: removed blob @81, error = 4354, heap = 294908
I (2173) main: removed blob @82, error = 4354, heap = 294908
I (2183) main: removed blob @83, error = 4354, heap = 294908
I (2183) main: removed blob @84, error = 4354, heap = 294908
I (2193) main: removed blob @85, error = 4354, heap = 294908
I (2203) main: removed blob @86, error = 4354, heap = 294908
I (2203) main: removed blob @87, error = 4354, heap = 294908
I (2213) main: removed blob @88, error = 4354, heap = 294908
I (2223) main: removed blob @89, error = 4354, heap = 294908
I (2223) main: removed blob @90, error = 4354, heap = 294908
I (2233) main: removed blob @91, error = 4354, heap = 294908
I (2243) main: removed blob @92, error = 4354, heap = 294908
I (2243) main: removed blob @93, error = 4354, heap = 294908
I (2253) main: removed blob @94, error = 4354, heap = 294908
I (2253) main: removed blob @95, error = 4354, heap = 294908
I (2263) main: removed blob @96, error = 4354, heap = 294908
I (2273) main: removed blob @97, error = 4354, heap = 294908
I (2273) main: removed blob @98, error = 4354, heap = 294908
I (2283) main: removed blob @99, error = 4354, heap = 294908
I (2293) main: get blob @0, error = 4354, heap = 294908
I (2293) main: get blob @1, error = 4354, heap = 294908
I (2303) main: get blob @2, error = 4354, heap = 294908
I (2303) main: get blob @3, error = 4354, heap = 294908
I (2313) main: get blob @4, error = 4354, heap = 294908
I (2323) main: get blob @5, error = 4354, heap = 294908
I (2323) main: get blob @6, error = 4354, heap = 294908
I (2333) main: get blob @7, error = 4354, heap = 294908
I (2333) main: get blob @8, error = 4354, heap = 294908
I (2343) main: get blob @9, error = 4354, heap = 294908
I (2343) main: get blob @10, error = 4354, heap = 294908
I (2353) main: get blob @11, error = 4354, heap = 294908
I (2363) main: get blob @12, error = 4354, heap = 294908
I (2363) main: get blob @13, error = 4354, heap = 294908
I (2373) main: get blob @14, error = 4354, heap = 294908
I (2373) main: get blob @15, error = 4354, heap = 294908
I (2383) main: get blob @16, error = 4354, heap = 294908
I (2393) main: get blob @17, error = 4354, heap = 294908
I (2393) main: get blob @18, error = 4354, heap = 294908
I (2403) main: get blob @19, error = 4354, heap = 294908
I (2403) main: get blob @20, error = 4354, heap = 294908
I (2413) main: get blob @21, error = 4354, heap = 294908
I (2423) main: get blob @22, error = 4354, heap = 294908
I (2423) main: get blob @23, error = 4354, heap = 294908
I (2433) main: get blob @24, error = 4354, heap = 294908
I (2433) main: get blob @25, error = 4354, heap = 294908
I (2443) main: get blob @26, error = 4354, heap = 294908
I (2453) main: get blob @27, error = 4354, heap = 294908
I (2453) main: get blob @28, error = 4354, heap = 294908
I (2463) main: get blob @29, error = 4354, heap = 294908
I (2463) main: get blob @30, error = 4354, heap = 294908
I (2473) main: get blob @31, error = 4354, heap = 294908
I (2483) main: get blob @32, error = 4354, heap = 294908
I (2483) main: get blob @33, error = 4354, heap = 294908
I (2493) main: get blob @34, error = 4354, heap = 294908
I (2493) main: get blob @35, error = 4354, heap = 294908
I (2503) main: get blob @36, error = 4354, heap = 294908
I (2513) main: get blob @37, error = 4354, heap = 294908
I (2513) main: get blob @38, error = 4354, heap = 294908
I (2523) main: get blob @39, error = 4354, heap = 294908
I (2523) main: get blob @40, error = 4354, heap = 294908
I (2533) main: get blob @41, error = 4354, heap = 294908
I (2543) main: get blob @42, error = 4354, heap = 294908
I (2543) main: get blob @43, error = 4354, heap = 294908
I (2553) main: get blob @44, error = 4354, heap = 294908
I (2553) main: get blob @45, error = 4354, heap = 294908
I (2563) main: get blob @46, error = 4354, heap = 294908
I (2573) main: get blob @47, error = 4354, heap = 294908
I (2573) main: get blob @48, error = 4354, heap = 294908
I (2583) main: get blob @49, error = 4354, heap = 294908
I (2583) main: get blob @50, error = 4354, heap = 294908
I (2593) main: get blob @51, error = 4354, heap = 294908
I (2603) main: get blob @52, error = 4354, heap = 294908
I (2603) main: get blob @53, error = 4354, heap = 294908
I (2613) main: get blob @54, error = 4354, heap = 294908
I (2613) main: get blob @55, error = 4354, heap = 294908
I (2623) main: get blob @56, error = 4354, heap = 294908
I (2633) main: get blob @57, error = 4354, heap = 294908
I (2633) main: get blob @58, error = 4354, heap = 294908
I (2643) main: get blob @59, error = 4354, heap = 294908
I (2643) main: get blob @60, error = 4354, heap = 294908
I (2653) main: get blob @61, error = 4354, heap = 294908
I (2663) main: get blob @62, error = 4354, heap = 294908
I (2663) main: get blob @63, error = 4354, heap = 294908
I (2673) main: get blob @64, error = 4354, heap = 294908
I (2673) main: get blob @65, error = 4354, heap = 294908
I (2683) main: get blob @66, error = 4354, heap = 294908
I (2693) main: get blob @67, error = 4354, heap = 294908
I (2693) main: get blob @68, error = 4354, heap = 294908
I (2703) main: get blob @69, error = 4354, heap = 294908
I (2703) main: get blob @70, error = 4354, heap = 294908
I (2713) main: get blob @71, error = 4354, heap = 294908
I (2723) main: get blob @72, error = 4354, heap = 294908
I (2723) main: get blob @73, error = 4354, heap = 294908
I (2733) main: get blob @74, error = 4354, heap = 294908
I (2733) main: get blob @75, error = 4354, heap = 294908
I (2743) main: get blob @76, error = 4354, heap = 294908
I (2753) main: get blob @77, error = 4354, heap = 294908
I (2753) main: get blob @78, error = 4354, heap = 294908
I (2763) main: get blob @79, error = 4354, heap = 294908
I (2763) main: get blob @80, error = 4354, heap = 294908
I (2773) main: get blob @81, error = 4354, heap = 294908
I (2783) main: get blob @82, error = 4354, heap = 294908
I (2783) main: get blob @83, error = 4354, heap = 294908
I (2793) main: get blob @84, error = 4354, heap = 294908
I (2793) main: get blob @85, error = 4354, heap = 294908
I (2803) main: get blob @86, error = 4354, heap = 294908
I (2813) main: get blob @87, error = 4354, heap = 294908
I (2813) main: get blob @88, error = 4354, heap = 294908
I (2823) main: get blob @89, error = 4354, heap = 294908
I (2823) main: get blob @90, error = 4354, heap = 294908
I (2833) main: get blob @91, error = 4354, heap = 294908
I (2843) main: get blob @92, error = 4354, heap = 294908
I (2843) main: get blob @93, error = 4354, heap = 294908
I (2853) main: get blob @94, error = 4354, heap = 294908
I (2853) main: get blob @95, error = 4354, heap = 294908
I (2863) main: get blob @96, error = 4354, heap = 294908
I (2873) main: get blob @97, error = 4354, heap = 294908
I (2873) main: get blob @98, error = 4354, heap = 294908
I (2883) main: get blob @99, error = 4354, heap = 294908
I (2883) main: 2 Heap after delete all 294908
I (2893) main: 2 Heap after close 294928
I (2893) main: 2 Heap after open 294908
I (2943) main: set int @0, error = 0, heap = 294908
I (2943) main: set int @1, error = 0, heap = 294908
I (2943) main: set int @2, error = 0, heap = 294908
I (2943) main: set int @3, error = 0, heap = 294908
I (2953) main: set int @4, error = 0, heap = 294908
I (2953) main: set int @5, error = 0, heap = 294908
I (2963) main: set int @6, error = 0, heap = 294908
I (2973) main: set int @7, error = 0, heap = 294908
I (2973) main: set int @8, error = 0, heap = 294908
I (2983) main: set int @9, error = 0, heap = 294908
I (2983) main: set int @10, error = 0, heap = 294908
I (2993) main: set int @11, error = 0, heap = 294908
I (2993) main: set int @12, error = 0, heap = 294908
I (3003) main: set int @13, error = 0, heap = 294908
I (3003) main: set int @14, error = 0, heap = 294908
I (3013) main: set int @15, error = 0, heap = 294908
I (3023) main: set int @16, error = 0, heap = 294908
I (3023) main: set int @17, error = 0, heap = 294908
I (3033) main: set int @18, error = 0, heap = 294908
I (3033) main: set int @19, error = 0, heap = 294908
I (3043) main: set int @20, error = 0, heap = 294908
I (3043) main: set int @21, error = 0, heap = 294908
I (3053) main: set int @22, error = 0, heap = 294908
I (3063) main: set int @23, error = 0, heap = 294908
I (3063) main: set int @24, error = 0, heap = 294908
I (3073) main: set int @25, error = 0, heap = 294908
I (3073) main: set int @26, error = 0, heap = 294908
I (3083) main: set int @27, error = 0, heap = 294908
I (3083) main: set int @28, error = 0, heap = 294908
I (3093) main: set int @29, error = 0, heap = 294776
I (3103) main: set int @30, error = 0, heap = 294776
I (3103) main: set int @31, error = 0, heap = 294776
I (3113) main: set int @32, error = 0, heap = 294776
I (3113) main: set int @33, error = 0, heap = 294776
I (3123) main: set int @34, error = 0, heap = 294776
I (3123) main: set int @35, error = 0, heap = 294776
I (3133) main: set int @36, error = 0, heap = 294776
I (3133) main: set int @37, error = 0, heap = 294776
I (3143) main: set int @38, error = 0, heap = 294776
I (3153) main: set int @39, error = 0, heap = 294776
I (3153) main: set int @40, error = 0, heap = 294776
I (3163) main: set int @41, error = 0, heap = 294776
I (3163) main: set int @42, error = 0, heap = 294776
I (3173) main: set int @43, error = 0, heap = 294776
I (3173) main: set int @44, error = 0, heap = 294776
I (3183) main: set int @45, error = 0, heap = 294776
I (3193) main: set int @46, error = 0, heap = 294776
I (3193) main: set int @47, error = 0, heap = 294776
I (3203) main: set int @48, error = 0, heap = 294776
I (3203) main: set int @49, error = 0, heap = 294776
I (3213) main: set int @50, error = 0, heap = 294776
I (3213) main: set int @51, error = 0, heap = 294776
I (3223) main: set int @52, error = 0, heap = 294776
I (3223) main: set int @53, error = 0, heap = 294776
I (3233) main: set int @54, error = 0, heap = 294776
I (3243) main: set int @55, error = 0, heap = 294776
I (3243) main: set int @56, error = 0, heap = 294776
I (3253) main: set int @57, error = 0, heap = 294776
I (3253) main: set int @58, error = 0, heap = 294644
I (3263) main: set int @59, error = 0, heap = 294644
I (3263) main: set int @60, error = 0, heap = 294644
I (3273) main: set int @61, error = 0, heap = 294644
I (3283) main: set int @62, error = 0, heap = 294644
I (3283) main: set int @63, error = 0, heap = 294644
I (3293) main: set int @64, error = 0, heap = 294644
I (3293) main: set int @65, error = 0, heap = 294644
I (3303) main: set int @66, error = 0, heap = 294644
I (3303) main: set int @67, error = 0, heap = 294644
I (3313) main: set int @68, error = 0, heap = 294644
I (3323) main: set int @69, error = 0, heap = 294644
I (3323) main: set int @70, error = 0, heap = 294644
I (3333) main: set int @71, error = 0, heap = 294644
I (3333) main: set int @72, error = 0, heap = 294644
I (3343) main: set int @73, error = 0, heap = 294644
I (3343) main: set int @74, error = 0, heap = 294644
I (3353) main: set int @75, error = 0, heap = 294644
I (3353) main: set int @76, error = 0, heap = 294644
I (3363) main: set int @77, error = 0, heap = 294644
I (3373) main: set int @78, error = 0, heap = 294644
I (3373) main: set int @79, error = 0, heap = 294644
I (3383) main: set int @80, error = 0, heap = 294644
I (3383) main: set int @81, error = 0, heap = 294644
I (3393) main: set int @82, error = 0, heap = 294644
I (3393) main: set int @83, error = 0, heap = 294644
I (3403) main: set int @84, error = 0, heap = 294644
I (3413) main: set int @85, error = 0, heap = 294644
I (3413) main: set int @86, error = 0, heap = 294644
I (3423) main: set int @87, error = 0, heap = 294512
I (3423) main: set int @88, error = 0, heap = 294512
I (3433) main: set int @89, error = 0, heap = 294512
I (3433) main: set int @90, error = 0, heap = 294512
I (3443) main: set int @91, error = 0, heap = 294512
I (3443) main: set int @92, error = 0, heap = 294512
I (3453) main: set int @93, error = 0, heap = 294512
I (3463) main: set int @94, error = 0, heap = 294512
I (3463) main: set int @95, error = 0, heap = 294512
I (3473) main: set int @96, error = 0, heap = 294512
I (3473) main: set int @97, error = 0, heap = 294512
I (3483) main: set int @98, error = 0, heap = 294512
I (3483) main: set int @99, error = 0, heap = 294512
I (3493) main: removed int @0, error = 0, heap = 294512
I (3503) main: removed int @1, error = 0, heap = 294512
I (3503) main: removed int @2, error = 0, heap = 294512
I (3513) main: removed int @3, error = 0, heap = 294512
I (3513) main: removed int @4, error = 0, heap = 294512
I (3523) main: removed int @5, error = 0, heap = 294512
I (3533) main: removed int @6, error = 0, heap = 294512
I (3533) main: removed int @7, error = 0, heap = 294512
I (3543) main: removed int @8, error = 0, heap = 294512
I (3543) main: removed int @9, error = 0, heap = 294512
I (3553) main: removed int @10, error = 0, heap = 294512
I (3563) main: removed int @11, error = 0, heap = 294512
I (3563) main: removed int @12, error = 0, heap = 294512
I (3573) main: removed int @13, error = 0, heap = 294512
I (3573) main: removed int @14, error = 0, heap = 294512
I (3583) main: removed int @15, error = 0, heap = 294512
I (3593) main: removed int @16, error = 0, heap = 294512
I (3593) main: removed int @17, error = 0, heap = 294512
I (3603) main: removed int @18, error = 0, heap = 294512
I (3603) main: removed int @19, error = 0, heap = 294512
I (3613) main: removed int @20, error = 0, heap = 294512
I (3623) main: removed int @21, error = 0, heap = 294512
I (3623) main: removed int @22, error = 0, heap = 294512
I (3633) main: removed int @23, error = 0, heap = 294512
I (3633) main: removed int @24, error = 0, heap = 294512
I (3643) main: removed int @25, error = 0, heap = 294512
I (3653) main: removed int @26, error = 0, heap = 294512
I (3653) main: removed int @27, error = 0, heap = 294512
I (3663) main: removed int @28, error = 0, heap = 294512
I (3663) main: removed int @29, error = 0, heap = 294640
I (3673) main: removed int @30, error = 0, heap = 294640
I (3683) main: removed int @31, error = 0, heap = 294640
I (3683) main: removed int @32, error = 0, heap = 294640
I (3693) main: removed int @33, error = 0, heap = 294640
I (3693) main: removed int @34, error = 0, heap = 294640
I (3703) main: removed int @35, error = 0, heap = 294640
I (3713) main: removed int @36, error = 0, heap = 294640
I (3713) main: removed int @37, error = 0, heap = 294640
I (3723) main: removed int @38, error = 0, heap = 294640
I (3723) main: removed int @39, error = 0, heap = 294640
I (3733) main: removed int @40, error = 0, heap = 294640
I (3733) main: removed int @41, error = 0, heap = 294640
I (3743) main: removed int @42, error = 0, heap = 294640
I (3753) main: removed int @43, error = 0, heap = 294640
I (3753) main: removed int @44, error = 0, heap = 294640
I (3763) main: removed int @45, error = 0, heap = 294640
I (3763) main: removed int @46, error = 0, heap = 294640
I (3773) main: removed int @47, error = 0, heap = 294640
I (3783) main: removed int @48, error = 0, heap = 294640
I (3783) main: removed int @49, error = 0, heap = 294640
I (3793) main: removed int @50, error = 0, heap = 294640
I (3793) main: removed int @51, error = 0, heap = 294640
I (3803) main: removed int @52, error = 0, heap = 294640
I (3813) main: removed int @53, error = 0, heap = 294640
I (3813) main: removed int @54, error = 0, heap = 294640
I (3823) main: removed int @55, error = 0, heap = 294640
I (3823) main: removed int @56, error = 0, heap = 294640
I (3833) main: removed int @57, error = 0, heap = 294640
I (3843) main: removed int @58, error = 0, heap = 294768
I (3843) main: removed int @59, error = 0, heap = 294768
I (3853) main: removed int @60, error = 0, heap = 294768
I (3853) main: removed int @61, error = 0, heap = 294768
I (3863) main: removed int @62, error = 0, heap = 294768
I (3873) main: removed int @63, error = 0, heap = 294768
I (3873) main: removed int @64, error = 0, heap = 294768
I (3883) main: removed int @65, error = 0, heap = 294768
I (3883) main: removed int @66, error = 0, heap = 294768
I (3893) main: removed int @67, error = 0, heap = 294768
I (3903) main: removed int @68, error = 0, heap = 294768
I (3903) main: removed int @69, error = 0, heap = 294768
I (3913) main: removed int @70, error = 0, heap = 294768
I (3913) main: removed int @71, error = 0, heap = 294768
I (3923) main: removed int @72, error = 0, heap = 294768
I (3933) main: removed int @73, error = 0, heap = 294768
I (3933) main: removed int @74, error = 0, heap = 294768
I (3943) main: removed int @75, error = 0, heap = 294768
I (3943) main: removed int @76, error = 0, heap = 294768
I (3953) main: removed int @77, error = 0, heap = 294768
I (3963) main: removed int @78, error = 0, heap = 294768
I (3963) main: removed int @79, error = 0, heap = 294768
I (3973) main: removed int @80, error = 0, heap = 294768
I (3973) main: removed int @81, error = 0, heap = 294768
I (3983) main: removed int @82, error = 0, heap = 294768
I (3993) main: removed int @83, error = 0, heap = 294768
I (3993) main: removed int @84, error = 0, heap = 294768
I (4003) main: removed int @85, error = 0, heap = 294768
I (4003) main: removed int @86, error = 0, heap = 294768
I (4013) main: removed int @87, error = 0, heap = 294900
I (4023) main: removed int @88, error = 0, heap = 294900
I (4023) main: removed int @89, error = 0, heap = 294900
I (4033) main: removed int @90, error = 0, heap = 294900
I (4033) main: removed int @91, error = 0, heap = 294900
I (4043) main: removed int @92, error = 0, heap = 294900
I (4053) main: removed int @93, error = 0, heap = 294900
I (4053) main: removed int @94, error = 0, heap = 294900
I (4063) main: removed int @95, error = 0, heap = 294900
I (4063) main: removed int @96, error = 0, heap = 294900
I (4073) main: removed int @97, error = 0, heap = 294900
I (4083) main: removed int @98, error = 0, heap = 294900
I (4083) main: removed int @99, error = 0, heap = 294900

@tommybazar
Copy link
Author

@agmuntianu
I was just compiling this again to run it. Your logs illustrate the leak pretty well, but it looks like your NVS partition is kind of small, as blobs 70+ don't get saved at all.

My logs (with all blobs getting saved, but same thing in the end)

Logs

(322) main: set blob @0, error = 0, heap = 271804
(324) main: set blob @1, error = 0, heap = 271620
(326) main: set blob @2, error = 0, heap = 271620
(330) main: set blob @3, error = 0, heap = 271620
(336) main: set blob @4, error = 0, heap = 271620
(340) main: set blob @5, error = 0, heap = 271620
(346) main: set blob @6, error = 0, heap = 271620
(352) main: set blob @7, error = 0, heap = 271620
(358) main: set blob @8, error = 0, heap = 271620
(364) main: set blob @9, error = 0, heap = 271620
(368) main: set blob @10, error = 0, heap = 271620
(374) main: set blob @11, error = 0, heap = 271620
(380) main: set blob @12, error = 0, heap = 271620
(386) main: set blob @13, error = 0, heap = 271620
(426) main: set blob @14, error = 0, heap = 271488
(428) main: set blob @15, error = 0, heap = 271488
(430) main: set blob @16, error = 0, heap = 271488
(434) main: set blob @17, error = 0, heap = 271488
(440) main: set blob @18, error = 0, heap = 271488
(446) main: set blob @19, error = 0, heap = 271488
(452) main: set blob @20, error = 0, heap = 271488
(456) main: set blob @21, error = 0, heap = 271488
(462) main: set blob @22, error = 0, heap = 271488
(468) main: set blob @23, error = 0, heap = 271488
(474) main: set blob @24, error = 0, heap = 271488
(480) main: set blob @25, error = 0, heap = 271488
(486) main: set blob @26, error = 0, heap = 271488
(490) main: set blob @27, error = 0, heap = 271488
(536) main: set blob @28, error = 0, heap = 271356
(538) main: set blob @29, error = 0, heap = 271356
(540) main: set blob @30, error = 0, heap = 271356
(544) main: set blob @31, error = 0, heap = 271356
(550) main: set blob @32, error = 0, heap = 271356
(556) main: set blob @33, error = 0, heap = 271356
(560) main: set blob @34, error = 0, heap = 271356
(566) main: set blob @35, error = 0, heap = 271356
(572) main: set blob @36, error = 0, heap = 271356
(578) main: set blob @37, error = 0, heap = 271356
(584) main: set blob @38, error = 0, heap = 271356
(588) main: set blob @39, error = 0, heap = 271356
(594) main: set blob @40, error = 0, heap = 271356
(600) main: set blob @41, error = 0, heap = 271356
(644) main: set blob @42, error = 0, heap = 271224
(648) main: set blob @43, error = 0, heap = 271224
(650) main: set blob @44, error = 0, heap = 271224
(652) main: set blob @45, error = 0, heap = 271224
(658) main: set blob @46, error = 0, heap = 271224
(664) main: set blob @47, error = 0, heap = 271224
(670) main: set blob @48, error = 0, heap = 271224
(676) main: set blob @49, error = 0, heap = 271224
(682) main: set blob @50, error = 0, heap = 271224
(686) main: set blob @51, error = 0, heap = 271224
(692) main: set blob @52, error = 0, heap = 271224
(698) main: set blob @53, error = 0, heap = 271224
(704) main: set blob @54, error = 0, heap = 271224
(710) main: set blob @55, error = 0, heap = 271224
(750) main: set blob @56, error = 0, heap = 271092
(754) main: set blob @57, error = 0, heap = 271092
(756) main: set blob @58, error = 0, heap = 271092
(758) main: set blob @59, error = 0, heap = 271092
(764) main: set blob @60, error = 0, heap = 271092
(770) main: set blob @61, error = 0, heap = 271092
(776) main: set blob @62, error = 0, heap = 271092
(782) main: set blob @63, error = 0, heap = 271092
(788) main: set blob @64, error = 0, heap = 271092
(792) main: set blob @65, error = 0, heap = 271092
(798) main: set blob @66, error = 0, heap = 271092
(804) main: set blob @67, error = 0, heap = 271092
(810) main: set blob @68, error = 0, heap = 271092
(816) main: set blob @69, error = 0, heap = 271092
(858) main: set blob @70, error = 0, heap = 270960
(860) main: set blob @71, error = 0, heap = 270960
(864) main: set blob @72, error = 0, heap = 270960
(866) main: set blob @73, error = 0, heap = 270960
(872) main: set blob @74, error = 0, heap = 270960
(878) main: set blob @75, error = 0, heap = 270960
(884) main: set blob @76, error = 0, heap = 270960
(890) main: set blob @77, error = 0, heap = 270960
(894) main: set blob @78, error = 0, heap = 270960
(900) main: set blob @79, error = 0, heap = 270960
(906) main: set blob @80, error = 0, heap = 270960
(912) main: set blob @81, error = 0, heap = 270960
(918) main: set blob @82, error = 0, heap = 270960
(924) main: set blob @83, error = 0, heap = 270960
(964) main: set blob @84, error = 0, heap = 270828
(966) main: set blob @85, error = 0, heap = 270828
(968) main: set blob @86, error = 0, heap = 270828
(972) main: set blob @87, error = 0, heap = 270828
(978) main: set blob @88, error = 0, heap = 270828
(984) main: set blob @89, error = 0, heap = 270828
(988) main: set blob @90, error = 0, heap = 270828
(994) main: set blob @91, error = 0, heap = 270828
(1000) main: set blob @92, error = 0, heap = 270828
(1006) main: set blob @93, error = 0, heap = 270828
(1012) main: set blob @94, error = 0, heap = 270828
(1018) main: set blob @95, error = 0, heap = 270828
(1022) main: set blob @96, error = 0, heap = 270828
(1028) main: set blob @97, error = 0, heap = 270828
(1072) main: set blob @98, error = 0, heap = 270696
(1074) main: set blob @99, error = 0, heap = 270696
(1076) main: get blob @0, error = 0, heap = 270696
(1080) main: get blob @1, error = 0, heap = 270696
(1086) main: get blob @2, error = 0, heap = 270696
(1092) main: get blob @3, error = 0, heap = 270696
(1098) main: get blob @4, error = 0, heap = 270696
(1102) main: get blob @5, error = 0, heap = 270696
(1108) main: get blob @6, error = 0, heap = 270696
(1114) main: get blob @7, error = 0, heap = 270696
(1120) main: get blob @8, error = 0, heap = 270696
(1126) main: get blob @9, error = 0, heap = 270696
(1132) main: get blob @10, error = 0, heap = 270696
(1136) main: get blob @11, error = 0, heap = 270696
(1142) main: get blob @12, error = 0, heap = 270696
(1148) main: get blob @13, error = 0, heap = 270696
(1154) main: get blob @14, error = 0, heap = 270696
(1160) main: get blob @15, error = 0, heap = 270696
(1166) main: get blob @16, error = 0, heap = 270696
(1172) main: get blob @17, error = 0, heap = 270696
(1176) main: get blob @18, error = 0, heap = 270696
(1182) main: get blob @19, error = 0, heap = 270696
(1188) main: get blob @20, error = 0, heap = 270696
(1194) main: get blob @21, error = 0, heap = 270696
(1200) main: get blob @22, error = 0, heap = 270696
(1206) main: get blob @23, error = 0, heap = 270696
(1212) main: get blob @24, error = 0, heap = 270696
(1218) main: get blob @25, error = 0, heap = 270696
(1222) main: get blob @26, error = 0, heap = 270696
(1228) main: get blob @27, error = 0, heap = 270696
(1234) main: get blob @28, error = 0, heap = 270696
(1240) main: get blob @29, error = 0, heap = 270696
(1246) main: get blob @30, error = 0, heap = 270696
(1252) main: get blob @31, error = 0, heap = 270696
(1258) main: get blob @32, error = 0, heap = 270696
(1262) main: get blob @33, error = 0, heap = 270696
(1268) main: get blob @34, error = 0, heap = 270696
(1274) main: get blob @35, error = 0, heap = 270696
(1280) main: get blob @36, error = 0, heap = 270696
(1286) main: get blob @37, error = 0, heap = 270696
(1292) main: get blob @38, error = 0, heap = 270696
(1298) main: get blob @39, error = 0, heap = 270696
(1302) main: get blob @40, error = 0, heap = 270696
(1308) main: get blob @41, error = 0, heap = 270696
(1314) main: get blob @42, error = 0, heap = 270696
(1320) main: get blob @43, error = 0, heap = 270696
(1326) main: get blob @44, error = 0, heap = 270696
(1332) main: get blob @45, error = 0, heap = 270696
(1338) main: get blob @46, error = 0, heap = 270696
(1344) main: get blob @47, error = 0, heap = 270696
(1348) main: get blob @48, error = 0, heap = 270696
(1354) main: get blob @49, error = 0, heap = 270696
(1360) main: get blob @50, error = 0, heap = 270696
(1366) main: get blob @51, error = 0, heap = 270696
(1372) main: get blob @52, error = 0, heap = 270696
(1378) main: get blob @53, error = 0, heap = 270696
(1384) main: get blob @54, error = 0, heap = 270696
(1388) main: get blob @55, error = 0, heap = 270696
(1394) main: get blob @56, error = 0, heap = 270696
(1400) main: get blob @57, error = 0, heap = 270696
(1406) main: get blob @58, error = 0, heap = 270696
(1412) main: get blob @59, error = 0, heap = 270696
(1418) main: get blob @60, error = 0, heap = 270696
(1424) main: get blob @61, error = 0, heap = 270696
(1430) main: get blob @62, error = 0, heap = 270696
(1434) main: get blob @63, error = 0, heap = 270696
(1440) main: get blob @64, error = 0, heap = 270696
(1446) main: get blob @65, error = 0, heap = 270696
(1452) main: get blob @66, error = 0, heap = 270696
(1458) main: get blob @67, error = 0, heap = 270696
(1464) main: get blob @68, error = 0, heap = 270696
(1470) main: get blob @69, error = 0, heap = 270696
(1474) main: get blob @70, error = 0, heap = 270696
(1480) main: get blob @71, error = 0, heap = 270696
(1486) main: get blob @72, error = 0, heap = 270696
(1492) main: get blob @73, error = 0, heap = 270696
(1498) main: get blob @74, error = 0, heap = 270696
(1504) main: get blob @75, error = 0, heap = 270696
(1510) main: get blob @76, error = 0, heap = 270696
(1514) main: get blob @77, error = 0, heap = 270696
(1520) main: get blob @78, error = 0, heap = 270696
(1526) main: get blob @79, error = 0, heap = 270696
(1532) main: get blob @80, error = 0, heap = 270696
(1538) main: get blob @81, error = 0, heap = 270696
(1544) main: get blob @82, error = 0, heap = 270696
(1550) main: get blob @83, error = 0, heap = 270696
(1556) main: get blob @84, error = 0, heap = 270696
(1560) main: get blob @85, error = 0, heap = 270696
(1566) main: get blob @86, error = 0, heap = 270696
(1572) main: get blob @87, error = 0, heap = 270696
(1578) main: get blob @88, error = 0, heap = 270696
(1584) main: get blob @89, error = 0, heap = 270696
(1590) main: get blob @90, error = 0, heap = 270696
(1596) main: get blob @91, error = 0, heap = 270696
(1600) main: get blob @92, error = 0, heap = 270696
(1606) main: get blob @93, error = 0, heap = 270696
(1612) main: get blob @94, error = 0, heap = 270696
(1618) main: get blob @95, error = 0, heap = 270696
(1624) main: get blob @96, error = 0, heap = 270696
(1630) main: get blob @97, error = 0, heap = 270696
(1636) main: get blob @98, error = 0, heap = 270696
(1642) main: get blob @99, error = 0, heap = 270696
(1662) main: removed blob @0, error = 0, heap = 270696
(1680) main: removed blob @1, error = 0, heap = 270696
(1698) main: removed blob @2, error = 0, heap = 270696
(1716) main: removed blob @3, error = 0, heap = 270696
(1732) main: removed blob @4, error = 0, heap = 270696
(1750) main: removed blob @5, error = 0, heap = 270696
(1768) main: removed blob @6, error = 0, heap = 270696
(1786) main: removed blob @7, error = 0, heap = 270696
(1804) main: removed blob @8, error = 0, heap = 270696
(1822) main: removed blob @9, error = 0, heap = 270696
(1838) main: removed blob @10, error = 0, heap = 270696
(1856) main: removed blob @11, error = 0, heap = 270696
(1874) main: removed blob @12, error = 0, heap = 270696
(1892) main: removed blob @13, error = 0, heap = 270696
(1910) main: removed blob @14, error = 0, heap = 270696
(1926) main: removed blob @15, error = 0, heap = 270696
(1944) main: removed blob @16, error = 0, heap = 270696
(1962) main: removed blob @17, error = 0, heap = 270696
(1980) main: removed blob @18, error = 0, heap = 270696
(1998) main: removed blob @19, error = 0, heap = 270696
(2014) main: removed blob @20, error = 0, heap = 270696
(2032) main: removed blob @21, error = 0, heap = 270696
(2050) main: removed blob @22, error = 0, heap = 270696
(2068) main: removed blob @23, error = 0, heap = 270696
(2086) main: removed blob @24, error = 0, heap = 270696
(2104) main: removed blob @25, error = 0, heap = 270696
(2120) main: removed blob @26, error = 0, heap = 270696
(2138) main: removed blob @27, error = 0, heap = 270696
(2156) main: removed blob @28, error = 0, heap = 270696
(2174) main: removed blob @29, error = 0, heap = 270696
(2192) main: removed blob @30, error = 0, heap = 270696
(2208) main: removed blob @31, error = 0, heap = 270696
(2226) main: removed blob @32, error = 0, heap = 270696
(2244) main: removed blob @33, error = 0, heap = 270696
(2262) main: removed blob @34, error = 0, heap = 270696
(2280) main: removed blob @35, error = 0, heap = 270696
(2296) main: removed blob @36, error = 0, heap = 270696
(2314) main: removed blob @37, error = 0, heap = 270696
(2332) main: removed blob @38, error = 0, heap = 270696
(2350) main: removed blob @39, error = 0, heap = 270696
(2368) main: removed blob @40, error = 0, heap = 270696
(2384) main: removed blob @41, error = 0, heap = 270696
(2402) main: removed blob @42, error = 0, heap = 270696
(2420) main: removed blob @43, error = 0, heap = 270696
(2438) main: removed blob @44, error = 0, heap = 270696
(2456) main: removed blob @45, error = 0, heap = 270696
(2472) main: removed blob @46, error = 0, heap = 270696
(2490) main: removed blob @47, error = 0, heap = 270696
(2508) main: removed blob @48, error = 0, heap = 270696
(2526) main: removed blob @49, error = 0, heap = 270696
(2544) main: removed blob @50, error = 0, heap = 270696
(2562) main: removed blob @51, error = 0, heap = 270696
(2578) main: removed blob @52, error = 0, heap = 270696
(2596) main: removed blob @53, error = 0, heap = 270696
(2614) main: removed blob @54, error = 0, heap = 270696
(2632) main: removed blob @55, error = 0, heap = 270696
(2650) main: removed blob @56, error = 0, heap = 270696
(2666) main: removed blob @57, error = 0, heap = 270696
(2684) main: removed blob @58, error = 0, heap = 270696
(2702) main: removed blob @59, error = 0, heap = 270696
(2720) main: removed blob @60, error = 0, heap = 270696
(2738) main: removed blob @61, error = 0, heap = 270696
(2754) main: removed blob @62, error = 0, heap = 270696
(2772) main: removed blob @63, error = 0, heap = 270696
(2790) main: removed blob @64, error = 0, heap = 270696
(2808) main: removed blob @65, error = 0, heap = 270696
(2826) main: removed blob @66, error = 0, heap = 270696
(2844) main: removed blob @67, error = 0, heap = 270696
(2860) main: removed blob @68, error = 0, heap = 270696
(2878) main: removed blob @69, error = 0, heap = 270696
(2896) main: removed blob @70, error = 0, heap = 270696
(2914) main: removed blob @71, error = 0, heap = 270696
(2932) main: removed blob @72, error = 0, heap = 270696
(2948) main: removed blob @73, error = 0, heap = 270696
(2966) main: removed blob @74, error = 0, heap = 270696
(2984) main: removed blob @75, error = 0, heap = 270696
(3002) main: removed blob @76, error = 0, heap = 270696
(3020) main: removed blob @77, error = 0, heap = 270696
(3036) main: removed blob @78, error = 0, heap = 270696
(3054) main: removed blob @79, error = 0, heap = 270696
(3072) main: removed blob @80, error = 0, heap = 270696
(3090) main: removed blob @81, error = 0, heap = 270696
(3108) main: removed blob @82, error = 0, heap = 270696
(3124) main: removed blob @83, error = 0, heap = 270696
(3142) main: removed blob @84, error = 0, heap = 270696
(3160) main: removed blob @85, error = 0, heap = 270696
(3178) main: removed blob @86, error = 0, heap = 270696
(3196) main: removed blob @87, error = 0, heap = 270696
(3214) main: removed blob @88, error = 0, heap = 270696
(3230) main: removed blob @89, error = 0, heap = 270696
(3248) main: removed blob @90, error = 0, heap = 270696
(3266) main: removed blob @91, error = 0, heap = 270696
(3284) main: removed blob @92, error = 0, heap = 270696
(3302) main: removed blob @93, error = 0, heap = 270696
(3318) main: removed blob @94, error = 0, heap = 270696
(3336) main: removed blob @95, error = 0, heap = 270696
(3354) main: removed blob @96, error = 0, heap = 270696
(3372) main: removed blob @97, error = 0, heap = 270696
(3390) main: removed blob @98, error = 0, heap = 270696
(3406) main: removed blob @99, error = 0, heap = 270696
(3408) main: get blob @0, error = 4354, heap = 270696
(3408) main: get blob @1, error = 4354, heap = 270696
(3414) main: get blob @2, error = 4354, heap = 270696
(3420) main: get blob @3, error = 4354, heap = 270696
(3426) main: get blob @4, error = 4354, heap = 270696
(3432) main: get blob @5, error = 4354, heap = 270696
(3438) main: get blob @6, error = 4354, heap = 270696
(3444) main: get blob @7, error = 4354, heap = 270696
(3450) main: get blob @8, error = 4354, heap = 270696
(3456) main: get blob @9, error = 4354, heap = 270696
(3462) main: get blob @10, error = 4354, heap = 270696
(3468) main: get blob @11, error = 4354, heap = 270696
(3474) main: get blob @12, error = 4354, heap = 270696
(3480) main: get blob @13, error = 4354, heap = 270696
(3486) main: get blob @14, error = 4354, heap = 270696
(3492) main: get blob @15, error = 4354, heap = 270696
(3498) main: get blob @16, error = 4354, heap = 270696
(3504) main: get blob @17, error = 4354, heap = 270696
(3510) main: get blob @18, error = 4354, heap = 270696
(3516) main: get blob @19, error = 4354, heap = 270696
(3522) main: get blob @20, error = 4354, heap = 270696
(3528) main: get blob @21, error = 4354, heap = 270696
(3534) main: get blob @22, error = 4354, heap = 270696
(3540) main: get blob @23, error = 4354, heap = 270696
(3546) main: get blob @24, error = 4354, heap = 270696
(3552) main: get blob @25, error = 4354, heap = 270696
(3558) main: get blob @26, error = 4354, heap = 270696
(3564) main: get blob @27, error = 4354, heap = 270696
(3570) main: get blob @28, error = 4354, heap = 270696
(3576) main: get blob @29, error = 4354, heap = 270696
(3582) main: get blob @30, error = 4354, heap = 270696
(3588) main: get blob @31, error = 4354, heap = 270696
(3594) main: get blob @32, error = 4354, heap = 270696
(3600) main: get blob @33, error = 4354, heap = 270696
(3606) main: get blob @34, error = 4354, heap = 270696
(3612) main: get blob @35, error = 4354, heap = 270696
(3618) main: get blob @36, error = 4354, heap = 270696
(3624) main: get blob @37, error = 4354, heap = 270696
(3630) main: get blob @38, error = 4354, heap = 270696
(3636) main: get blob @39, error = 4354, heap = 270696
(3642) main: get blob @40, error = 4354, heap = 270696
(3648) main: get blob @41, error = 4354, heap = 270696
(3654) main: get blob @42, error = 4354, heap = 270696
(3660) main: get blob @43, error = 4354, heap = 270696
(3666) main: get blob @44, error = 4354, heap = 270696
(3672) main: get blob @45, error = 4354, heap = 270696
(3678) main: get blob @46, error = 4354, heap = 270696
(3684) main: get blob @47, error = 4354, heap = 270696
(3690) main: get blob @48, error = 4354, heap = 270696
(3696) main: get blob @49, error = 4354, heap = 270696
(3702) main: get blob @50, error = 4354, heap = 270696
(3708) main: get blob @51, error = 4354, heap = 270696
(3714) main: get blob @52, error = 4354, heap = 270696
(3720) main: get blob @53, error = 4354, heap = 270696
(3726) main: get blob @54, error = 4354, heap = 270696
(3732) main: get blob @55, error = 4354, heap = 270696
(3738) main: get blob @56, error = 4354, heap = 270696
(3744) main: get blob @57, error = 4354, heap = 270696
(3750) main: get blob @58, error = 4354, heap = 270696
(3756) main: get blob @59, error = 4354, heap = 270696
(3762) main: get blob @60, error = 4354, heap = 270696
(3768) main: get blob @61, error = 4354, heap = 270696
(3774) main: get blob @62, error = 4354, heap = 270696
(3780) main: get blob @63, error = 4354, heap = 270696
(3786) main: get blob @64, error = 4354, heap = 270696
(3792) main: get blob @65, error = 4354, heap = 270696
(3798) main: get blob @66, error = 4354, heap = 270696
(3804) main: get blob @67, error = 4354, heap = 270696
(3810) main: get blob @68, error = 4354, heap = 270696
(3816) main: get blob @69, error = 4354, heap = 270696
(3822) main: get blob @70, error = 4354, heap = 270696
(3828) main: get blob @71, error = 4354, heap = 270696
(3834) main: get blob @72, error = 4354, heap = 270696
(3840) main: get blob @73, error = 4354, heap = 270696
(3846) main: get blob @74, error = 4354, heap = 270696
(3852) main: get blob @75, error = 4354, heap = 270696
(3858) main: get blob @76, error = 4354, heap = 270696
(3864) main: get blob @77, error = 4354, heap = 270696
(3870) main: get blob @78, error = 4354, heap = 270696
(3876) main: get blob @79, error = 4354, heap = 270696
(3882) main: get blob @80, error = 4354, heap = 270696
(3888) main: get blob @81, error = 4354, heap = 270696
(3894) main: get blob @82, error = 4354, heap = 270696
(3900) main: get blob @83, error = 4354, heap = 270696
(3906) main: get blob @84, error = 4354, heap = 270696
(3912) main: get blob @85, error = 4354, heap = 270696
(3918) main: get blob @86, error = 4354, heap = 270696
(3924) main: get blob @87, error = 4354, heap = 270696
(3930) main: get blob @88, error = 4354, heap = 270696
(3936) main: get blob @89, error = 4354, heap = 270696
(3940) main: get blob @90, error = 4354, heap = 270696
(3946) main: get blob @91, error = 4354, heap = 270696
(3952) main: get blob @92, error = 4354, heap = 270696
(3958) main: get blob @93, error = 4354, heap = 270696
(3964) main: get blob @94, error = 4354, heap = 270696
(3970) main: get blob @95, error = 4354, heap = 270696
(3976) main: get blob @96, error = 4354, heap = 270696
(3982) main: get blob @97, error = 4354, heap = 270696
(3988) main: get blob @98, error = 4354, heap = 270696
(3994) main: get blob @99, error = 4354, heap = 270696
(4018) main: 2 Heap after delete all 270696
(4018) main: 2 Heap after close 270716
(4018) main: 2 Heap after open 270696
(4022) main: set int @0, error = 0, heap = 270696
(4026) main: set int @1, error = 0, heap = 270696
(4032) main: set int @2, error = 0, heap = 270696
(4038) main: set int @3, error = 0, heap = 270696
(4044) main: set int @4, error = 0, heap = 270696
(4050) main: set int @5, error = 0, heap = 270696
(4054) main: set int @6, error = 0, heap = 270696
(4060) main: set int @7, error = 0, heap = 270696
(4066) main: set int @8, error = 0, heap = 270696
(4072) main: set int @9, error = 0, heap = 270696
(4076) main: set int @10, error = 0, heap = 270696
(4082) main: set int @11, error = 0, heap = 270696
(4088) main: set int @12, error = 0, heap = 270696
(4094) main: set int @13, error = 0, heap = 270696
(4100) main: set int @14, error = 0, heap = 270696
(4106) main: set int @15, error = 0, heap = 270696
(4110) main: set int @16, error = 0, heap = 270696
(4116) main: set int @17, error = 0, heap = 270696
(4122) main: set int @18, error = 0, heap = 270696
(4128) main: set int @19, error = 0, heap = 270696
(4134) main: set int @20, error = 0, heap = 270696
(4140) main: set int @21, error = 0, heap = 270696
(4144) main: set int @22, error = 0, heap = 270696
(4150) main: set int @23, error = 0, heap = 270696
(4156) main: set int @24, error = 0, heap = 270696
(4162) main: set int @25, error = 0, heap = 270696
(4168) main: set int @26, error = 0, heap = 270696
(4172) main: set int @27, error = 0, heap = 270564
(4178) main: set int @28, error = 0, heap = 270564
(4184) main: set int @29, error = 0, heap = 270564
(4190) main: set int @30, error = 0, heap = 270564
(4196) main: set int @31, error = 0, heap = 270564
(4202) main: set int @32, error = 0, heap = 270564
(4206) main: set int @33, error = 0, heap = 270564
(4212) main: set int @34, error = 0, heap = 270564
(4218) main: set int @35, error = 0, heap = 270564
(4224) main: set int @36, error = 0, heap = 270564
(4230) main: set int @37, error = 0, heap = 270564
(4234) main: set int @38, error = 0, heap = 270564
(4240) main: set int @39, error = 0, heap = 270564
(4246) main: set int @40, error = 0, heap = 270564
(4252) main: set int @41, error = 0, heap = 270564
(4258) main: set int @42, error = 0, heap = 270564
(4264) main: set int @43, error = 0, heap = 270564
(4268) main: set int @44, error = 0, heap = 270564
(4274) main: set int @45, error = 0, heap = 270564
(4280) main: set int @46, error = 0, heap = 270564
(4286) main: set int @47, error = 0, heap = 270564
(4292) main: set int @48, error = 0, heap = 270564
(4298) main: set int @49, error = 0, heap = 270564
(4302) main: set int @50, error = 0, heap = 270564
(4308) main: set int @51, error = 0, heap = 270564
(4314) main: set int @52, error = 0, heap = 270564
(4320) main: set int @53, error = 0, heap = 270564
(4326) main: set int @54, error = 0, heap = 270564
(4330) main: set int @55, error = 0, heap = 270564
(4336) main: set int @56, error = 0, heap = 270432
(4342) main: set int @57, error = 0, heap = 270432
(4348) main: set int @58, error = 0, heap = 270432
(4354) main: set int @59, error = 0, heap = 270432
(4360) main: set int @60, error = 0, heap = 270432
(4364) main: set int @61, error = 0, heap = 270432
(4370) main: set int @62, error = 0, heap = 270432
(4376) main: set int @63, error = 0, heap = 270432
(4382) main: set int @64, error = 0, heap = 270432
(4388) main: set int @65, error = 0, heap = 270432
(4392) main: set int @66, error = 0, heap = 270432
(4398) main: set int @67, error = 0, heap = 270432
(4404) main: set int @68, error = 0, heap = 270432
(4410) main: set int @69, error = 0, heap = 270432
(4416) main: set int @70, error = 0, heap = 270432
(4422) main: set int @71, error = 0, heap = 270432
(4426) main: set int @72, error = 0, heap = 270432
(4432) main: set int @73, error = 0, heap = 270432
(4438) main: set int @74, error = 0, heap = 270432
(4444) main: set int @75, error = 0, heap = 270432
(4450) main: set int @76, error = 0, heap = 270432
(4454) main: set int @77, error = 0, heap = 270432
(4460) main: set int @78, error = 0, heap = 270432
(4466) main: set int @79, error = 0, heap = 270432
(4472) main: set int @80, error = 0, heap = 270432
(4478) main: set int @81, error = 0, heap = 270432
(4484) main: set int @82, error = 0, heap = 270432
(4488) main: set int @83, error = 0, heap = 270432
(4494) main: set int @84, error = 0, heap = 270432
(4500) main: set int @85, error = 0, heap = 270300
(4506) main: set int @86, error = 0, heap = 270300
(4512) main: set int @87, error = 0, heap = 270300
(4518) main: set int @88, error = 0, heap = 270300
(4522) main: set int @89, error = 0, heap = 270300
(4528) main: set int @90, error = 0, heap = 270300
(4534) main: set int @91, error = 0, heap = 270300
(4540) main: set int @92, error = 0, heap = 270300
(4546) main: set int @93, error = 0, heap = 270300
(4550) main: set int @94, error = 0, heap = 270300
(4556) main: set int @95, error = 0, heap = 270300
(4562) main: set int @96, error = 0, heap = 270300
(4568) main: set int @97, error = 0, heap = 270300
(4574) main: set int @98, error = 0, heap = 270300
(4580) main: set int @99, error = 0, heap = 270300
(4602) main: removed int @0, error = 0, heap = 270300
(4618) main: removed int @1, error = 0, heap = 270300
(4636) main: removed int @2, error = 0, heap = 270300
(4654) main: removed int @3, error = 0, heap = 270300
(4672) main: removed int @4, error = 0, heap = 270300
(4690) main: removed int @5, error = 0, heap = 270300
(4706) main: removed int @6, error = 0, heap = 270300
(4724) main: removed int @7, error = 0, heap = 270300
(4742) main: removed int @8, error = 0, heap = 270300
(4760) main: removed int @9, error = 0, heap = 270300
(4776) main: removed int @10, error = 0, heap = 270300
(4794) main: removed int @11, error = 0, heap = 270300
(4812) main: removed int @12, error = 0, heap = 270300
(4830) main: removed int @13, error = 0, heap = 270300
(4848) main: removed int @14, error = 0, heap = 270300
(4864) main: removed int @15, error = 0, heap = 270300
(4882) main: removed int @16, error = 0, heap = 270300
(4900) main: removed int @17, error = 0, heap = 270300
(4918) main: removed int @18, error = 0, heap = 270300
(4936) main: removed int @19, error = 0, heap = 270300
(4952) main: removed int @20, error = 0, heap = 270300
(4970) main: removed int @21, error = 0, heap = 270300
(4988) main: removed int @22, error = 0, heap = 270300
(5006) main: removed int @23, error = 0, heap = 270300
(5024) main: removed int @24, error = 0, heap = 270300
(5040) main: removed int @25, error = 0, heap = 270300
(5058) main: removed int @26, error = 0, heap = 270300
(5076) main: removed int @27, error = 0, heap = 270428
(5094) main: removed int @28, error = 0, heap = 270428
(5112) main: removed int @29, error = 0, heap = 270428
(5128) main: removed int @30, error = 0, heap = 270428
(5146) main: removed int @31, error = 0, heap = 270428
(5164) main: removed int @32, error = 0, heap = 270428
(5182) main: removed int @33, error = 0, heap = 270428
(5200) main: removed int @34, error = 0, heap = 270428
(5216) main: removed int @35, error = 0, heap = 270428
(5236) main: removed int @36, error = 0, heap = 270428
(5254) main: removed int @37, error = 0, heap = 270428
(5272) main: removed int @38, error = 0, heap = 270428
(5290) main: removed int @39, error = 0, heap = 270428
(5308) main: removed int @40, error = 0, heap = 270428
(5324) main: removed int @41, error = 0, heap = 270428
(5342) main: removed int @42, error = 0, heap = 270428
(5360) main: removed int @43, error = 0, heap = 270428
(5378) main: removed int @44, error = 0, heap = 270428
(5396) main: removed int @45, error = 0, heap = 270428
(5412) main: removed int @46, error = 0, heap = 270428
(5430) main: removed int @47, error = 0, heap = 270428
(5448) main: removed int @48, error = 0, heap = 270428
(5466) main: removed int @49, error = 0, heap = 270428
(5484) main: removed int @50, error = 0, heap = 270428
(5500) main: removed int @51, error = 0, heap = 270428
(5518) main: removed int @52, error = 0, heap = 270428
(5536) main: removed int @53, error = 0, heap = 270428
(5554) main: removed int @54, error = 0, heap = 270428
(5572) main: removed int @55, error = 0, heap = 270428
(5588) main: removed int @56, error = 0, heap = 270560
(5606) main: removed int @57, error = 0, heap = 270560
(5624) main: removed int @58, error = 0, heap = 270560
(5642) main: removed int @59, error = 0, heap = 270560
(5660) main: removed int @60, error = 0, heap = 270560
(5676) main: removed int @61, error = 0, heap = 270560
(5694) main: removed int @62, error = 0, heap = 270560
(5712) main: removed int @63, error = 0, heap = 270560
(5730) main: removed int @64, error = 0, heap = 270560
(5748) main: removed int @65, error = 0, heap = 270560
(5764) main: removed int @66, error = 0, heap = 270560
(5782) main: removed int @67, error = 0, heap = 270560
(5800) main: removed int @68, error = 0, heap = 270560
(5818) main: removed int @69, error = 0, heap = 270560
(5836) main: removed int @70, error = 0, heap = 270560
(5852) main: removed int @71, error = 0, heap = 270560
(5870) main: removed int @72, error = 0, heap = 270560
(5888) main: removed int @73, error = 0, heap = 270560
(5906) main: removed int @74, error = 0, heap = 270560
(5924) main: removed int @75, error = 0, heap = 270560
(5940) main: removed int @76, error = 0, heap = 270560
(5958) main: removed int @77, error = 0, heap = 270560
(5976) main: removed int @78, error = 0, heap = 270560
(5994) main: removed int @79, error = 0, heap = 270560
(6010) main: removed int @80, error = 0, heap = 270560
(6028) main: removed int @81, error = 0, heap = 270560
(6046) main: removed int @82, error = 0, heap = 270560
(6064) main: removed int @83, error = 0, heap = 270560
(6082) main: removed int @84, error = 0, heap = 270560
(6098) main: removed int @85, error = 0, heap = 270692
(6116) main: removed int @86, error = 0, heap = 270692
(6134) main: removed int @87, error = 0, heap = 270692
(6152) main: removed int @88, error = 0, heap = 270692
(6170) main: removed int @89, error = 0, heap = 270692
(6186) main: removed int @90, error = 0, heap = 270692
(6204) main: removed int @91, error = 0, heap = 270692
(6222) main: removed int @92, error = 0, heap = 270692
(6240) main: removed int @93, error = 0, heap = 270692
(6258) main: removed int @94, error = 0, heap = 270692
(6274) main: removed int @95, error = 0, heap = 270692
(6292) main: removed int @96, error = 0, heap = 270692
(6310) main: removed int @97, error = 0, heap = 270692
(6328) main: removed int @98, error = 0, heap = 270692
(6346) main: removed int @99, error = 0, heap = 270692

@agmuntianu
Copy link

Yeah , I know about the size. I actually made the project from scratch and erased the flash, so this was allocated with whatever is the default parameters for the partition table.
The logs confirm the issue , it might be the root cause of my problem also .

@negativekelvin
Copy link
Contributor

It looks like blobs use 2 hashlist entries per blob as a new hashlistblock is created every 14 blobs vs 28 ints. So probably hashlistbloocks are not being freed because both entries are not deleted since mHashList.erase is only called once or is called at the wrong time
@igrr

@igrr igrr self-assigned this Feb 20, 2018
@igrr
Copy link
Member

igrr commented Feb 20, 2018

It looks like blobs use 2 hashlist entries per blob as a new hashlistblock is created every 14 blobs vs 28 ints.

No, that's because HashListBlocks are not shared among pages, and 14 blobs of 255 bytes are enough to fill one page. So the HashListBlock still has some free entries, but the page is full. So we start writing to the next page, hence create a new HashListBlock.

The issue here is that HashListBlocks which have no items in them are deleted lazily, when the next HashListBlock::erase is called. So there's at least one HashListBlock per page hanging around, even if all the items in the page are deleted. I wouldn't go as far as calling this a memory leak (HashListBlock will be deleted along with the page if you re-initialize the storage), but i can understand why this looks unexpected. Fix is coming up.

@negativekelvin
Copy link
Contributor

negativekelvin commented Feb 20, 2018 •
edited
Loading

@igrr ok that makes sense but why are none of the 8 blob HashListBlocks deleted as erase is being called while iterating the 100 items?

@igrr
Copy link
Member

igrr commented Feb 21, 2018

There ends up being just one HashListBlock per page. As mentioned above, HashListBlock::erase removes blocks lazily — it first only removes hash item, and the next call can end up deleting the HashListBlock itself. The problem is that once there are no items left, the last HashListBlock does not get deleted. So I'm going to replace lazy deletion with deletion at once.

@negativekelvin
Copy link
Contributor

negativekelvin commented Feb 21, 2018 •
edited
Loading

Something like this?

https://gist.github.com/negativekelvin/f49d08ef0c153e8e154b2e417e549c33/revisions

void HashList::erase(size_t index)
{
    for (auto it = mBlockList.begin(); it != mBlockList.end();) {
        bool haveEntries = false;
        bool foundIndex = false;
        for (size_t i = 0; i < it->mCount; ++i) {
            if (it->mNodes[i].mIndex == index) {
                it->mNodes[i].mIndex = 0xff;
                if (haveEntries)
                   return;
                foundIndex = true;
            }
            if (it->mNodes[i].mIndex != 0xff) {
                haveEntries = true;
            }
        }
        if (!haveEntries) {
            auto tmp = it;
            ++it;
            mBlockList.erase(tmp);
            delete static_cast<HashListBlock*>(tmp);
        } else {
            ++it;
        }
        if (foundIndex)
            return;
    }
    assert(false && "item should have been present in cache");
}

I still don't understand root cause of blob never finds and deletes empty blocks while int does.

@negativekelvin
Copy link
Contributor

negativekelvin commented Feb 23, 2018 •
edited
Loading

Here is the result with the patch:

Log 
I (124) main: set blob @0, error = 0, heap = 292292
I (134) main: set blob @1, error = 0, heap = 292124
I (134) main: set blob @2, error = 0, heap = 291992
I (134) main: set blob @3, error = 0, heap = 291992
I (144) main: set blob @4, error = 0, heap = 291992
I (144) main: set blob @5, error = 0, heap = 291992
I (154) main: set blob @6, error = 0, heap = 291992
I (154) main: set blob @7, error = 0, heap = 291992
I (164) main: set blob @8, error = 0, heap = 291992
I (174) main: set blob @9, error = 0, heap = 291992
I (174) main: set blob @10, error = 0, heap = 291992
I (184) main: set blob @11, error = 0, heap = 291992
I (184) main: set blob @12, error = 0, heap = 291992
I (194) main: set blob @13, error = 0, heap = 291992
I (194) main: set blob @14, error = 0, heap = 291992
I (204) main: set blob @15, error = 0, heap = 291992
I (214) main: set blob @16, error = 0, heap = 291860
I (214) main: set blob @17, error = 0, heap = 291860
I (224) main: set blob @18, error = 0, heap = 291860
I (224) main: set blob @19, error = 0, heap = 291860
I (234) main: set blob @20, error = 0, heap = 291860
I (234) main: set blob @21, error = 0, heap = 291860
I (244) main: set blob @22, error = 0, heap = 291860
I (244) main: set blob @23, error = 0, heap = 291860
I (254) main: set blob @24, error = 0, heap = 291860
I (264) main: set blob @25, error = 0, heap = 291860
I (264) main: set blob @26, error = 0, heap = 291860
I (274) main: set blob @27, error = 0, heap = 291860
I (274) main: set blob @28, error = 0, heap = 291860
I (284) main: set blob @29, error = 0, heap = 291860
I (334) main: set blob @30, error = 0, heap = 291728
I (334) main: set blob @31, error = 0, heap = 291728
I (334) main: set blob @32, error = 0, heap = 291728
I (344) main: set blob @33, error = 0, heap = 291728
I (344) main: set blob @34, error = 0, heap = 291728
I (354) main: set blob @35, error = 0, heap = 291728
I (364) main: set blob @36, error = 0, heap = 291728
I (364) main: set blob @37, error = 0, heap = 291728
I (374) main: set blob @38, error = 0, heap = 291728
I (374) main: set blob @39, error = 0, heap = 291728
I (384) main: set blob @40, error = 0, heap = 291728
I (384) main: set blob @41, error = 0, heap = 291728
I (394) main: set blob @42, error = 0, heap = 291728
I (394) main: set blob @43, error = 0, heap = 291728
I (454) main: set blob @44, error = 0, heap = 291596
I (454) main: set blob @45, error = 0, heap = 291596
I (454) main: set blob @46, error = 0, heap = 291596
I (464) main: set blob @47, error = 0, heap = 291596
I (464) main: set blob @48, error = 0, heap = 291596
I (474) main: set blob @49, error = 0, heap = 291596
I (474) main: set blob @50, error = 0, heap = 291596
I (484) main: set blob @51, error = 0, heap = 291596
I (484) main: set blob @52, error = 0, heap = 291596
I (494) main: set blob @53, error = 0, heap = 291596
I (504) main: set blob @54, error = 0, heap = 291596
I (504) main: set blob @55, error = 0, heap = 291596
I (514) main: set blob @56, error = 0, heap = 291596
I (514) main: set blob @57, error = 0, heap = 291596
I (574) main: set blob @58, error = 0, heap = 291464
I (574) main: set blob @59, error = 0, heap = 291464
I (574) main: set blob @60, error = 0, heap = 291464
I (574) main: set blob @61, error = 0, heap = 291464
I (584) main: set blob @62, error = 0, heap = 291464
I (584) main: set blob @63, error = 0, heap = 291464
I (594) main: set blob @64, error = 0, heap = 291464
I (604) main: set blob @65, error = 0, heap = 291464
I (604) main: set blob @66, error = 0, heap = 291464
I (614) main: set blob @67, error = 0, heap = 291464
I (614) main: set blob @68, error = 0, heap = 291464
I (624) main: set blob @69, error = 0, heap = 291464
I (624) main: set blob @70, error = 0, heap = 291464
I (634) main: set blob @71, error = 0, heap = 291464
I (684) main: set blob @72, error = 0, heap = 291332
I (684) main: set blob @73, error = 0, heap = 291332
I (694) main: set blob @74, error = 0, heap = 291332
I (694) main: set blob @75, error = 0, heap = 291332
I (704) main: set blob @76, error = 0, heap = 291332
I (704) main: set blob @77, error = 0, heap = 291332
I (714) main: set blob @78, error = 0, heap = 291332
I (714) main: set blob @79, error = 0, heap = 291332
I (724) main: set blob @80, error = 0, heap = 291332
I (724) main: set blob @81, error = 0, heap = 291332
I (734) main: set blob @82, error = 0, heap = 291332
I (744) main: set blob @83, error = 0, heap = 291332
I (744) main: set blob @84, error = 0, heap = 291332
I (754) main: set blob @85, error = 0, heap = 291332
I (804) main: set blob @86, error = 0, heap = 291200
I (804) main: set blob @87, error = 0, heap = 291200
I (804) main: set blob @88, error = 0, heap = 291200
I (814) main: set blob @89, error = 0, heap = 291200
I (814) main: set blob @90, error = 0, heap = 291200
I (824) main: set blob @91, error = 0, heap = 291200
I (824) main: set blob @92, error = 0, heap = 291200
I (834) main: set blob @93, error = 0, heap = 291200
I (844) main: set blob @94, error = 0, heap = 291200
I (844) main: set blob @95, error = 0, heap = 291200
I (854) main: set blob @96, error = 0, heap = 291200
I (854) main: set blob @97, error = 0, heap = 291200
I (864) main: set blob @98, error = 0, heap = 291200
I (864) main: set blob @99, error = 0, heap = 291200
I (874) main: get blob @0, error = 0, heap = 291200
I (874) main: get blob @1, error = 0, heap = 291200
I (884) main: get blob @2, error = 0, heap = 291200
I (894) main: get blob @3, error = 0, heap = 291200
I (894) main: get blob @4, error = 0, heap = 291200
I (904) main: get blob @5, error = 0, heap = 291200
I (904) main: get blob @6, error = 0, heap = 291200
I (914) main: get blob @7, error = 0, heap = 291200
I (914) main: get blob @8, error = 0, heap = 291200
I (924) main: get blob @9, error = 0, heap = 291200
I (924) main: get blob @10, error = 0, heap = 291200
I (934) main: get blob @11, error = 0, heap = 291200
I (944) main: get blob @12, error = 0, heap = 291200
I (944) main: get blob @13, error = 0, heap = 291200
I (954) main: get blob @14, error = 0, heap = 291200
I (954) main: get blob @15, error = 0, heap = 291200
I (964) main: get blob @16, error = 0, heap = 291200
I (964) main: get blob @17, error = 0, heap = 291200
I (974) main: get blob @18, error = 0, heap = 291200
I (974) main: get blob @19, error = 0, heap = 291200
I (984) main: get blob @20, error = 0, heap = 291200
I (994) main: get blob @21, error = 0, heap = 291200
I (994) main: get blob @22, error = 0, heap = 291200
I (1004) main: get blob @23, error = 0, heap = 291200
I (1004) main: get blob @24, error = 0, heap = 291200
I (1014) main: get blob @25, error = 0, heap = 291200
I (1014) main: get blob @26, error = 0, heap = 291200
I (1024) main: get blob @27, error = 0, heap = 291200
I (1034) main: get blob @28, error = 0, heap = 291200
I (1034) main: get blob @29, error = 0, heap = 291200
I (1044) main: get blob @30, error = 0, heap = 291200
I (1044) main: get blob @31, error = 0, heap = 291200
I (1054) main: get blob @32, error = 0, heap = 291200
I (1054) main: get blob @33, error = 0, heap = 291200
I (1064) main: get blob @34, error = 0, heap = 291200
I (1074) main: get blob @35, error = 0, heap = 291200
I (1074) main: get blob @36, error = 0, heap = 291200
I (1084) main: get blob @37, error = 0, heap = 291200
I (1084) main: get blob @38, error = 0, heap = 291200
I (1094) main: get blob @39, error = 0, heap = 291200
I (1094) main: get blob @40, error = 0, heap = 291200
I (1104) main: get blob @41, error = 0, heap = 291200
I (1114) main: get blob @42, error = 0, heap = 291200
I (1114) main: get blob @43, error = 0, heap = 291200
I (1124) main: get blob @44, error = 0, heap = 291200
I (1124) main: get blob @45, error = 0, heap = 291200
I (1134) main: get blob @46, error = 0, heap = 291200
I (1134) main: get blob @47, error = 0, heap = 291200
I (1144) main: get blob @48, error = 0, heap = 291200
I (1154) main: get blob @49, error = 0, heap = 291200
I (1154) main: get blob @50, error = 0, heap = 291200
I (1164) main: get blob @51, error = 0, heap = 291200
I (1164) main: get blob @52, error = 0, heap = 291200
I (1174) main: get blob @53, error = 0, heap = 291200
I (1174) main: get blob @54, error = 0, heap = 291200
I (1184) main: get blob @55, error = 0, heap = 291200
I (1194) main: get blob @56, error = 0, heap = 291200
I (1194) main: get blob @57, error = 0, heap = 291200
I (1204) main: get blob @58, error = 0, heap = 291200
I (1204) main: get blob @59, error = 0, heap = 291200
I (1214) main: get blob @60, error = 0, heap = 291200
I (1224) main: get blob @61, error = 0, heap = 291200
I (1224) main: get blob @62, error = 0, heap = 291200
I (1234) main: get blob @63, error = 0, heap = 291200
I (1234) main: get blob @64, error = 0, heap = 291200
I (1244) main: get blob @65, error = 0, heap = 291200
I (1244) main: get blob @66, error = 0, heap = 291200
I (1254) main: get blob @67, error = 0, heap = 291200
I (1264) main: get blob @68, error = 0, heap = 291200
I (1264) main: get blob @69, error = 0, heap = 291200
I (1274) main: get blob @70, error = 0, heap = 291200
I (1274) main: get blob @71, error = 0, heap = 291200
I (1284) main: get blob @72, error = 0, heap = 291200
I (1284) main: get blob @73, error = 0, heap = 291200
I (1294) main: get blob @74, error = 0, heap = 291200
I (1304) main: get blob @75, error = 0, heap = 291200
I (1304) main: get blob @76, error = 0, heap = 291200
I (1314) main: get blob @77, error = 0, heap = 291200
I (1314) main: get blob @78, error = 0, heap = 291200
I (1324) main: get blob @79, error = 0, heap = 291200
I (1324) main: get blob @80, error = 0, heap = 291200
I (1334) main: get blob @81, error = 0, heap = 291200
I (1344) main: get blob @82, error = 0, heap = 291200
I (1344) main: get blob @83, error = 0, heap = 291200
I (1354) main: get blob @84, error = 0, heap = 291200
I (1354) main: get blob @85, error = 0, heap = 291200
I (1364) main: get blob @86, error = 0, heap = 291200
I (1364) main: get blob @87, error = 0, heap = 291200
I (1374) main: get blob @88, error = 0, heap = 291200
I (1384) main: get blob @89, error = 0, heap = 291200
I (1384) main: get blob @90, error = 0, heap = 291200
I (1394) main: get blob @91, error = 0, heap = 291200
I (1394) main: get blob @92, error = 0, heap = 291200
I (1404) main: get blob @93, error = 0, heap = 291200
I (1404) main: get blob @94, error = 0, heap = 291200
I (1414) main: get blob @95, error = 0, heap = 291200
I (1424) main: get blob @96, error = 0, heap = 291200
I (1424) main: get blob @97, error = 0, heap = 291200
I (1434) main: get blob @98, error = 0, heap = 291200
I (1434) main: get blob @99, error = 0, heap = 291200
I (1444) main: removed blob @0, error = 0, heap = 291200
I (1444) nvs_flash: Deleting empty HashListBlock 0
I (1454) main: removed blob @1, error = 0, heap = 291328
I (1464) main: removed blob @2, error = 0, heap = 291328
I (1464) main: removed blob @3, error = 0, heap = 291328
I (1474) main: removed blob @4, error = 0, heap = 291328
I (1474) main: removed blob @5, error = 0, heap = 291328
I (1484) main: removed blob @6, error = 0, heap = 291328
I (1494) main: removed blob @7, error = 0, heap = 291328
I (1494) main: removed blob @8, error = 0, heap = 291328
I (1504) main: removed blob @9, error = 0, heap = 291328
I (1504) main: removed blob @10, error = 0, heap = 291328
I (1514) main: removed blob @11, error = 0, heap = 291328
I (1524) main: removed blob @12, error = 0, heap = 291328
I (1524) main: removed blob @13, error = 0, heap = 291328
I (1534) main: removed blob @14, error = 0, heap = 291328
I (1534) nvs_flash: Deleting empty HashListBlock 0
I (1544) main: removed blob @15, error = 0, heap = 291456
I (1554) main: removed blob @16, error = 0, heap = 291456
I (1554) main: removed blob @17, error = 0, heap = 291456
I (1564) main: removed blob @18, error = 0, heap = 291456
I (1564) main: removed blob @19, error = 0, heap = 291456
I (1574) main: removed blob @20, error = 0, heap = 291456
I (1584) main: removed blob @21, error = 0, heap = 291456
I (1584) main: removed blob @22, error = 0, heap = 291456
I (1594) main: removed blob @23, error = 0, heap = 291456
I (1594) main: removed blob @24, error = 0, heap = 291456
I (1604) main: removed blob @25, error = 0, heap = 291456
I (1614) main: removed blob @26, error = 0, heap = 291456
I (1614) main: removed blob @27, error = 0, heap = 291456
I (1624) main: removed blob @28, error = 0, heap = 291456
I (1624) nvs_flash: Deleting empty HashListBlock 0
I (1634) main: removed blob @29, error = 0, heap = 291588
I (1644) main: removed blob @30, error = 0, heap = 291588
I (1644) main: removed blob @31, error = 0, heap = 291588
I (1654) main: removed blob @32, error = 0, heap = 291588
I (1664) main: removed blob @33, error = 0, heap = 291588
I (1664) main: removed blob @34, error = 0, heap = 291588
I (1674) main: removed blob @35, error = 0, heap = 291588
I (1674) main: removed blob @36, error = 0, heap = 291588
I (1684) main: removed blob @37, error = 0, heap = 291588
I (1694) main: removed blob @38, error = 0, heap = 291588
I (1694) main: removed blob @39, error = 0, heap = 291588
I (1704) main: removed blob @40, error = 0, heap = 291588
I (1704) main: removed blob @41, error = 0, heap = 291588
I (1714) main: removed blob @42, error = 0, heap = 291588
I (1724) nvs_flash: Deleting empty HashListBlock 0
I (1724) main: removed blob @43, error = 0, heap = 291720
I (1734) main: removed blob @44, error = 0, heap = 291720
I (1734) main: removed blob @45, error = 0, heap = 291720
I (1744) main: removed blob @46, error = 0, heap = 291720
I (1754) main: removed blob @47, error = 0, heap = 291720
I (1754) main: removed blob @48, error = 0, heap = 291720
I (1764) main: removed blob @49, error = 0, heap = 291720
I (1764) main: removed blob @50, error = 0, heap = 291720
I (1774) main: removed blob @51, error = 0, heap = 291720
I (1784) main: removed blob @52, error = 0, heap = 291720
I (1784) main: removed blob @53, error = 0, heap = 291720
I (1794) main: removed blob @54, error = 0, heap = 291720
I (1794) main: removed blob @55, error = 0, heap = 291720
I (1804) main: removed blob @56, error = 0, heap = 291720
I (1814) nvs_flash: Deleting empty HashListBlock 0
I (1814) main: removed blob @57, error = 0, heap = 291852
I (1824) main: removed blob @58, error = 0, heap = 291852
I (1824) main: removed blob @59, error = 0, heap = 291852
I (1834) main: removed blob @60, error = 0, heap = 291852
I (1844) main: removed blob @61, error = 0, heap = 291852
I (1844) main: removed blob @62, error = 0, heap = 291852
I (1854) main: removed blob @63, error = 0, heap = 291852
I (1854) main: removed blob @64, error = 0, heap = 291852
I (1864) main: removed blob @65, error = 0, heap = 291852
I (1874) main: removed blob @66, error = 0, heap = 291852
I (1874) main: removed blob @67, error = 0, heap = 291852
I (1884) main: removed blob @68, error = 0, heap = 291852
I (1884) main: removed blob @69, error = 0, heap = 291852
I (1894) main: removed blob @70, error = 0, heap = 291852
I (1904) nvs_flash: Deleting empty HashListBlock 0
I (1904) main: removed blob @71, error = 0, heap = 291984
I (1914) main: removed blob @72, error = 0, heap = 291984
I (1914) main: removed blob @73, error = 0, heap = 291984
I (1924) main: removed blob @74, error = 0, heap = 291984
I (1934) main: removed blob @75, error = 0, heap = 291984
I (1934) main: removed blob @76, error = 0, heap = 291984
I (1944) main: removed blob @77, error = 0, heap = 291984
I (1944) main: removed blob @78, error = 0, heap = 291984
I (1954) main: removed blob @79, error = 0, heap = 291984
I (1964) main: removed blob @80, error = 0, heap = 291984
I (1964) main: removed blob @81, error = 0, heap = 291984
I (1974) main: removed blob @82, error = 0, heap = 291984
I (1984) main: removed blob @83, error = 0, heap = 291984
I (1984) main: removed blob @84, error = 0, heap = 291984
I (1994) nvs_flash: Deleting empty HashListBlock 0
I (1994) main: removed blob @85, error = 0, heap = 292116
I (2004) main: removed blob @86, error = 0, heap = 292116
I (2014) main: removed blob @87, error = 0, heap = 292116
I (2014) main: removed blob @88, error = 0, heap = 292116
I (2024) main: removed blob @89, error = 0, heap = 292116
I (2024) main: removed blob @90, error = 0, heap = 292116
I (2034) main: removed blob @91, error = 0, heap = 292116
I (2044) main: removed blob @92, error = 0, heap = 292116
I (2044) main: removed blob @93, error = 0, heap = 292116
I (2054) main: removed blob @94, error = 0, heap = 292116
I (2054) main: removed blob @95, error = 0, heap = 292116
I (2064) main: removed blob @96, error = 0, heap = 292116
I (2074) main: removed blob @97, error = 0, heap = 292116
I (2074) main: removed blob @98, error = 0, heap = 292116
I (2084) nvs_flash: Deleting empty HashListBlock 0
I (2084) main: removed blob @99, error = 0, heap = 292252
I (2094) main: get blob @0, error = 4354, heap = 292252
I (2094) main: get blob @1, error = 4354, heap = 292252
I (2104) main: get blob @2, error = 4354, heap = 292252
I (2114) main: get blob @3, error = 4354, heap = 292252
I (2114) main: get blob @4, error = 4354, heap = 292252
I (2124) main: get blob @5, error = 4354, heap = 292252
I (2124) main: get blob @6, error = 4354, heap = 292252
I (2134) main: get blob @7, error = 4354, heap = 292252
I (2144) main: get blob @8, error = 4354, heap = 292252
I (2144) main: get blob @9, error = 4354, heap = 292252
I (2154) main: get blob @10, error = 4354, heap = 292252
I (2154) main: get blob @11, error = 4354, heap = 292252
I (2164) main: get blob @12, error = 4354, heap = 292252
I (2164) main: get blob @13, error = 4354, heap = 292252
I (2174) main: get blob @14, error = 4354, heap = 292252
I (2184) main: get blob @15, error = 4354, heap = 292252
I (2184) main: get blob @16, error = 4354, heap = 292252
I (2194) main: get blob @17, error = 4354, heap = 292252
I (2194) main: get blob @18, error = 4354, heap = 292252
I (2204) main: get blob @19, error = 4354, heap = 292252
I (2214) main: get blob @20, error = 4354, heap = 292252
I (2214) main: get blob @21, error = 4354, heap = 292252
I (2224) main: get blob @22, error = 4354, heap = 292252
I (2224) main: get blob @23, error = 4354, heap = 292252
I (2234) main: get blob @24, error = 4354, heap = 292252
I (2244) main: get blob @25, error = 4354, heap = 292252
I (2244) main: get blob @26, error = 4354, heap = 292252
I (2254) main: get blob @27, error = 4354, heap = 292252
I (2254) main: get blob @28, error = 4354, heap = 292252
I (2264) main: get blob @29, error = 4354, heap = 292252
I (2274) main: get blob @30, error = 4354, heap = 292252
I (2274) main: get blob @31, error = 4354, heap = 292252
I (2284) main: get blob @32, error = 4354, heap = 292252
I (2284) main: get blob @33, error = 4354, heap = 292252
I (2294) main: get blob @34, error = 4354, heap = 292252
I (2304) main: get blob @35, error = 4354, heap = 292252
I (2304) main: get blob @36, error = 4354, heap = 292252
I (2314) main: get blob @37, error = 4354, heap = 292252
I (2314) main: get blob @38, error = 4354, heap = 292252
I (2324) main: get blob @39, error = 4354, heap = 292252
I (2334) main: get blob @40, error = 4354, heap = 292252
I (2334) main: get blob @41, error = 4354, heap = 292252
I (2344) main: get blob @42, error = 4354, heap = 292252
I (2344) main: get blob @43, error = 4354, heap = 292252
I (2354) main: get blob @44, error = 4354, heap = 292252
I (2364) main: get blob @45, error = 4354, heap = 292252
I (2364) main: get blob @46, error = 4354, heap = 292252
I (2374) main: get blob @47, error = 4354, heap = 292252
I (2374) main: get blob @48, error = 4354, heap = 292252
I (2384) main: get blob @49, error = 4354, heap = 292252
I (2394) main: get blob @50, error = 4354, heap = 292252
I (2394) main: get blob @51, error = 4354, heap = 292252
I (2404) main: get blob @52, error = 4354, heap = 292252
I (2404) main: get blob @53, error = 4354, heap = 292252
I (2414) main: get blob @54, error = 4354, heap = 292252
I (2424) main: get blob @55, error = 4354, heap = 292252
I (2424) main: get blob @56, error = 4354, heap = 292252
I (2434) main: get blob @57, error = 4354, heap = 292252
I (2434) main: get blob @58, error = 4354, heap = 292252
I (2444) main: get blob @59, error = 4354, heap = 292252
I (2454) main: get blob @60, error = 4354, heap = 292252
I (2454) main: get blob @61, error = 4354, heap = 292252
I (2464) main: get blob @62, error = 4354, heap = 292252
I (2464) main: get blob @63, error = 4354, heap = 292252
I (2474) main: get blob @64, error = 4354, heap = 292252
I (2484) main: get blob @65, error = 4354, heap = 292252
I (2484) main: get blob @66, error = 4354, heap = 292252
I (2494) main: get blob @67, error = 4354, heap = 292252
I (2494) main: get blob @68, error = 4354, heap = 292252
I (2504) main: get blob @69, error = 4354, heap = 292252
I (2514) main: get blob @70, error = 4354, heap = 292252
I (2514) main: get blob @71, error = 4354, heap = 292252
I (2524) main: get blob @72, error = 4354, heap = 292252
I (2524) main: get blob @73, error = 4354, heap = 292252
I (2534) main: get blob @74, error = 4354, heap = 292252
I (2544) main: get blob @75, error = 4354, heap = 292252
I (2544) main: get blob @76, error = 4354, heap = 292252
I (2554) main: get blob @77, error = 4354, heap = 292252
I (2554) main: get blob @78, error = 4354, heap = 292252
I (2564) main: get blob @79, error = 4354, heap = 292252
I (2574) main: get blob @80, error = 4354, heap = 292252
I (2574) main: get blob @81, error = 4354, heap = 292252
I (2584) main: get blob @82, error = 4354, heap = 292252
I (2584) main: get blob @83, error = 4354, heap = 292252
I (2594) main: get blob @84, error = 4354, heap = 292252
I (2604) main: get blob @85, error = 4354, heap = 292252
I (2604) main: get blob @86, error = 4354, heap = 292252
I (2614) main: get blob @87, error = 4354, heap = 292252
I (2614) main: get blob @88, error = 4354, heap = 292252
I (2624) main: get blob @89, error = 4354, heap = 292252
I (2634) main: get blob @90, error = 4354, heap = 292252
I (2634) main: get blob @91, error = 4354, heap = 292252
I (2644) main: get blob @92, error = 4354, heap = 292252
I (2644) main: get blob @93, error = 4354, heap = 292252
I (2654) main: get blob @94, error = 4354, heap = 292252
I (2664) main: get blob @95, error = 4354, heap = 292252
I (2664) main: get blob @96, error = 4354, heap = 292252
I (2674) main: get blob @97, error = 4354, heap = 292252
I (2674) main: get blob @98, error = 4354, heap = 292252
I (2684) main: get blob @99, error = 4354, heap = 292252
I (2694) main: 2 Heap after delete all 292252
I (2694) main: 2 Heap after close 292272
I (2704) main: 2 Heap after open 292252
I (2754) main: set int @0, error = 0, heap = 292124
I (2754) main: set int @1, error = 0, heap = 292124
I (2754) main: set int @2, error = 0, heap = 292124
I (2754) main: set int @3, error = 0, heap = 292124
I (2764) main: set int @4, error = 0, heap = 292124
I (2774) main: set int @5, error = 0, heap = 292124
I (2774) main: set int @6, error = 0, heap = 292124
I (2784) main: set int @7, error = 0, heap = 292124
I (2784) main: set int @8, error = 0, heap = 292124
I (2794) main: set int @9, error = 0, heap = 292124
I (2794) main: set int @10, error = 0, heap = 292124
I (2804) main: set int @11, error = 0, heap = 292124
I (2804) main: set int @12, error = 0, heap = 292124
I (2814) main: set int @13, error = 0, heap = 292124
I (2824) main: set int @14, error = 0, heap = 292124
I (2824) main: set int @15, error = 0, heap = 292124
I (2834) main: set int @16, error = 0, heap = 292124
I (2834) main: set int @17, error = 0, heap = 292124
I (2844) main: set int @18, error = 0, heap = 292124
I (2844) main: set int @19, error = 0, heap = 292124
I (2854) main: set int @20, error = 0, heap = 292124
I (2864) main: set int @21, error = 0, heap = 292124
I (2864) main: set int @22, error = 0, heap = 292124
I (2874) main: set int @23, error = 0, heap = 292124
I (2874) main: set int @24, error = 0, heap = 292124
I (2884) main: set int @25, error = 0, heap = 292124
I (2884) main: set int @26, error = 0, heap = 292124
I (2894) main: set int @27, error = 0, heap = 292124
I (2894) main: set int @28, error = 0, heap = 292124
I (2904) main: set int @29, error = 0, heap = 291992
I (2914) main: set int @30, error = 0, heap = 291992
I (2914) main: set int @31, error = 0, heap = 291992
I (2924) main: set int @32, error = 0, heap = 291992
I (2924) main: set int @33, error = 0, heap = 291992
I (2934) main: set int @34, error = 0, heap = 291992
I (2934) main: set int @35, error = 0, heap = 291992
I (2944) main: set int @36, error = 0, heap = 291992
I (2954) main: set int @37, error = 0, heap = 291992
I (2954) main: set int @38, error = 0, heap = 291992
I (2964) main: set int @39, error = 0, heap = 291992
I (2964) main: set int @40, error = 0, heap = 291992
I (2974) main: set int @41, error = 0, heap = 291992
I (2974) main: set int @42, error = 0, heap = 291992
I (2984) main: set int @43, error = 0, heap = 291992
I (2994) main: set int @44, error = 0, heap = 291992
I (2994) main: set int @45, error = 0, heap = 291992
I (3004) main: set int @46, error = 0, heap = 291992
I (3004) main: set int @47, error = 0, heap = 291992
I (3014) main: set int @48, error = 0, heap = 291992
I (3014) main: set int @49, error = 0, heap = 291992
I (3024) main: set int @50, error = 0, heap = 291992
I (3024) main: set int @51, error = 0, heap = 291992
I (3034) main: set int @52, error = 0, heap = 291992
I (3044) main: set int @53, error = 0, heap = 291992
I (3044) main: set int @54, error = 0, heap = 291992
I (3054) main: set int @55, error = 0, heap = 291992
I (3054) main: set int @56, error = 0, heap = 291992
I (3064) main: set int @57, error = 0, heap = 291992
I (3064) main: set int @58, error = 0, heap = 291860
I (3074) main: set int @59, error = 0, heap = 291860
I (3084) main: set int @60, error = 0, heap = 291860
I (3084) main: set int @61, error = 0, heap = 291860
I (3094) main: set int @62, error = 0, heap = 291860
I (3094) main: set int @63, error = 0, heap = 291860
I (3104) main: set int @64, error = 0, heap = 291860
I (3104) main: set int @65, error = 0, heap = 291860
I (3114) main: set int @66, error = 0, heap = 291860
I (3114) main: set int @67, error = 0, heap = 291860
I (3124) main: set int @68, error = 0, heap = 291860
I (3134) main: set int @69, error = 0, heap = 291860
I (3134) main: set int @70, error = 0, heap = 291860
I (3144) main: set int @71, error = 0, heap = 291860
I (3144) main: set int @72, error = 0, heap = 291860
I (3154) main: set int @73, error = 0, heap = 291860
I (3154) main: set int @74, error = 0, heap = 291860
I (3164) main: set int @75, error = 0, heap = 291860
I (3174) main: set int @76, error = 0, heap = 291860
I (3174) main: set int @77, error = 0, heap = 291860
I (3184) main: set int @78, error = 0, heap = 291860
I (3184) main: set int @79, error = 0, heap = 291860
I (3194) main: set int @80, error = 0, heap = 291860
I (3194) main: set int @81, error = 0, heap = 291860
I (3204) main: set int @82, error = 0, heap = 291860
I (3214) main: set int @83, error = 0, heap = 291860
I (3214) main: set int @84, error = 0, heap = 291860
I (3224) main: set int @85, error = 0, heap = 291860
I (3224) main: set int @86, error = 0, heap = 291860
I (3234) main: set int @87, error = 0, heap = 291728
I (3234) main: set int @88, error = 0, heap = 291728
I (3244) main: set int @89, error = 0, heap = 291728
I (3244) main: set int @90, error = 0, heap = 291728
I (3254) main: set int @91, error = 0, heap = 291728
I (3264) main: set int @92, error = 0, heap = 291728
I (3264) main: set int @93, error = 0, heap = 291728
I (3274) main: set int @94, error = 0, heap = 291728
I (3274) main: set int @95, error = 0, heap = 291728
I (3284) main: set int @96, error = 0, heap = 291728
I (3284) main: set int @97, error = 0, heap = 291728
I (3294) main: set int @98, error = 0, heap = 291728
I (3304) main: set int @99, error = 0, heap = 291728
I (3304) main: removed int @0, error = 0, heap = 291728
I (3314) main: removed int @1, error = 0, heap = 291728
I (3314) main: removed int @2, error = 0, heap = 291728
I (3324) main: removed int @3, error = 0, heap = 291728
I (3334) main: removed int @4, error = 0, heap = 292496
I (3334) main: removed int @5, error = 0, heap = 292496
I (3344) main: removed int @6, error = 0, heap = 292496
I (3344) main: removed int @7, error = 0, heap = 292496
I (3354) main: removed int @8, error = 0, heap = 292496
I (3364) main: removed int @9, error = 0, heap = 292496
I (3364) main: removed int @10, error = 0, heap = 292496
I (3374) main: removed int @11, error = 0, heap = 292496
I (3374) main: removed int @12, error = 0, heap = 292496
I (3384) main: removed int @13, error = 0, heap = 292496
I (3394) main: removed int @14, error = 0, heap = 292496
I (3394) main: removed int @15, error = 0, heap = 292496
I (3404) main: removed int @16, error = 0, heap = 292496
I (3404) main: removed int @17, error = 0, heap = 292496
I (3414) main: removed int @18, error = 0, heap = 292496
I (3414) main: removed int @19, error = 0, heap = 292496
I (3424) main: removed int @20, error = 0, heap = 292496
I (3434) main: removed int @21, error = 0, heap = 292496
I (3434) main: removed int @22, error = 0, heap = 292496
I (3444) main: removed int @23, error = 0, heap = 292496
I (3444) main: removed int @24, error = 0, heap = 292496
I (3454) main: removed int @25, error = 0, heap = 292496
I (3464) main: removed int @26, error = 0, heap = 292496
I (3464) main: removed int @27, error = 0, heap = 292496
I (3474) nvs_flash: Deleting empty HashListBlock 0
I (3474) main: removed int @28, error = 0, heap = 292624
I (3484) main: removed int @29, error = 0, heap = 292624
I (3494) main: removed int @30, error = 0, heap = 292624
I (3494) main: removed int @31, error = 0, heap = 292624
I (3504) main: removed int @32, error = 0, heap = 292624
I (3504) main: removed int @33, error = 0, heap = 292624
I (3514) main: removed int @34, error = 0, heap = 292624
I (3524) main: removed int @35, error = 0, heap = 292624
I (3524) main: removed int @36, error = 0, heap = 292624
I (3534) main: removed int @37, error = 0, heap = 292624
I (3534) main: removed int @38, error = 0, heap = 292624
I (3544) main: removed int @39, error = 0, heap = 292624
I (3554) main: removed int @40, error = 0, heap = 292984
I (3554) main: removed int @41, error = 0, heap = 292984
I (3564) main: removed int @42, error = 0, heap = 292984
I (3564) main: removed int @43, error = 0, heap = 292984
I (3574) main: removed int @44, error = 0, heap = 292984
I (3584) main: removed int @45, error = 0, heap = 292984
I (3584) main: removed int @46, error = 0, heap = 292984
I (3594) main: removed int @47, error = 0, heap = 292984
I (3594) main: removed int @48, error = 0, heap = 292984
I (3604) main: removed int @49, error = 0, heap = 292984
I (3614) main: removed int @50, error = 0, heap = 292984
I (3614) main: removed int @51, error = 0, heap = 292984
I (3624) main: removed int @52, error = 0, heap = 292984
I (3624) main: removed int @53, error = 0, heap = 292984
I (3634) main: removed int @54, error = 0, heap = 292984
I (3644) main: removed int @55, error = 0, heap = 292984
I (3644) main: removed int @56, error = 0, heap = 292984
I (3654) nvs_flash: Deleting empty HashListBlock 0
I (3654) main: removed int @57, error = 0, heap = 293112
I (3664) main: removed int @58, error = 0, heap = 293112
I (3674) main: removed int @59, error = 0, heap = 293112
I (3674) main: removed int @60, error = 0, heap = 293112
I (3684) main: removed int @61, error = 0, heap = 293112
I (3684) main: removed int @62, error = 0, heap = 293112
I (3694) main: removed int @63, error = 0, heap = 293112
I (3704) main: removed int @64, error = 0, heap = 293112
I (3704) main: removed int @65, error = 0, heap = 293112
I (3714) main: removed int @66, error = 0, heap = 293112
I (3714) main: removed int @67, error = 0, heap = 293112
I (3724) main: removed int @68, error = 0, heap = 293112
I (3734) main: removed int @69, error = 0, heap = 293112
I (3734) main: removed int @70, error = 0, heap = 293112
I (3744) main: removed int @71, error = 0, heap = 293112
I (3744) main: removed int @72, error = 0, heap = 293112
I (3754) main: removed int @73, error = 0, heap = 293112
I (3764) main: removed int @74, error = 0, heap = 293112
I (3764) main: removed int @75, error = 0, heap = 293112
I (3774) main: removed int @76, error = 0, heap = 293112
I (3774) main: removed int @77, error = 0, heap = 293112
I (3784) main: removed int @78, error = 0, heap = 293112
I (3794) main: removed int @79, error = 0, heap = 293112
I (3794) main: removed int @80, error = 0, heap = 293112
I (3804) main: removed int @81, error = 0, heap = 293112
I (3804) main: removed int @82, error = 0, heap = 293112
I (3814) main: removed int @83, error = 0, heap = 293112
I (3824) main: removed int @84, error = 0, heap = 293112
I (3824) main: removed int @85, error = 0, heap = 293112
I (3834) nvs_flash: Deleting empty HashListBlock 0
I (3834) main: removed int @86, error = 0, heap = 293244
I (3844) main: removed int @87, error = 0, heap = 293244
I (3844) main: removed int @88, error = 0, heap = 293244
I (3854) main: removed int @89, error = 0, heap = 293244
I (3864) main: removed int @90, error = 0, heap = 293244
I (3864) main: removed int @91, error = 0, heap = 293244
I (3874) main: removed int @92, error = 0, heap = 293244
I (3874) main: removed int @93, error = 0, heap = 293244
I (3884) main: removed int @94, error = 0, heap = 293244
I (3894) main: removed int @95, error = 0, heap = 293244
I (3894) main: removed int @96, error = 0, heap = 293244
I (3904) main: removed int @97, error = 0, heap = 293244
I (3904) main: removed int @98, error = 0, heap = 293244
I (3914) nvs_flash: Deleting empty HashListBlock 0
I (3914) main: removed int @99, error = 0, heap = 293380

@FayeY FayeY changed the title NVS Blob and String operations leak memory [TW#18645] NVS Blob and String operations leak memory Feb 23, 2018
@negativekelvin
Copy link
Contributor

The latest changes to nvs seem to improve this issue though the block deletion is not as quick as with the patch shown above. Maybe the performance trade-off is beneficial?

@Oromis
Copy link

Oromis commented Feb 19, 2019

We ran into this issue as well (ESP-IDF v3.1.2). It causes our application to run out of memory over the course of a few days. This is intolerable for us and re-initializing NVS from time to time to get the memory back seems like an awful hack.

I integrated @negativekelvin 's patch into a fork of nvs_flash and the leak disappeared completely. I don't see any negative effects of this patch either. It would be great to have this in master since updating a modified ESP-IDF version to new releases is a nightmare.

@igrr igrr closed this as completed in 02a76ab Feb 26, 2019
igrr added a commit that referenced this issue Feb 28, 2019
@igrr
nvs: do eager cleanup of HashListBlocks
203f752 
Previously when HashList was removing items, HashListBlocks were
removed lazily. This resulted in empty HashListBlocks dangling around
in full pages, even when all items have been erased. These blocks
would only be deleted when NVS was re-initialized
(nvs_flash_deinit/nvs_flash_init).

This change does eager cleanup instead, based on the code from
@negativekelvin offered in
#1642 (comment).

Closes #1642.
igrr added a commit that referenced this issue Mar 1, 2019
@igrr
nvs: do eager cleanup of HashListBlocks
7ee102a 
Previously when HashList was removing items, HashListBlocks were
removed lazily. This resulted in empty HashListBlocks dangling around
in full pages, even when all items have been erased. These blocks
would only be deleted when NVS was re-initialized
(nvs_flash_deinit/nvs_flash_init).

This change does eager cleanup instead, based on the code from
@negativekelvin offered in
#1642 (comment).

Closes #1642.
catalinio pushed a commit to catalinio/pycom-esp-idf that referenced this issue Jun 28, 2019
@igrr
nvs: do eager cleanup of HashListBlocks
294af3d 
Previously when HashList was removing items, HashListBlocks were
removed lazily. This resulted in empty HashListBlocks dangling around
in full pages, even when all items have been erased. These blocks
would only be deleted when NVS was re-initialized
(nvs_flash_deinit/nvs_flash_init).

This change does eager cleanup instead, based on the code from
@negativekelvin offered in
espressif/esp-idf#1642 (comment).

Closes espressif/esp-idf#1642.
@hkbora hkbora mentioned this issue Jan 27, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@igrr igrr

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@Oromis @igrr @tommybazar @agmuntianu @negativekelvin