AT-Firmware: SEND FAIL caused by E:M1584. What does E:M1584 mean?

[Maintraxx]

I'm using the actual SDK 3.1.0 dev. All files are up2date.
I implemented an webserver on an STM32 which is connected to the ESP8266. I encounted after some page reloads, that the ESP8266 accepts data to send, but does not send them out. It seems, that the espconn is out of memory. But how can i fix it?!

The following screenshot is a trace. I recorded the AT interface and the debug line.

[xcguang]

Hi,
Is it TCP, SSL or UDP? Could you provide your all log form ESP8266 debug UART?
It seems lack of memory.

[Maintraxx]

I used only TCP-Connections.
Here is another screenshot, which gives you an overview:

The data have been traced with Salaea Logic (download free:)
https://www.saleae.com/de/downloads/

Here is the completed log:
Saleae_Logic_trace.zip

I also exported the Date in a Microsoft Word File (Debug is green, AT in red, AT out blue), due to the timing and the slower baud rate of the debug interface, some outputs are mixed together:
REMOVED (Bug has been closed)

Here another version without ESP-Debug (plain AT-Interface):
REMOVED (Bug has been closed)

[xcguang]

Thanks for your detailed log. Only after May Day can we debug it.
Could you send AT+SYSRAM? to query memory after every AT command, and confirm whether there is a memory leak?

[Maintraxx]

Before i put it under stress (Pressing Strg+F5 to request all pages from my webserver) SYSRAM returns 40.000 - 35.000 Bytes. Before it crashed, it still has ~ 26.000 Bytes left.

I'm wondering why there can be a value greater than 32562 Bytes (see Section info below)

(I also see EM:2936 and the AT-Rx Buffer has a problem)

I had some trouble to get a working make file. Maybe I made I mistake there?
This is the section map:

Section info:

build/app.out:     file format elf32-xtensa-le

Sections:
Idx Name          Size      VMA       LMA       File off  Algn
  0 .data         00000ab2  3ffe8000  3ffe8000  000000e0  2**4
                  CONTENTS, ALLOC, LOAD, DATA
  1 .rodata       0000393c  3ffe8ac0  3ffe8ac0  00000ba0  2**4
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  2 .bss          00007ce0  3ffec400  3ffec400  000044e0  2**4
                  ALLOC
  3 .irom0.text   00079248  40240000  40240000  0000b880  2**4
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  4 .text         0000739e  40100000  40100000  000044dc  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
------------------------------------------------------------------------------
   Section|                   Description| Start (hex)|   End (hex)|Used space
------------------------------------------------------------------------------
      data|        Initialized Data (RAM)|    3FFE8000|    3FFE8AB2|    2738
    rodata|           ReadOnly Data (RAM)|    3FFE8AC0|    3FFEC3FC|   14652
       bss|      Uninitialized Data (RAM)|    3FFEC400|    3FFF40E0|   31968
      text|          Uncached Code (IRAM)|    40100000|    4010739E|   29598
irom0_text|             Cached Code (SPI)|    40240000|    402B9248|  496200
------------------------------------------------------------------------------
Entry Point : 40100004 call_user_start()
Total Used RAM : 49358
Free RAM : 32562
Free IRam : 3170 or 19554 if 48k IRam
------------------------------------------------------------------------------
Generate 0x00000.bin and 0x40000.bin successully in folder firmware.
0x00000.bin-------->0x00000
0x40000.bin-------->0x40000

This is my makefile:
Makefile.txt
Makefile_subfolder.txt

[Maintraxx]

The memory leak leads to a wide variety of effects. At the moment I have the problem, that I want to open a new connection with "CIPSTART" but I get no response to that. If I try a simple AT (after about a minute) I always get a "busy p".
That's really annoying.

I also tried the precompiled "new" AT-Firmwer (#239) from LoPo and I see exactly the same problems. So I don't think that it is a problem of my compilation.

[kriegste]

I wonder if this is correct:
https://github.com/espressif/ESP8266_NONOS_SDK/blob/master/third_party/mbedtls/app/espconn_mbedtls.c#L506
In case of an (out of memory) error this function exits without calling mbedtls_finished_free, thus leaving a memory leak (possibly).

Should it not be like this:

...
    os_memcpy(ssl->out_ctr, finished->finished_buf, finished->finished_len);

exit:
    if (&msg->pfinished) mbedtls_finished_free(&msg->pfinished);
    return ret;
}
...

I noticed that years ago when ssl was an even bigger pain than it is now and changed it. My project works fine for days and weeks with the current master branch and ssl calls every 30 seconds- but I have to admit that I have lots of free memory and I don't get out of memory errors in the first place.

[Maintraxx]

@kriegste +1 But I don't using SSL, so in my case, this is seems not to be the matter.
@xcguang What does the SYSRAM mean? Should I be able to us all the "free bytes"? - I encounted, that the problems starts, when the RAM falls below 30k. Do I have to take other parts into account? (Stack?!).

[kriegste]

There is even more:
https://github.com/espressif/ESP8266_NONOS_SDK/blob/master/third_party/mbedtls/app/lwIPSocket.c#L324
The check needs to be against newconn, not conn.
lwIP_REQUIRE_ACTION(newconn, exit, err = ERR_MEM);

[Maintraxx]

@xcguang Can Heap-Fragmentation be responsible for the problem?

[xcguang]

@Maintraxx
We test it for days, but it works well. Do you change any code or add your code?
Maybe heap-fragmentation cannot cause the fail, I guess, if the memory is more than 20KB.

[Maintraxx]

@xcguang Thank you very much for your support!
Normally I'm using eclipse on windows. After a lot of weeks searching for the cause, i setup a linux workstation and install the "crosstool-ng" toolchain. (I used this instruction: https://cutmywire.wordpress.com/2017/05/08/esp8266-windows-10-sdk-installation/comment-page-1/ )

I compiled the same source code and tried again. With this new binarys, I have 60k of free memory (instead of 40k)! With this new binarys, everythings seems to work now! I have no idea where the difference are.

As already said, i also tried the binaries from another users and got the same issues, but for me it works now.