Update documentation and arduino debug options (#8837)

* Update documentation

Describe how to improve Exception Decoder results.

Updated build option details.

* Added d-a-v's code. And updated description and debug macro.

* Update doc

Updated platform.txt - positioned -Os such that it can be overridden by `build.opt`.

Add suggestion of using `-Og` which may improve the Decoder's line number correlation
with the problem in the source code.

Untested adjustments to platformio-build.py

* Fixed code formatting

* Fixed example

Previously `-O3` in the example had no effect. It was overridden by later flags.
Reording the `-Os` allowed changing optimization to  work. Somehow this triggered
a new warning "Stream.h:50:7: warning: 'f.Stream::_startMillis' may be used
uninitialized in this function". Replaced `-O3` with `-Og`.

* Proposed "Debug port" menu change
Preserves a pathway for `build.opt` override.

* Update boards.txt.py and docs - WIP

* Improve organization of optimization content.
Use "Debug Optimization" for menu name.
The menu item defines the initial optimization.
platform.txt - optimization parameter can be overridden by build.opt.

* Add fallback value for build.debug_optim to platform.txt

* update text and undo changes to platformio-build.py

* tweak text

* Added ':orphan:' mark to a06-global-build-options.rst

* Update doc. Added link in page index to 'Improving Exception Decoder Results'

* Update text to reference PR#8868 improvements for leaf function.

cores/esp8266/debug.h

@@ -4,8 +4,13 @@
 #include <stddef.h>
 #include <stdint.h>
 
+#define _DEBUG_LEAF_FUNCTION(...) __asm__ __volatile__("" ::: "a0", "memory")
+
 #ifdef DEBUG_ESP_CORE
 #define DEBUGV(fmt, ...) ::printf((PGM_P)PSTR(fmt), ##__VA_ARGS__)
+#define DEBUG_LEAF_FUNCTION _DEBUG_LEAF_FUNCTION
+#else
+#define DEBUG_LEAF_FUNCTION(...)
 #endif
 
 #ifndef DEBUGV

doc/Troubleshooting/stack_dump.rst

@@ -12,42 +12,42 @@ Example:
 
     Exception (0): epc1=0x402103f4 epc2=0x00000000 epc3=0x00000000 excvaddr=0x00000000 depc=0x00000000
 
-    ctx: sys 
+    ctx: sys
     sp: 3ffffc10 end: 3fffffb0 offset: 01a0
 
     >>>stack>>>
-    3ffffdb0:  40223e00 3fff6f50 00000010 60000600  
-    3ffffdc0:  00000001 4021f774 3fffc250 4000050c  
-    3ffffdd0:  400043d5 00000030 00000016 ffffffff  
-    3ffffde0:  400044ab 3fffc718 3ffffed0 08000000  
-    3ffffdf0:  60000200 08000000 00000003 00000000  
-    3ffffe00:  0000ffff 00000001 04000002 003fd000  
-    3ffffe10:  3fff7188 000003fd 3fff2564 00000030  
-    3ffffe20:  40101709 00000008 00000008 00000020  
-    3ffffe30:  c1948db3 394c5e70 7f2060f2 c6ba0c87  
-    3ffffe40:  3fff7058 00000001 40238d41 3fff6ff0  
-    3ffffe50:  3fff6f50 00000010 60000600 00000020  
-    3ffffe60:  402301a8 3fff7098 3fff7014 40238c77  
-    3ffffe70:  4022fb6c 40230ebe 3fff1a5b 3fff6f00  
-    3ffffe80:  3ffffec8 00000010 40231061 3fff0f90  
-    3ffffe90:  3fff6848 3ffed0c0 60000600 3fff6ae0  
-    3ffffea0:  3fff0f90 3fff0f90 3fff6848 3fff6d40  
-    3ffffeb0:  3fff28e8 40101233 d634fe1a fffeffff  
-    3ffffec0:  00000001 00000000 4022d5d6 3fff6848  
-    3ffffed0:  00000002 4000410f 3fff2394 3fff6848  
-    3ffffee0:  3fffc718 40004a3c 000003fd 3fff7188  
-    3ffffef0:  3fffc718 40101510 00000378 3fff1a5b  
-    3fffff00:  000003fd 4021d2e7 00000378 000003ff  
-    3fffff10:  00001000 4021d37d 3fff2564 000003ff  
-    3fffff20:  000003fd 60000600 003fd000 3fff2564  
-    3fffff30:  ffffff00 55aa55aa 00000312 0000001c  
-    3fffff40:  0000001c 0000008a 0000006d 000003ff  
-    3fffff50:  4021d224 3ffecf90 00000000 3ffed0c0  
-    3fffff60:  00000001 4021c2e9 00000003 3fff1238  
-    3fffff70:  4021c071 3ffecf84 3ffecf30 0026a2b0  
-    3fffff80:  4021c0b6 3fffdab0 00000000 3fffdcb0  
-    3fffff90:  3ffecf40 3fffdab0 00000000 3fffdcc0  
-    3fffffa0:  40000f49 40000f49 3fffdab0 40000f49  
+    3ffffdb0:  40223e00 3fff6f50 00000010 60000600
+    3ffffdc0:  00000001 4021f774 3fffc250 4000050c
+    3ffffdd0:  400043d5 00000030 00000016 ffffffff
+    3ffffde0:  400044ab 3fffc718 3ffffed0 08000000
+    3ffffdf0:  60000200 08000000 00000003 00000000
+    3ffffe00:  0000ffff 00000001 04000002 003fd000
+    3ffffe10:  3fff7188 000003fd 3fff2564 00000030
+    3ffffe20:  40101709 00000008 00000008 00000020
+    3ffffe30:  c1948db3 394c5e70 7f2060f2 c6ba0c87
+    3ffffe40:  3fff7058 00000001 40238d41 3fff6ff0
+    3ffffe50:  3fff6f50 00000010 60000600 00000020
+    3ffffe60:  402301a8 3fff7098 3fff7014 40238c77
+    3ffffe70:  4022fb6c 40230ebe 3fff1a5b 3fff6f00
+    3ffffe80:  3ffffec8 00000010 40231061 3fff0f90
+    3ffffe90:  3fff6848 3ffed0c0 60000600 3fff6ae0
+    3ffffea0:  3fff0f90 3fff0f90 3fff6848 3fff6d40
+    3ffffeb0:  3fff28e8 40101233 d634fe1a fffeffff
+    3ffffec0:  00000001 00000000 4022d5d6 3fff6848
+    3ffffed0:  00000002 4000410f 3fff2394 3fff6848
+    3ffffee0:  3fffc718 40004a3c 000003fd 3fff7188
+    3ffffef0:  3fffc718 40101510 00000378 3fff1a5b
+    3fffff00:  000003fd 4021d2e7 00000378 000003ff
+    3fffff10:  00001000 4021d37d 3fff2564 000003ff
+    3fffff20:  000003fd 60000600 003fd000 3fff2564
+    3fffff30:  ffffff00 55aa55aa 00000312 0000001c
+    3fffff40:  0000001c 0000008a 0000006d 000003ff
+    3fffff50:  4021d224 3ffecf90 00000000 3ffed0c0
+    3fffff60:  00000001 4021c2e9 00000003 3fff1238
+    3fffff70:  4021c071 3ffecf84 3ffecf30 0026a2b0
+    3fffff80:  4021c0b6 3fffdab0 00000000 3fffdcb0
+    3fffff90:  3ffecf40 3fffdab0 00000000 3fffdcc0
+    3fffffa0:  40000f49 40000f49 3fffdab0 40000f49
     <<<stack<<<
 
 The first number after ``Exception`` gives the cause of the reset. a
@@ -57,7 +57,10 @@ the hex after are the stack dump.
 Decode
 ~~~~~~
 
-It's possible to decode the Stack to readable information. For more info see the `Esp Exception Decoder <https://github.com/me-no-dev/EspExceptionDecoder>`__ tool.
+It's possible to decode the Stack to readable information.
+You can get a copy and read about the `Esp Exception Decoder <https://github.com/me-no-dev/EspExceptionDecoder>`__ tool.
+
+For a troubleshooting example using the Exception Decoder Tool, read `FAQ: My ESP Crashes <../faq/a02-my-esp-crashes.rst#exception-decoder>`__.
 
 .. figure:: ESP_Exception_Decoderp.png
    :alt: ESP Exception Decoder

doc/faq/a02-my-esp-crashes.rst

@@ -11,6 +11,7 @@ My ESP crashes running some code. How to troubleshoot it?
 -  `Exception <#exception>`__
 -  `Watchdog <#watchdog>`__
 -  `Exception Decoder <#exception-decoder>`__
+-  `Improving Exception Decoder Results <#improving-exception-decoder-results>`__
 -  `Other Common Causes for Crashes <#other-causes-for-crashes>`__
 -  `If at the Wall, Enter an Issue
    Report <#if-at-the-wall-enter-an-issue-report>`__
@@ -236,6 +237,7 @@ If you don't have any code for troubleshooting, use the example below:
 
     void loop(){}
 
+
 Enable the Out-Of-Memory (*OOM*) debug option (in the *Tools > Debug Level*
 menu), compile/flash/upload this code to your ESP (Ctrl+U) and start Serial
 Monitor (Ctrl+Shift+M).  You should shortly see ESP restarting every couple
@@ -270,31 +272,92 @@ Decoder <https://github.com/me-no-dev/EspExceptionDecoder>`__ you can
 track down where the module is crashing whenever you see the stack trace
 dropped. The same procedure applies to crashes caused by exceptions.
 
-    Note: To decode the exact line of code where the application
+    Note, to decode the exact line of code where the application
     crashed, you need to use ESP Exception Decoder in context of sketch
     you have just loaded to the module for diagnosis. Decoder is not
     able to correctly decode the stack trace dropped by some other
     application not compiled and loaded from your Arduino IDE.
 
 
+Improving Exception Decoder Results
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Due to the limited resources on the device, our default compiler optimizations
+focus on creating the smallest code size (``.bin`` file). The GCC compiler's
+option ``-Os`` contains the base set of optimizations used. This set is fine for
+release but not ideal for debugging.
+
+Our view of a crash is often the `Stack Dump <../Troubleshooting/stack_dump.rst>`__
+which gets copy/pasted into an Exception Decoder.
+For some situations, the optimizer doesn't write caller return addresses to the
+stack. When we crash, the list of functions called is missing. And when the
+crash occurs in a leaf function, there is seldom if ever any evidence of who
+called.
+
+With the ``-Os`` option, functions called once are inlined into the calling
+function. A chain of these functions can optimize down to the calling function.
+When the crash occurs in one of these chain functions, the actual location in
+the source code is no longer available.
+
+When you select ``Debug Optimization: Lite`` on the Arduino IDE Tools menu, it
+turns off ``optimize-sibling-calls``. Turning off this optimization allows more
+caller addresses to be written to the stack, improving the results from the
+Exception Decoder. Without this option, the callers involved in the crash may be
+missing from the Decoder results. Because of the limited stack space, there is
+the remote possibility that removing this optimization could lead to more
+frequent stack overflows. You only want to do this in a debug setting. This
+option does not help the chained function issue.
+
+When you select ``Debug Optimization: Optimum``, you get an even more complete
+stack trace. For example, chained function calls may show up. This selection
+uses the compiler option ``-Og``. GCC considers this the ideal optimization for
+the "edit-compile-debug cycle" ... "producing debuggable code." You can read the
+specifics at `GCC's Optimize Options <https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html>`__
+
+When global optimization creates build size issues or stack overflow issues,
+select ``Debug Optimization: None``, and use a targeted approach with
+``#pragma GCC optimize("Og")`` at the module level. Or, if you want to use a
+different set of optimizations, you can set optimizations through build options.
+Read more at `Global Build Options <a06-global-build-options.rst>`__.
+
+For non-Arduino IDE build platforms, you may need to research how to add build
+options. Some build platforms already use ``-Og`` for debug builds.
+
+A crash in a leaf function may not leave the caller's address on the stack.
+The return address can stay in a register for the duration of the call.
+Resulting in a crash report identifying the crashing function without a
+trace of who called. You can encourage the compiler to save the caller's
+return address by adding an inline assembly trick
+``__asm__ __volatile__("" ::: "a0", "memory");`` at the beginning of the
+function's body. Or instead, for a debug build conditional option, use the
+macro ``DEBUG_LEAF_FUNCTION()`` from ``#include <debug.h>``. For compiler
+toolchain 3.2.0 and above, the ``-Og`` option is an alternative solution.
+
+In some cases, adding ``#pragma GCC optimize("Og,no-ipa-pure-const")`` to a
+module as well as using ``DEBUG_LEAF_FUNCTION()`` in a leaf function were
+needed to display a complete call chain. Or use
+``#pragma GCC optimize("Os,no-inline,no-optimize-sibling-calls,no-ipa-pure-const")``
+if you require optimization ``-Os``.
+
+
 Other Causes for Crashes
 ~~~~~~~~~~~~~~~~~~~~~~~~
 
 Interrupt Service Routines
-   By default, all functions are compiled into flash, which means that the 
-   cache may kick in for that code. However, the cache currently can't be used 
-   during hardware interrupts. That means that, if you use a hardware ISR, such as 
-   attachInterrupt(gpio, myISR, CHANGE) for a GPIO change, the ISR must have the 
-   IRAM_ATTR attribute declared. Not only that, but the entire function tree 
+   By default, all functions are compiled into flash, which means that the
+   cache may kick in for that code. However, the cache currently can't be used
+   during hardware interrupts. That means that, if you use a hardware ISR, such as
+   attachInterrupt(gpio, myISR, CHANGE) for a GPIO change, the ISR must have the
+   IRAM_ATTR attribute declared. Not only that, but the entire function tree
    called from the ISR must also have the IRAM_ATTR declared.
    Be aware that every function that has this attribute reduces available memory.
 
-   In addition, it is not possible to execute delay() or yield() from an ISR, 
+   In addition, it is not possible to execute delay() or yield() from an ISR,
    or do blocking operations, or operations that disable the interrupts, e.g.: read
    a DHT.
 
    Finally, an ISR has very high restrictions on timing for the executed code, meaning
-   that executed code should not take longer than a very few microseconds. It is 
+   that executed code should not take longer than a very few microseconds. It is
    considered best practice to set a flag within the ISR, and then from within the loop()
    check and clear that flag, and execute code.
 
@@ -303,16 +366,16 @@ Asynchronous Callbacks
    than ISRs, but some restrictions still apply.
    It is not possible to execute delay() or yield() from an asynchronous callback.
    Timing is not as tight as an ISR, but it should remain below a few milliseconds. This
-   is a guideline. The hard timing requirements depend on the WiFi configuration and 
+   is a guideline. The hard timing requirements depend on the WiFi configuration and
    amount of traffic. In general, the CPU must not be hogged by the user code, as the
    longer it is away from servicing the WiFi stack, the more likely that memory corruption
    can happen.
 
 Memory, memory, memory
    Running out of heap is the **most common cause for crashes**. Because the build process for
    the ESP leaves out exceptions (they use memory), memory allocations that fail will do
-   so silently. A typical example is when setting or concatenating a large String. If 
-   allocation has failed internally, then the internal string copy can corrupt data, and 
+   so silently. A typical example is when setting or concatenating a large String. If
+   allocation has failed internally, then the internal string copy can corrupt data, and
    the ESP will crash.
 
    In addition, doing many String concatenations in sequence, e.g.: using operator+()
@@ -348,9 +411,9 @@ Memory, memory, memory
    * If you use std libs like std::vector, make sure to call its ::reserve() method before filling it. This allows allocating only once, which reduces mem fragmentation, and makes sure that there are no empty unused slots left over in the container at the end.
 
 Stack
-   The amount of stack in the ESP is tiny at only 4KB. For normal development in large systems, it 
+   The amount of stack in the ESP is tiny at only 4KB. For normal development in large systems, it
    is good practice to use and abuse the stack, because it is faster for allocation/deallocation, the scope of the object is well defined, and deallocation automatically happens in reverse order as allocation, which means no mem fragmentation. However, with the tiny amount of stack available in the ESP, that practice is not really viable, at least not for big objects.
-   
+
    * Large objects that have internally managed memory, such as String, std::string, std::vector, etc, are ok on the stack, because they internally allocate their buffers on the heap.
    * Large arrays on the stack, such as uint8_t buffer[2048] should be avoided on the stack and should be dynamically allocated instead (consider smart pointers).
    * Objects that have large data members, such as large arrays, should also be avoided on the stack, and should be dynamically allocated (consider smart pointers).
@@ -392,7 +455,7 @@ or `esp8266 / Arduino <https://github.com/esp8266/Arduino>`__ core,
 types and versions of O/S, you need to provide exact information on what
 your application is about. Only then, people willing to look into your
 issue may be able to compare it to a configuration they are familiar with.
-If you are lucky, they may even attempt to reproduce your issue on their 
+If you are lucky, they may even attempt to reproduce your issue on their
 own equipment!
 This will be far more difficult if you provide only vague details,
 so somebody would need to ask you to find out what is really happening.

doc/faq/a06-global-build-options.rst

@@ -181,10 +181,10 @@ their builds.
 
 There are two solutions to this issue:
 
-1. Turn off the “Aggressively Cache Compiled core” feature, by setting
+1. Do nothing, and rely on aggressive cache workaround built into the
+   script.
+2. Turn off the “Aggressively Cache Compiled core” feature, by setting
    ``compiler.cache_core=false``.
-2. Rely on the not ideal fail-safe, aggressive cache workaround built
-   into the script.
 
 Using “compiler.cache_core=false”
 ---------------------------------
@@ -253,14 +253,10 @@ problem would be cleared after a reboot. Or you can manually cleanup the
 
 **Arduino command-line option overrides**
 
-The script needs to know the working value of ``compiler.cache_core``
-that the Arduino IDE uses when building. This script can learn the state
-through documented locations; however, the Arduino IDE has two
-command-line options that can alter the results the Arduino IDE uses
-internally. And, the Arduino IDE does not provide a means for a script
-to learn the override value.
+If you are building with ``compiler.cache_core=true`` no action is
+needed. If ``false`` the script would benefit by knowing that.
 
-These two command-line options are the problem:
+When using either of these two command-line options:
 
 ::
 

doc/ideoptions.rst

@@ -127,7 +127,31 @@ There are a number of options.
 - The last (``NoAssert - NDEBUG``) is even quieter than the first (some
   internal guards are skipped to save more flash).
 - The other ones may be used when asked by a maintainer or if you are a
-  developper trying to debug some issues.
+  developer trying to debug some issues.
+
+Debug Optimization
+~~~~~~~~~~~~~~~~~~
+
+Due to the limited resources on the device, our default compiler optimizations
+focus on creating the smallest code size (``.bin`` file). That is fine for
+release but not ideal for debugging.
+
+``Debug Optimization`` use to improve Exception Decoder results.
+
+- ``Lite`` impact on code size uses ``-fno-optimize-sibling-calls`` to alter
+  the ``-Os`` compiler option to place more caller addresses on the Stack.
+- ``Optimum`` offers better quality stack content for the Exception Decoder at
+  the expense of a larger code size. It uses the ``-Og`` compiler option, which
+  turns off optimizations that can make debugging difficult while keeping
+  others.
+- ``None`` no changes for debugging continue using ``-Os``.
+
+Take note some sketches may start working after changing the optimization. Or
+fail less often. And it is also possible (not likely) that source code that
+was working with ``-Os`` may break with ``-Og``.
+
+For more topic depth, read `Improving Exception Decoder Results <faq/a02-my-esp-crashes.rst#improving-exception-decoder-results>`__
+
 
 lwIP variant
 ~~~~~~~~~~~~

libraries/esp8266/examples/GlobalBuildOptions/GlobalBuildOptions.ino.globals.h

@@ -8,7 +8,7 @@
  * this line is ignored
   *@create-file:build.opt@
   # this line is ignored
-  -O3
+  -Og
   // -fanalyzer
   -DUMM_STATS_FULL=1
 */