Merge tag 'mega-20191208' into 3WV

Release mega-20191208

Changes in mega-20191208 (since mega-20191130):

Gijs Noorlander (17):
      [WiFi] Use last known BSSID & channel from RTC + MQTT fixes
      [PIO] Update to core 2.6.2
      [WiFi] Improve ESP32 WiFi connect + fix mDNS updates
      [PIO] Fix build failure due to incorrect flags.
      [PIO] Don't use Python 3 specific since Travis still uses Python 2.7
      [WiFi] Call WiFi disconnect in setup() (letscontrolit#2757)
      [Rules] Parse template for all command calls
      [Rules] Execute some events asynchronous
      [Commands] Proper error logs when processing commands.
      [Commands] Add flag to be a bit more tolerant in last argument parsing
      [Events] Run events from rules immediately + add AsyncEvent command
      [FEATURE_SD] Add build in Platformio.ini with SD enabled (letscontrolit#2700 )
      [Rules] Add some checks for rules consistency before saving
      [Rules] Add check for correct received POST data when saving rules
      [JSON] Fix issues with generated JSON
      [JSON] Fix non valid timing stats json when no plugins defined (letscontrolit#2767)
      [JSON] Extend timing stats JSON end point with controller and misc stats

TD-er (1):
      automatically updated release notes for mega-20191208

stefan (1):
      Fix Arduino IDE include path

.gitignore

@@ -13,6 +13,7 @@
 .gcc-flags.json
 .piolibdeps
 pre_extra_script.py
+*.code-workspace
 
 ## Project #####################
 lib/readme.txt
@@ -40,3 +41,7 @@ tools/vagrant/.vagrant/
 tools/vagrant/build/
 
 tools/vagrant/Custom.h
+
+.buildcache/
+
+tools/vagrant/pio_envlist.txt

.travis.yml

@@ -33,31 +33,41 @@ script:
   # - cppcheck --enable=warning src/*.ino -q --force -I src --include=src/ESPEasy.ino --error-exitcode=1
   # - ./memanalyzer.py ~/.platformio/packages/toolchain-xtensa/bin/xtensa-lx106-elf-objdump
   # - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run --target clean
+  # Same list as in tools/build_ESPeasy.sh  and before_deploy
+  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e custom_ESP32_4M316k
   - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e custom_ESP8266_4M1M
-  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e esp-wrover-kit_test_1M8_partition
-  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e esp32test_1M8_partition
+  #- PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e custom_ESP8266_4M2M
+  #- PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e custom_beta_ESP8266_4M1M
+  #- PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e custom_sdk3_ESP8266_4M1M
+  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e dev_ESP8266_4M1M
+  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e hard_SONOFF_POW_4M1M
+  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e hard_Shelly_1_2M256
+  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e hard_Ventus_W266
+  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e hard_other_POW_ESP8285_1M
+  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e minimal_IRext_ESP8266_1M
+  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e minimal_IRext_ESP8266_4M1M
+  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e minimal_IRext_ESP8266_4M2M
+  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e minimal_core_242_ESP8266_1M_OTA
+  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e minimal_core_242_ESP8285_1M_OTA
+  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e minimal_core_261_ESP8266_1M_OTA
+  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e minimal_core_261_ESP8285_1M_OTA
+  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e minimal_core_261_sdk3_ESP8266_1M_OTA
+  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e minimal_core_261_sdk3_ESP8285_1M_OTA
+  #- PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e normal_ESP8266_16M
   - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e normal_ESP8266_1M
   - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e normal_ESP8266_1M_VCC
-  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e normal_ESP8285_1M
-  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e normal_core_241_ESP8266_1M
-  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e normal_WROOM02_2M
-  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e normal_core_252_WROOM02_2M256
   - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e normal_ESP8266_4M1M
-  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e normal_core_241_ESP8266_4M1M
-  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e normal_core_260_sdk222_alpha_ESP8266_4M1M
-  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e minimal_core_242_ESP8266_1M_OTA
-  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e minimal_core_242_ESP8285_1M_OTA
-  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e minimal_core_252_ESP8266_1M_OTA
-  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e minimal_core_252_ESP8285_1M_OTA
-  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e minimal_IRext_ESP8266_1M
-  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e minimal_IRext_ESP8266_4M2M
+  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e normal_ESP8285_1M
   - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e normal_IRext_no_rx_ESP8266_4M2M
-  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e test_core_260_sdk222_alpha_ESP8266_4M1M
-  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e test_core_260_sdk3_alpha_ESP8266_4M1M
-  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e test_ESP8266_4M_VCC
-  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e dev_ESP8266_4M1M
-  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e hard_SONOFF_POW_4M1M
-
+  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e normal_WROOM02_2M
+  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e normal_WROOM02_2M256
+  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e normal_sdk3_ESP8266_1M
+  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e test_ESP32-wrover-kit_4M316k
+  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e test_ESP32_4M316k
+  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e test_ESP8266_4M1M_VCC
+  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e test_ESP8266_4M1M_VCC_MDNS_SD
+  #- PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e test_beta_ESP8266_16M
+  - PLATFORMIO_BUILD_FLAGS="-D CONTINUOUS_INTEGRATION" platformio run -e test_beta_ESP8266_4M1M
 
 before_deploy:
   - ./before_deploy

before_deploy

@@ -39,54 +39,27 @@ fi
 
 mkdir -p ${TMP_DIST}/memstats
 
-for ENV in \
-  custom_ESP8266_4M1M\
-  esp-wrover-kit_test_1M8_partition\
-  esp32test_1M8_partition\
-  normal_ESP8266_1M\
-  normal_ESP8266_1M_VCC\
-  normal_ESP8285_1M\
-  normal_core_241_ESP8266_1M\
-  normal_WROOM02_2M\
-  normal_core_252_WROOM02_2M256\
-  normal_ESP8266_4M1M\
-  normal_core_241_ESP8266_4M1M\
-  normal_core_252_ESP8266_16M\
-  normal_core_260_sdk222_alpha_ESP8266_4M1M\
-  normal_core_260_sdk222_alpha_ESP8266_16M\
-  minimal_core_242_ESP8266_1M_OTA\
-  minimal_core_242_ESP8285_1M_OTA\
-  minimal_core_252_ESP8266_1M_OTA\
-  minimal_core_252_ESP8285_1M_OTA\
-  minimal_IRext_ESP8266_1M\
-  minimal_IRext_ESP8266_4M1M\
-  minimal_IRext_ESP8266_4M2M\
-  normal_IRext_no_rx_ESP8266_4M2M\
-  test_core_260_sdk222_alpha_ESP8266_4M1M\
-  test_core_260_sdk3_alpha_ESP8266_4M1M\
-  test_core_260_sdk222_alpha_ESP8266_16M\
-  test_ESP8266_4M_VCC\
-  dev_ESP8266_4M1M\
-  hard_SONOFF_POW_4M1M\
-  hard_other_POW_ESP8285_1M\
-  hard_Shelly_1_2M256\
-  hard_Ventus_W266;\
-do
+# Must look into all possible env definitions.
+# Exclude so called "spec_" (special) builds
+for ENV in `grep "^\[env:" platformio*.ini |cut -d'[' -f2|cut -d']' -f1|cut -d':' -f2|sort -n|grep -v spec_`;
+do 
   MAX_FILESIZE=1044464
   if [[ ${ENV} == *"_1M"* ]]; then
-    # max 871 kiB
+    # max 872 kiB - 16 bytes
     MAX_FILESIZE=892912
   fi
   if [[ ${ENV} == *"_1M_OTA"* ]]; then
-    # max 602 kiB
-    MAX_FILESIZE=616448
-  fi
-  if [[ ${ENV} == *"esp32"* ]]; then
-    MAX_FILESIZE=1310720
+    # max 600 kiB - 16 bytes
+    MAX_FILESIZE=614384
   fi
-  if [[ ${ENV} == *"1M8_partition"* ]]; then
+  if [[ ${ENV} == *"4M316k"* ]]; then
+    # ESP32 with 1800k of sketch space.
     MAX_FILESIZE=1900544
   fi
+  if [[ ${ENV} == *"debug_"* ]]; then
+    # Debug env, used for analysis, not to be run on a node.
+    MAX_FILESIZE=0
+  fi
 
   FIRMWARE_BIN=`echo "${BINARY_PATH}/${ENV}/firmware.bin"`
 
@@ -122,7 +95,7 @@ done
 mkdir -p ${TMP_DIST}/source
 cp -r lib ${TMP_DIST}/source/
 cp -r src ${TMP_DIST}/source/
-cp platformio.ini ${TMP_DIST}/source/
+cp platformio*.ini ${TMP_DIST}/source/
 
 cd ${TMP_DIST}
 

dist/README.txt

@@ -23,11 +23,8 @@ There is also a number of special builds:
 - normal_IR => "Normal" + IR receiver/transmitter plugins and library
 - hard_xxxxx => Special builds for some off-the-shelf hardware.
 - minimal_ESP82xx_1M_OTA => Minimum number of plugins and a limited set of controllers included to be able to perform a 2-step OTA on 1 MB flash nodes.
-- normal_core_241 => "Normal" using core 2.4.1, since 2.4.2 has issues with PWM
-- xxx_core_260_sdk2_alpha -> core 2.6.0 alpha version (under development) using SDK 2.2.1
-- xxx_core_260_sdk222_alpha -> core 2.6.0 alpha version (under development) using SDK 2.2.2
-- xxx_core_260_sdk3_alpha -> core 2.6.0 alpha version (under development) using SDK 3.0.0-dev (under development too)
-
+- normal_core_xxx => "Normal" using core xxx (e.g. 2.4.1)
+- normal_beta => "Normal" using the staged (beta) branch of the esp8266/Arduino repository.
 
 Chip can be:
 - ESP8266      => Most likely option
@@ -49,13 +46,14 @@ If these speed issues will be fixed, it is very likely the SPIFFS must then be r
 
 Special memory partitioning:
 - 2M256  => 2 MB flash modules (e.g. Shelly1/WROOM02) with 256k SPIFFS (only core 2.5.0 or newer)
-- 1M8_partition => For ESP32 with 4MB flash, sketch size is set to 1.8 MByte (default: 1.4 MByte)
+- 4M316k => For ESP32 with 4MB flash, sketch size is set to 1.8 MByte (default: 1.4 MByte)
+- 4M1M   => 4MB flash, 1 MB SPIFFS. Default layout for 4MB flash.
+- 4M2M   => 4MB flash, 2 MB SPIFFS. Introduced in October 2019. Only possible with core 2.5.2 or newer.
 
 
 ESP32 now has 3 builds:
-- esp32dev   Using the default partition layout (1.4 MB for the sketch)
-- esp32test_1M8_partition   Larger sketch partition (1.8MB) smaller SPIFFS (316 kB)
-- esp-wrover-kit_test_1M8_partition  A build for ESP32 including build flags for the official WRover test kit.
+- esp32test_4M316k   Larger sketch partition (1.8MB) smaller SPIFFS (316 kB)
+- test_ESP32-wrover-kit_4M316k  A build for ESP32 including build flags for the official WRover test kit.
 
 Please note that changing between those versions will destroy the settings!
 The SPIFFS partition will be lost and that contains all settings.