Testers for new upcoming build end September '23 #4825

TD-er · 2023-09-25T09:21:10Z

There has been a major overhaul in code to improve code structure in ESPEasy.

Split ESPEasy scheduler code to make it easier to read/extend. (needed for future changes in how tasks will read on a shared bus)
Restructure how deviceIndex, PluginID and Devices array interact for Plugins
Restructure how protocolIndex, CPluginID and Controllers array interact for CPlugins (Controllers)
Restructure how notifierIndex, NPluginID and Notifier array interact for NPlugins (Notifications)
Reduction of binary size

All-in-all this was about 60 commits and 235 files.
So I hope I did fix all the bugs that may have introduced by this massive overhaul.
I also fixed a number of small bugs along the way, but I don't know what side effects those fixes may have on other parts of the code. (that's why the restructure is needed, to minimize unexpected side effects in future commits)

It would be nice if the current 'mega' test builds could be tested before the new official build.
I was planning to make this build in this month, so we can have a September build.

Maybe the usual 'suspects' like @fly74 , @iz8mbw , @ghtester (and anyone else) can test this latest GH Actions test build or use the latest webflasher ?

N.B. I uploaded the last test build including Ton's PR for the ESP32 internal temperature sensor (#4820) to the web flasher.

Next on the agenda for the following build after this one:

WiFi/Ethernet should have a clear intended state (set by the user) not trying to guess what may be a good idea based on events and assumptions.
Update ESP32 SDK to latest (last?) 4.4.x as it also has a few important fixes regarding WiFi, streams and re-assignment of serial ports.
Change pending PR for MQTT TLS to no longer use mbedTLS as that's going to be removed on Arduino 3.0.0 for ESP32, but use BearSSL. + merge that PR.

fly74 · 2023-09-25T09:38:23Z

@TD-er Flashing....done. Testing in progress....

iz8mbw · 2023-09-25T10:14:05Z

Flashed, until now all OK. I'll update you after,

fly74 · 2023-09-25T10:37:21Z

@TD-er looks good with these:

├───ESP32-D0WDQ6_neopixel_ESP32_4M316k
│       ESP_Easy_mega_20230924_neopixel_ESP32_4M316k.bin
│
├───ESP32-D0WDQ6_normal_ESP32_4M316k
│       ESP_Easy_mega_20230924_normal_ESP32_4M316k.bin
│
├───ESP8266_climate_ESP8266_4M1M
│       ESP_Easy_mega_20230924_climate_ESP8266_4M1M.bin
│
├───ESP8266_neopixel_ESP8266_4M1M
│       ESP_Easy_mega_20230924_neopixel_ESP8266_4M1M.bin
│
└───ESP8266_normal_ESP8266_4M1M
        ESP_Easy_mega_20230924_normal_ESP8266_4M1M.bin

TD-er · 2023-09-25T11:04:21Z

@fly74 p2p is also working fine?

iz8mbw · 2023-09-25T11:58:37Z

All OK here on ESP32 boards using Climate, Energy and MAX.
p2p also OK.

fly74 · 2023-09-25T11:59:44Z

@fly74 p2p is also working fine?

P2p also ok!

fly74 · 2023-09-25T12:04:22Z

N.B. I uploaded the last test build including Ton's PR for the ESP32 internal temperature sensor (#4820) to the web flasher.

Can't see the variable in sysvars :/

tonhuisman · 2023-09-25T12:14:30Z

Only on ESP32 builds...
It's in the System group

fly74 · 2023-09-25T12:15:56Z

It is:

Firmware

TD-er · 2023-09-25T12:56:34Z

Git HEAD:⋄ | mega_3409e9e

That looks like it is a build from the 'mega' branch, not Ton's pull request.
The GH Actions build for that pull request is here: https://github.com/letscontrolit/ESPEasy/actions/runs/6291558098

TD-er · 2023-09-25T13:06:00Z

I just installed that build on an ESP32 node here in my network.

I thought the git branch name would be shown there, as it does when I build my own.
So that's something I need to look into as that's extremely useful to have.
No idea yet why it isn't showing on GH Actions builds.

fly74 · 2023-09-25T14:01:09Z

Git HEAD:⋄ | mega_3409e9e

That looks like it is a build from the 'mega' branch, not Ton's pull request. The GH Actions build for that pull request is here: https://github.com/letscontrolit/ESPEasy/actions/runs/6291558098

Ah, I thought it was in your release.

TD-er · 2023-09-25T14:08:31Z

Not yet as there are still things to 'fix' with the internal temperature sensor on ESP32-classic.
I tested it on a node here and it does show the error value (0 Kelvin) quite often.
So I already discussed with Ton what would be a good approach to prevent those obviously wrong values.

Keep in mind the temperature sensor on the ESP32 is absolutely not accurate and not factory calibrated. (ESP32-S2/S3/C3/etc do seem to have a factory calibration table present in each chip)
On my test node it does look quite 'believable' as in it is about 12 degree C warmer compared to the DS18b20 next to it (open PCB, not in an enclosure) and it seems to get along with the 1wire sensor regarding temperature changes.

chromoxdor · 2023-09-26T12:51:25Z

So far everything looks fine here. But i only flashed two devices for now (I am hesitant to update my critical infrastructure 🙂)
I assume WiFiAPMode and "Settings to keep" in "Factory Reset" is still WIP?

fly74 · 2023-09-26T12:53:46Z

I am hesitant to update my critical infrastructure 🙂

You must be brave like a lion.

fly74 · 2023-09-26T12:55:26Z

My critical infrastructure ist patched without any issues :)

TD-er · 2023-09-26T12:56:16Z

I assume WiFiAPMode and "Settings to keep" in "Factory Reset" is still WIP?

Yep, still a WIP

However some fixes have already been merged. But not yet all.

chromoxdor · 2023-09-26T12:57:31Z

You must be brave like a lion.

Thursday my parents in law are leaving and than the kitten can become a lion again...

fly74 · 2023-09-26T12:58:01Z

What is WIP ? Work in progess?

chromoxdor · 2023-09-26T12:58:22Z

What is WIP ? Work in progess?

Yes

TD-er · 2023-09-26T12:59:39Z

Most important acronyms:

WIP : Work-in-progress
WAF : Wife-Acceptance-Factor (very important!)

fly74 · 2023-09-26T13:00:07Z

Most important acronyms:

WIP : Work-in-progress

WAF : Wife-Acceptance-Factor (very important!)

The second is well known to me :/

ghtester · 2023-09-26T14:32:56Z

Well, I am going to compile a fresh custom build and upgrade at least one node with encountered hang with the one month old firmware release (see the picture).
I am still experiencing some random memory leaks, as the ESP nodes are configured differently, it's hard to find the reason.
As outlined on the picture, the FW releases used are from 20221108 to 20230825. BTW. the white parts of the graph are due to target MQTT broker outages.

TD-er · 2023-09-26T15:33:02Z

Are those ESP's sending their logs to a syslog server?

ghtester · 2023-09-26T21:02:56Z

No, as outlined above, it's sent through MQTT protocol to Home Assistant, not a log but just the variable values ( %sysheap% and %uptime% converted to days).

TD-er · 2023-09-26T21:40:02Z

The reason I'm asking is because there is a known issue when sending to a syslog server with flaky WiFi connection.

ghtester · 2023-09-27T04:54:37Z

I understand but currently I don't use any syslog server so the Syslog Log Level: is set to None on all of my ESP nodes.

TD-er · 2023-09-27T06:22:22Z

Does that unit have serial log level set to anything other than 'none' ?
Do you send sensor data via C013 to other p2p nodes?
Do you have any other UDP related traffic to/from that node?
Does that node have NTP enabled, or does it receive the time via the ESPEasy p2p layer?

(see any pattern here in 4 out of 5 options I mentioned? ;) )

iz8mbw · 2023-09-27T20:00:38Z

IMHO since the issue reported by @ghtester is not related to the specific "September 2023" future build since the memory leak issue reported is born in some times between the end 2022 and 2023, it should be covered on a separate GitHub issue.

TD-er · 2023-09-27T20:18:06Z

Yep and it isn't a "show stopper" for this release.

But it might be a good idea for the "regular testers" to keep in mind that this might be an issue. Maybe one of you here may have an idea on what is causing this.... and we can fix it for the October build :)

iz8mbw · 2023-09-27T20:39:46Z

Can this memory leak be caused by a device?
Well, we don't know which devices are connected to the board affected by the memory leak.
This is just an example that all of us needs more info to try to have ideas.

TD-er · 2023-09-27T21:28:01Z

More likely some service (like mDNS) or received but unprocessed traffic (e.g. UDP packets)
When it is about unprocessed packets, it could be related to some plugin which for whatever reason takes more time to process stuff.
But you have to keep in mind that memory allocation takes more time when memory gets more fragmented. -> receive more unprocessed packets -> more memory fragmentation and less room for allocation.
So it could be some snowball effect which is really hard to reproduce.

Micha-he · 2023-09-28T05:44:28Z

I've tested the latest code on my humidity switch (custom 2M256).

i2c scanner, AM2320-plugin and my rules are working flawless, no other abnormalities found on this system.

TD-er · 2023-09-28T06:46:49Z

OK, so we only need to merge Ton's RTTTL fix and then make a build.

TD-er · 2023-09-28T07:34:04Z

Found another issue regarding check for valid GPIO...
So this one will also be included in the upcoming build: #4830

ghtester · 2023-09-28T10:15:11Z

Does that unit have serial log level set to anything other than 'none' ? Do you send sensor data via C013 to other p2p nodes?

All my units with August FW release have the serial log level set to None. The unit which hanged is using the Communication - Serial Proxy plugin through HW Serial 0 (D9/D10 pins). Yesterday updated with fresh FW custom release 20230926, no issues so far. I don't use C013 controller so there's no p2p communication.

Do you have any other UDP related traffic to/from that node?

No / I don't know about it, the ESPEasy p2p UDP port: set to 0 on all my ESP nodes.

Does that node have NTP enabled, or does it receive the time via the ESPEasy p2p layer?

Yes the NTP server configured (with DNS name) on all my nodes.

BTW. it looks like cosmethics detail but I have encountered sometimes after Cold boot the Boot: reason on sysinfo page is not Cold boot but something else (don't remember what exactly it was). Also sometimes after FW upgrade the node does not reboot (Soft Reboot) but hanges for while until rebooted with External Watchdog (as sysinfo says). But I don't think it's an issue.

fly74 · 2023-09-28T19:44:29Z

Only on ESP32 builds... It's in the System group

@tonhuisman Looks good on ESP32-D0WDQ6-V3

TD-er · 2023-09-29T08:04:06Z

Just uploaded the currently latest merges to the mega branch to the web flasher:
https://td-er.nl/ESPEasy/latest/

I think this will probably be the same code as the next build, unless something really urgent will pop up.

fly74 · 2023-09-29T09:35:30Z

I need the zip Files for my batch updater.

tonhuisman · 2023-09-29T09:53:31Z

I need the zip Files for my batch updater.

Last Actions run for mega

TD-er · 2023-09-29T15:03:38Z

OK, breaking bug found....
Reading internal temperature on ESP32-S2/S3/C3 can result in a crash and this is a bug in the ESP-IDF SDK.
It seems to be fixed in the upcoming IDF 5.1, but we're still on < 5.0 as it requires quite some code changes to support IDF 5.x

So I will make a work-around.

TD-er · 2023-09-29T21:26:16Z

Uploaded the latest fix for the internal temperature sensor to the 'latest' web flasher: https://td-er.nl/ESPEasy/latest/

I used these bin files: https://github.com/letscontrolit/ESPEasy/actions/runs/6356259841

PR for this fix: #4834

If there are no critical breaking changes, I would like to make a new official build tomorrow.

iz8mbw · 2023-09-30T06:01:57Z

All OK here with "HEAD_fd914cd" on ESP32 boards using Climate, Energy and MAX.

TD-er · 2023-09-30T07:55:25Z

@iz8mbw Did you happen to test ESP32-S2 as well?
I have ESP32, ESP32-C3 and ESP32-S3 running over night with the temperature being measured every second and those are still running fine now.

iz8mbw · 2023-09-30T08:01:45Z

I have only ESP32 boards (4MB and 16MB) and on these boards all is OK.
Maybe I discovered a general more CPU usage since recent commits, but for the rest is OK.

TD-er · 2023-09-30T08:13:23Z

Depending on how recent...
Major changes since the last official build:

Scheduler
Plugin stats now store upto 256 samples per task value for which it is enabled
Internal temperature is measured every second
Some extra checks have been added (but others have been removed)

What I did notice was that on some ESP32 boards, the WiFi may take a bit longer to get a good WiFi. At least when comparing to a April or June build.
Somewhere between April and June builds the memory usage has grown, but that's also when I changed the console stuff and added some extra buffering for serial I/O.

Next on the agenda for the following build:

WiFi/Ethernet should have a clear intended state (set by the user) not trying to guess what may be a good idea based on events and assumptions.
Update ESP32 SDK to latest (last?) 4.4.x as it also has a few important fixes regarding WiFi, streams and re-assignment of serial ports.
Change pending PR for MQTT TLS to no longer use mbedTLS as that's going to be removed on Arduino 3.0.0 for ESP32, but use BearSSL. + merge that PR.

TD-er · 2023-09-30T14:44:08Z

And new build has been made, so this can be closed.

Thanks for testing :)

TD-er added the Category: Build Related to building/IDE/releases label Sep 25, 2023

TD-er pinned this issue Sep 25, 2023

TD-er closed this as completed Sep 30, 2023

TD-er unpinned this issue Sep 30, 2023

Testers for new upcoming build end September '23 #4825

Testers for new upcoming build end September '23 #4825

Comments

TD-er commented Sep 25, 2023 • edited

fly74 commented Sep 25, 2023

iz8mbw commented Sep 25, 2023

fly74 commented Sep 25, 2023 • edited

TD-er commented Sep 25, 2023

iz8mbw commented Sep 25, 2023

fly74 commented Sep 25, 2023

fly74 commented Sep 25, 2023

tonhuisman commented Sep 25, 2023 • edited

fly74 commented Sep 25, 2023

Firmware

TD-er commented Sep 25, 2023 • edited

TD-er commented Sep 25, 2023

fly74 commented Sep 25, 2023

TD-er commented Sep 25, 2023

chromoxdor commented Sep 26, 2023

fly74 commented Sep 26, 2023

fly74 commented Sep 26, 2023

TD-er commented Sep 26, 2023 • edited

chromoxdor commented Sep 26, 2023

fly74 commented Sep 26, 2023

chromoxdor commented Sep 26, 2023

TD-er commented Sep 26, 2023

fly74 commented Sep 26, 2023

ghtester commented Sep 26, 2023 • edited

TD-er commented Sep 26, 2023

ghtester commented Sep 26, 2023

TD-er commented Sep 26, 2023

ghtester commented Sep 27, 2023

TD-er commented Sep 27, 2023

iz8mbw commented Sep 27, 2023

TD-er commented Sep 27, 2023 • edited

iz8mbw commented Sep 27, 2023

TD-er commented Sep 27, 2023

Micha-he commented Sep 28, 2023

TD-er commented Sep 28, 2023

TD-er commented Sep 28, 2023

ghtester commented Sep 28, 2023 • edited

fly74 commented Sep 28, 2023

TD-er commented Sep 29, 2023

fly74 commented Sep 29, 2023

tonhuisman commented Sep 29, 2023

TD-er commented Sep 29, 2023

TD-er commented Sep 29, 2023

iz8mbw commented Sep 30, 2023

TD-er commented Sep 30, 2023

iz8mbw commented Sep 30, 2023

TD-er commented Sep 30, 2023

TD-er commented Sep 30, 2023

TD-er commented Sep 25, 2023 •

edited

fly74 commented Sep 25, 2023 •

edited

tonhuisman commented Sep 25, 2023 •

edited

TD-er commented Sep 25, 2023 •

edited

TD-er commented Sep 26, 2023 •

edited

ghtester commented Sep 26, 2023 •

edited

TD-er commented Sep 27, 2023 •

edited

ghtester commented Sep 28, 2023 •

edited