Skip to content

Conversation

@lucasssvaz
Copy link
Member

@lucasssvaz lucasssvaz commented Oct 16, 2025

Description of Change

This pull request updates the continuous integration (CI) and test infrastructure to transition from using ci.json files to ci.yml files for configuration, and introduces new scripts for improved test reporting and missing test detection. The PR also include small bugfixes to the workflows and enable hardware tests on the ESP32-C5. The most important changes are grouped below:

Migration from ci.json to ci.yml for CI configuration

  • Updated all references to sketch and project build configuration files from ci.json to ci.yml in .github/CODEOWNERS, .github/scripts/get_affected.py, .github/scripts/sketch_utils.sh, and .github/scripts/on-push-idf.sh, ensuring the CI system now expects YAML configuration files for easier maintainability and readability. [1] [2] [3] [4] [5] [6] [7]

New and improved CI/test scripts

  • Added .github/scripts/generate_missing_junits.py to detect missing test runs based on build artifacts and generate placeholder JUnit XML files, improving visibility into incomplete test coverage.
  • Added .github/scripts/runtime_table_generator.py to generate a Markdown summary table of runtime test results, including pass/fail/error status for each platform and target, and saving results to a JSON file for future reference.

These changes modernize the CI configuration format and enhance test reporting and diagnostics, making it easier to identify missing or failing tests and improving overall maintainability.

Other Changes

  • Enabled ESP32-C5 in hardware tests
  • Fixed bugs in workflows

Test Scenarios

Tested in fork

@lucasssvaz lucasssvaz self-assigned this Oct 16, 2025
@lucasssvaz lucasssvaz requested a review from a team as a code owner October 16, 2025 12:36
@lucasssvaz lucasssvaz added the Type: CI & Testing Related to continuous integration, automated testing, or test infrastructure. label Oct 16, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Oct 16, 2025

Messages
📖 This PR seems to be quite large (total lines of code: 4093), you might consider splitting it into smaller PRs

👋 Hello lucasssvaz, we appreciate your contribution to this project!


📘 Please review the project's Contributions Guide for key guidelines on code, documentation, testing, and more.

🖊️ Please also make sure you have read and signed the Contributor License Agreement for this project.

Click to see more instructions ...


This automated output is generated by the PR linter DangerJS, which checks if your Pull Request meets the project's requirements and helps you fix potential issues.

DangerJS is triggered with each push event to a Pull Request and modify the contents of this comment.

Please consider the following:
- Danger mainly focuses on the PR structure and formatting and can't understand the meaning behind your code or changes.
- Danger is not a substitute for human code reviews; it's still important to request a code review from your colleagues.
- Addressing info messages (📖) is strongly recommended; they're less critical but valuable.
- To manually retry these Danger checks, please navigate to the Actions tab and re-run last Danger workflow.

Review and merge process you can expect ...


We do welcome contributions in the form of bug reports, feature requests and pull requests.

1. An internal issue has been created for the PR, we assign it to the relevant engineer.
2. They review the PR and either approve it or ask you for changes or clarifications.
3. Once the GitHub PR is approved we do the final review, collect approvals from core owners and make sure all the automated tests are passing.
- At this point we may do some adjustments to the proposed change, or extend it by adding tests or documentation.
4. If the change is approved and passes the tests it is merged into the default branch.

Generated by 🚫 dangerJS against c2cdeb6

@github-actions
Copy link
Contributor

github-actions bot commented Oct 16, 2025

Test Results

0 tests   0 ✅  0s ⏱️
0 suites  0 💤
0 files    0 ❌

Results for commit c2cdeb6.

♻️ This comment has been updated with latest results.

@github-actions
Copy link
Contributor

github-actions bot commented Oct 16, 2025

Memory usage test (comparing PR against master branch)

The table below shows the summary of memory usage change (decrease - increase) in bytes and percentage for each target.

MemoryFLASH [bytes]FLASH [%]RAM [bytes]RAM [%]
TargetDECINCDECINCDECINCDECINC
ESP32C5000.000.00000.000.00
ESP32P4000.000.00000.000.00
ESP32S3💚 -1600.000.00000.000.00
ESP32S2000.000.00000.000.00
ESP32C3000.000.00000.000.00
ESP32C6000.000.00000.000.00
ESP32H2000.000.00000.000.00
ESP32000.000.00000.000.00
Click to expand the detailed deltas report [usage change in BYTES]
TargetESP32C5ESP32P4ESP32S3ESP32S2ESP32C3ESP32C6ESP32H2ESP32
ExampleFLASHRAMFLASHRAMFLASHRAMFLASHRAMFLASHRAMFLASHRAMFLASHRAMFLASHRAM
libraries/ArduinoOTA/examples/BasicOTA000000000000--00
libraries/AsyncUDP/examples/AsyncUDPClient000000000000--00
libraries/AsyncUDP/examples/AsyncUDPMulticastServer000000000000--00
libraries/AsyncUDP/examples/AsyncUDPServer000000000000--00
libraries/BLE/examples/Beacon_Scanner000000--00000000
libraries/BLE/examples/Client000000--00000000
libraries/BLE/examples/Client_secure_static_passkey000000--00000000
libraries/BLE/examples/EddystoneTLM_Beacon000000--00000000
libraries/BLE/examples/EddystoneURL_Beacon000000--00000000
libraries/BLE/examples/Notify000000--00000000
libraries/BLE/examples/Scan000000--00000000
libraries/BLE/examples/Server000000--00000000
libraries/BLE/examples/Server_multiconnect000000--00000000
libraries/BLE/examples/Server_secure_authorization00--00--000000--
libraries/BLE/examples/Server_secure_static_passkey000000--00000000
libraries/BLE/examples/UART000000--00000000
libraries/BLE/examples/Write000000--00000000
libraries/BLE/examples/iBeacon000000--00000000
libraries/DNSServer/examples/CaptivePortal000000000000--00
libraries/ESP32/examples/AnalogOut/LEDCGammaFade0000------0000--
libraries/ESP32/examples/DeepSleep/TimerWakeUp000000000000--00
libraries/ESP32/examples/HWCDC_Events000000--000000--
libraries/ESP32/examples/Time/SimpleTime000000000000--00
libraries/ESP_I2S/examples/ES8388_loopback0000000000000000
libraries/ESP_I2S/examples/Simple_tone0000000000000000
libraries/ESP_NOW/examples/ESP_NOW_Broadcast_Master00--00000000--00
libraries/ESP_NOW/examples/ESP_NOW_Broadcast_Slave00--00000000--00
libraries/ESP_NOW/examples/ESP_NOW_Network00--00000000--00
libraries/ESP_NOW/examples/ESP_NOW_Serial00--00000000--00
libraries/ESPmDNS/examples/mDNS-SD_Extended000000000000--00
libraries/ESPmDNS/examples/mDNS_Web_Server000000000000--00
libraries/Ethernet/examples/ETH_WIFI_BRIDGE000000000000--00
libraries/FFat/examples/FFat_time000000000000--00
libraries/HTTPClient/examples/Authorization000000000000--00
libraries/HTTPClient/examples/BasicHttpClient000000000000--00
libraries/HTTPClient/examples/BasicHttpsClient000000000000--00
libraries/HTTPClient/examples/CustomHeaders000000000000--00
libraries/HTTPClient/examples/HTTPClientEnterprise00--00000000--00
libraries/HTTPClient/examples/ReuseConnection000000000000--00
libraries/HTTPClient/examples/StreamHttpClient000000000000--00
libraries/HTTPUpdate/examples/httpUpdate000000000000--00
libraries/HTTPUpdate/examples/httpUpdateSPIFFS000000000000--00
libraries/HTTPUpdate/examples/httpUpdateSecure000000000000--00
libraries/HTTPUpdateServer/examples/WebUpdater000000000000--00
libraries/Insights/examples/DiagnosticsSmokeTest00--00000000--00
libraries/Insights/examples/MinimalDiagnostics00--00000000--00
libraries/LittleFS/examples/LITTLEFS_time000000000000--00
libraries/Matter/examples/MatterColorLight00--000000000000
libraries/Matter/examples/MatterCommissionTest00--000000000000
libraries/Matter/examples/MatterComposedLights00--000000000000
libraries/Matter/examples/MatterContactSensor00--000000000000
libraries/Matter/examples/MatterDimmableLight00--000000000000
libraries/Matter/examples/MatterEnhancedColorLight00--000000000000
libraries/Matter/examples/MatterEvents00--000000000000
libraries/Matter/examples/MatterFan00--000000000000
libraries/Matter/examples/MatterHumiditySensor00--000000000000
libraries/Matter/examples/MatterLambdaSingleCallbackManyEPs00--000000000000
libraries/Matter/examples/MatterMinimum00--000000000000
libraries/Matter/examples/MatterOccupancySensor00--000000000000
libraries/Matter/examples/MatterOnIdentify00--000000000000
libraries/Matter/examples/MatterOnOffLight00--000000000000
libraries/Matter/examples/MatterOnOffPlugin00--000000000000
libraries/Matter/examples/MatterPressureSensor00--000000000000
libraries/Matter/examples/MatterSmartButon00--000000000000
libraries/Matter/examples/MatterTemperatureLight00--000000000000
libraries/Matter/examples/MatterTemperatureSensor00--000000000000
libraries/Matter/examples/MatterThermostat00--000000000000
libraries/NetBIOS/examples/ESP_NBNST000000000000--00
libraries/NetworkClientSecure/examples/WiFiClientInsecure000000000000--00
libraries/NetworkClientSecure/examples/WiFiClientPSK000000000000--00
libraries/NetworkClientSecure/examples/WiFiClientSecure000000000000--00
libraries/NetworkClientSecure/examples/WiFiClientSecureEnterprise00--00000000--00
libraries/NetworkClientSecure/examples/WiFiClientSecureProtocolUpgrade000000000000--00
libraries/NetworkClientSecure/examples/WiFiClientShowPeerCredentials000000000000--00
libraries/NetworkClientSecure/examples/WiFiClientTrustOnFirstUse000000000000--00
libraries/OpenThread/examples/CLI/COAP/coap_lamp00--------0000--
libraries/OpenThread/examples/CLI/COAP/coap_switch00--------0000--
libraries/OpenThread/examples/CLI/SimpleCLI00--------0000--
libraries/OpenThread/examples/CLI/SimpleNode00--------0000--
libraries/OpenThread/examples/CLI/SimpleThreadNetwork/ExtendedRouterNode00--------0000--
libraries/OpenThread/examples/CLI/SimpleThreadNetwork/LeaderNode00--------0000--
libraries/OpenThread/examples/CLI/SimpleThreadNetwork/RouterNode00--------0000--
libraries/OpenThread/examples/CLI/ThreadScan00--------0000--
libraries/OpenThread/examples/CLI/onReceive00--------0000--
libraries/OpenThread/examples/Native/SimpleThreadNetwork/LeaderNode00--------0000--
libraries/OpenThread/examples/Native/SimpleThreadNetwork/RouterNode00--------0000--
libraries/PPP/examples/PPP_Basic0000000000000000
libraries/PPP/examples/PPP_WIFI_BRIDGE000000000000--00
libraries/RainMaker/examples/RMakerCustom00--00000000----
libraries/RainMaker/examples/RMakerCustomAirCooler00--00000000----
libraries/RainMaker/examples/RMakerSonoffDualR300--00000000----
libraries/RainMaker/examples/RMakerSwitch00--00000000----
libraries/SD/examples/SD_time000000000000--00
libraries/SPI/examples/SPI_Multiple_Buses0000000000000000
libraries/SPIFFS/examples/SPIFFS_time000000000000--00
libraries/Update/examples/AWS_S3_OTA_Update000000000000--00
libraries/Update/examples/HTTPS_OTA_Update000000000000--00
libraries/Update/examples/HTTP_Client_AES_OTA_Update000000000000--00
libraries/Update/examples/HTTP_Server_AES_OTA_Update000000000000--00
libraries/Update/examples/OTAWebUpdater000000000000--00
libraries/WebServer/examples/AdvancedWebServer000000000000--00
libraries/WebServer/examples/ChunkWriting000000000000--00
libraries/WebServer/examples/FSBrowser000000000000--00
libraries/WebServer/examples/Filters000000000000--00
libraries/WebServer/examples/HelloServer000000000000--00
libraries/WebServer/examples/HttpAdvancedAuth000000000000--00
libraries/WebServer/examples/HttpAuthCallback000000000000--00
libraries/WebServer/examples/HttpAuthCallbackInline000000000000--00
libraries/WebServer/examples/HttpBasicAuth000000000000--00
libraries/WebServer/examples/HttpBasicAuthSHA1000000000000--00
libraries/WebServer/examples/HttpBasicAuthSHA1orBearerToken000000000000--00
libraries/WebServer/examples/Middleware00--00000000--00
libraries/WebServer/examples/MultiHomedServers000000000000--00
libraries/WebServer/examples/PathArgServer000000000000--00
libraries/WebServer/examples/SDWebServer000000000000--00
libraries/WebServer/examples/SimpleAuthentification000000000000--00
libraries/WebServer/examples/UploadHugeFile000000000000--00
libraries/WebServer/examples/WebServer000000000000--00
libraries/WebServer/examples/WebUpdate000000000000--00
libraries/WiFi/examples/FTM/FTM_Initiator000000000000--00
libraries/WiFi/examples/FTM/FTM_Responder000000000000--00
libraries/WiFi/examples/SimpleWiFiServer000000000000--00
libraries/WiFi/examples/WPS00--00000000--00
libraries/WiFi/examples/WiFiAccessPoint000000000000--00
libraries/WiFi/examples/WiFiBlueToothSwitch00--00--0000--00
libraries/WiFi/examples/WiFiClient000000000000--00
libraries/WiFi/examples/WiFiClientBasic000000000000--00
libraries/WiFi/examples/WiFiClientConnect000000000000--00
libraries/WiFi/examples/WiFiClientEnterprise00--00000000--00
libraries/WiFi/examples/WiFiClientEvents000000000000--00
libraries/WiFi/examples/WiFiClientStaticIP000000000000--00
libraries/WiFi/examples/WiFiExtender000000000000--00
libraries/WiFi/examples/WiFiIPv6000000000000--00
libraries/WiFi/examples/WiFiMulti000000000000--00
libraries/WiFi/examples/WiFiMultiAdvanced000000000000--00
libraries/WiFi/examples/WiFiScan000000000000--00
libraries/WiFi/examples/WiFiScanAsync000000000000--00
libraries/WiFi/examples/WiFiScanDualAntenna000000000000--00
libraries/WiFi/examples/WiFiScanTime000000000000--00
libraries/WiFi/examples/WiFiSmartConfig00--00000000--00
libraries/WiFi/examples/WiFiTelnetToSerial000000000000--00
libraries/WiFi/examples/WiFiUDPClient000000000000--00
libraries/WiFiProv/examples/WiFiProv00--00000000--00
libraries/Wire/examples/WireMaster0000000000000000
libraries/Wire/examples/WireScan0000000000000000
libraries/Wire/examples/WireSlave0000000000000000
libraries/Wire/examples/WireSlaveFunctionalCallback0000000000000000
libraries/Zigbee/examples/Zigbee_Analog_Input_Output00--000000000000
libraries/Zigbee/examples/Zigbee_Binary_Input_Output00--------0000--
libraries/Zigbee/examples/Zigbee_CarbonDioxide_Sensor00--------0000--
libraries/Zigbee/examples/Zigbee_Color_Dimmable_Light00--------0000--
libraries/Zigbee/examples/Zigbee_Color_Dimmer_Switch00--000000000000
libraries/Zigbee/examples/Zigbee_Contact_Switch00--------0000--
libraries/Zigbee/examples/Zigbee_Dimmable_Light00--------0000--
libraries/Zigbee/examples/Zigbee_Electrical_AC_Sensor00--000000000000
libraries/Zigbee/examples/Zigbee_Electrical_AC_Sensor_MultiPhase00--000000000000
libraries/Zigbee/examples/Zigbee_Electrical_DC_Sensor00--------0000--
libraries/Zigbee/examples/Zigbee_Fan_Control00--000000000000
libraries/Zigbee/examples/Zigbee_Gateway00--000000----00
libraries/Zigbee/examples/Zigbee_Illuminance_Sensor00--------0000--
libraries/Zigbee/examples/Zigbee_Multistate_Input_Output00--000000000000
libraries/Zigbee/examples/Zigbee_OTA_Client00--------0000--
libraries/Zigbee/examples/Zigbee_Occupancy_Sensor00--------0000--
libraries/Zigbee/examples/Zigbee_On_Off_Light00--------0000--
libraries/Zigbee/examples/Zigbee_On_Off_MultiSwitch00--000000000000
libraries/Zigbee/examples/Zigbee_On_Off_Switch00--000000000000
libraries/Zigbee/examples/Zigbee_PM25_Sensor00--------0000--
libraries/Zigbee/examples/Zigbee_Power_Outlet00--000000000000
libraries/Zigbee/examples/Zigbee_Pressure_Flow_Sensor00--------0000--
libraries/Zigbee/examples/Zigbee_Range_Extender00--000000000000
libraries/Zigbee/examples/Zigbee_Scan_Networks00--------0000--
libraries/Zigbee/examples/Zigbee_Temp_Hum_Sensor_Sleepy00--------0000--
libraries/Zigbee/examples/Zigbee_Temperature_Sensor00--------0000--
libraries/Zigbee/examples/Zigbee_Thermostat00--000000000000
libraries/Zigbee/examples/Zigbee_Vibration_Sensor00--------0000--
libraries/Zigbee/examples/Zigbee_Wind_Speed_Sensor00--------0000--
libraries/Zigbee/examples/Zigbee_Window_Covering00--------0000--
libraries/ESP32/examples/DeepSleep/TouchWakeUp--000000------00
libraries/ESP32/examples/TWAI/TWAIreceive--00000000000000
libraries/ESP32/examples/TWAI/TWAItransmit--00000000000000
libraries/ESP32/examples/Touch/TouchButton--000000------00
libraries/ESP32/examples/Touch/TouchInterrupt--000000------00
libraries/ESP32/examples/Touch/TouchRead--000000------00
libraries/ESP_I2S/examples/Record_to_WAV--0000--------00
libraries/ESP_SR/examples/Basic--0000----------
libraries/Ethernet/examples/ETH_TLK110--00----------00
libraries/SD_MMC/examples/SD2USBMSC--0000----------
libraries/SD_MMC/examples/SDMMC_Test--0000--------00
libraries/SD_MMC/examples/SDMMC_time--0000--------00
libraries/USB/examples/CompositeDevice--000000--------
libraries/USB/examples/ConsumerControl--000000--------
libraries/USB/examples/CustomHIDDevice--000000--------
libraries/USB/examples/FirmwareMSC--000000--------
libraries/USB/examples/Gamepad--000000--------
libraries/USB/examples/HIDVendor--000000--------
libraries/USB/examples/Keyboard/KeyboardLogout--000000--------
libraries/USB/examples/Keyboard/KeyboardMessage--000000--------
libraries/USB/examples/Keyboard/KeyboardReprogram--000000--------
libraries/USB/examples/Keyboard/KeyboardSerial--000000--------
libraries/USB/examples/KeyboardAndMouseControl--000000--------
libraries/USB/examples/MIDI/MidiController--000000--------
libraries/USB/examples/MIDI/MidiInterface--000000--------
libraries/USB/examples/MIDI/MidiMusicBox--000000--------
libraries/USB/examples/MIDI/ReceiveMidi--000000--------
libraries/USB/examples/Mouse/ButtonMouseControl--000000--------
libraries/USB/examples/SystemControl--000000--------
libraries/USB/examples/USBMSC--000000--------
libraries/USB/examples/USBSerial--000000--------
libraries/USB/examples/USBVendor--000000--------
libraries/ESP32/examples/Camera/CameraWebServer----💚 -16000------00
ESP32/examples/Camera/CameraWebServer (2)----0000------00
ESP32/examples/Camera/CameraWebServer (3)----00----------
libraries/ESP32/examples/DeepSleep/ExternalWakeUp----0000------00
libraries/BluetoothSerial/examples/DiscoverConnect--------------00
libraries/BluetoothSerial/examples/GetLocalMAC--------------00
libraries/BluetoothSerial/examples/SerialToSerialBT--------------00
libraries/BluetoothSerial/examples/SerialToSerialBTM--------------00
libraries/BluetoothSerial/examples/SerialToSerialBT_Legacy--------------00
libraries/BluetoothSerial/examples/SerialToSerialBT_SSP--------------00
libraries/BluetoothSerial/examples/bt_classic_device_discovery--------------00
libraries/BluetoothSerial/examples/bt_remove_paired_devices--------------00
libraries/ESP32/examples/DeepSleep/SmoothBlink_ULP_Code--------------00
libraries/Ethernet/examples/ETH_LAN8720--------------00
libraries/SimpleBLE/examples/SimpleBleDevice--------------00

@lucasssvaz lucasssvaz added the CI Failure Expected For PRs where CI failure is expected label Oct 16, 2025
Copy link
Member

@pedrominatel pedrominatel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docs part LGTM.

@me-no-dev me-no-dev added the Status: Pending Merge Pull Request is ready to be merged label Oct 17, 2025
@me-no-dev me-no-dev merged commit 8db8b55 into master Oct 17, 2025
72 of 73 checks passed
@me-no-dev me-no-dev deleted the ci/fixes branch October 17, 2025 17:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI Failure Expected For PRs where CI failure is expected Status: Pending Merge Pull Request is ready to be merged Type: CI & Testing Related to continuous integration, automated testing, or test infrastructure.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants