CI: Windows tests fail randomly with current runner image

[sebastianhaberey]

Description

Windows tests fail on current CI on runner windows-2022 image tag win22/20230307.2 which contains Windows Server 202210.0.20348 Build 1547. The tests that fail vary from run to run, but at least one test fails. These tests are run automatically if Windows sources change, or manually. Last successful run was on 06.06.2022 on runner windows-2022 image tag win22/20220529.1 which contains Windows Server 2022 10.0.203348 Build 707.

It doesn't seem possible to revert runner images to previous versions and I don't have a live Windows Server 2022 installation to test this. Tests pass reliably on current Windows 10 (10.0.19045.2673) machine (see below).

Test output on Windows 10 test machine (3 runs)

C:\Dokumente\Projekte\nsd\nsd\example(main -> origin)                                 
λ flutter test -r expanded -d windows integration_test\nsd_test.dart                  
Building Windows application...                                                       
00:00 +0: Registration, discovery and unregistration of multiple services             
00:11 +1: Verify basic attributes of registered service                               
00:12 +2: Verify txt attribute of registered service                                  
00:14 +3: Registration is possible if port is in use                                  
00:15 +4: Find all available service types                                            
00:15 +5: Look up IP addresses for service                                            
00:16 +6: (tearDownAll)                                                               
00:16 +6: All tests passed!                                                           
                                                                                      
C:\Dokumente\Projekte\nsd\nsd\example(main -> origin)                                 
λ flutter test -r expanded -d windows integration_test\nsd_test.dart                  
Building Windows application...                                                       
00:00 +0: Registration, discovery and unregistration of multiple services             
00:11 +1: Verify basic attributes of registered service                               
00:12 +2: Verify txt attribute of registered service                                  
00:14 +3: Registration is possible if port is in use                                  
00:14 +4: Find all available service types                                            
00:15 +5: Look up IP addresses for service                                            
00:16 +6: (tearDownAll)                                                               
00:16 +6: All tests passed!                                                           
                                                                                      
C:\Dokumente\Projekte\nsd\nsd\example(main -> origin)                                 
λ flutter test -r expanded -d windows integration_test\nsd_test.dart                  
Building Windows application...                                                       
00:00 +0: Registration, discovery and unregistration of multiple services             
00:11 +1: Verify basic attributes of registered service                               
00:12 +2: Verify txt attribute of registered service                                  
00:14 +3: Registration is possible if port is in use                                  
00:15 +4: Find all available service types                                            
00:15 +5: Look up IP addresses for service                                            
00:16 +6: (tearDownAll)                                                               
00:16 +6: All tests passed!                                                           

Flutter doctor output on Windows 10 test machine

[√] Flutter (Channel stable, 3.7.7, on Microsoft Windows [Version 10.0.19045.2673], locale de-DE)
    • Flutter version 3.7.7 on channel stable at C:\Tools\flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 2ad6cd72c0 (6 days ago), 2023-03-08 09:41:59 -0800
    • Engine revision 1837b5be5f
    • Dart version 2.19.4
    • DevTools version 2.20.1

[X] Windows Version (Unable to confirm if installed Windows version is 10 or greater)

[√] Android toolchain - develop for Android devices (Android SDK version 32.1.0-rc1)
    • Android SDK at C:\Users\User\AppData\Local\Android\sdk
    • Platform android-32, build-tools 32.1.0-rc1
    • Java binary at: C:\Program Files\Android\Android Studio\jre\bin\java
    • Java version OpenJDK Runtime Environment (build 11.0.13+0-b1751.21-8125866)
    • All Android licenses accepted.

[√] Chrome - develop for the web
    • Chrome at C:\Program Files\Google\Chrome\Application\chrome.exe

[√] Visual Studio - develop for Windows (Visual Studio Community 2022 17.2.1)
    • Visual Studio at C:\Program Files\Microsoft Visual Studio\2022\Community
    • Visual Studio Community 2022 version 17.2.32516.85
    • Windows 10 SDK version 10.0.19041.0

[√] Android Studio (version 2021.3)
    • Android Studio at C:\Program Files\Android\Android Studio
    • Flutter plugin can be installed from:
       https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
       https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.13+0-b1751.21-8125866)

[√] VS Code (version 1.67.1)
    • VS Code at C:\Users\User\AppData\Local\Programs\Microsoft VS Code
    • Flutter extension can be installed from:
       https://marketplace.visualstudio.com/items?itemName=Dart-Code.flutter

[√] Connected device (3 available)
    • Windows (desktop) • windows • windows-x64    • Microsoft Windows [Version 10.0.19045.2673]
    • Chrome (web)      • chrome  • web-javascript • Google Chrome 102.0.5005.63
    • Edge (web)        • edge    • web-javascript • Microsoft Edge 110.0.1587.56

[√] HTTP Host Availability
    • All required HTTP hosts are available

! Doctor found issues in 1 category.

Scenario

Other (please describe above)

Platform

Windows

[sebastianhaberey]

Self-hosted runner on Windows 10 failed to run job, see https://github.com/sebastianhaberey/nsd/actions/runs/4445462134/jobs/7804618812.

[sebastianhaberey]

The last successful run was based on commit 32d7a1a which corresponds to plugin version 2.2.3 (2.2.2 was a broken version).

[sebastianhaberey]

Older Plugin versions (2.2.3, 2.3.0) fail on the current runner, too.

[sebastianhaberey]

Since this seems to be related to the Windows Server 2000 environment, which is not a very likely platform for the plugin to run on, and the tests pass in an Windows 10 environment, I am removing the Windows badge from the documentation until the CI succeeds again (maybe with new versions of the Github runner) or there is a way to run Github actions on Windows 10/11.

[sebastianhaberey]

I decided to leave this open as a reminder that the Windows CI is broken.

[sebastianhaberey]

Won't fix for now, see issue #41