Testing

Testing

How the claims on What works where are checked, and how to run the same checks yourself.

The goal in one line: prove the published NuGet packages work, on every platform, without a person clicking anything.

The emphasis on published packages is the whole design. It would be easier to test the source tree — and the source tree is not what breaks. doc/Packaging.md and doc/KnownGaps.md record the same failure shape repeatedly: the code was right, the package was wrong, and nothing noticed until a device found out.

Five tiers

flowchart TB
    t5["<b>Blazor</b> — headless Chromium<br/>the one binding that only exists in a browser"]
    t4["<b>Runtime, on device</b> — Android · iOS · Mac Catalyst<br/>an app that runs the scenarios and prints results"]
    t3["<b>Runtime, Windows</b> — a plain test process<br/>does the native half load and negotiate?"]
    t2["<b>Consumption</b> — restore + compile per TFM<br/>does the package work for a consumer?"]
    t1["<b>Unit + integration</b> — any machine, no hardware<br/>is the logic right?"]

    t1 --> t2 --> t3 --> t4 --> t5

Loading

Deliberately not a UI-driving pyramid. WebRTCme's product is an API, and an API can be called directly — there is no UI to drive, so there is no Appium.

Tier	Project	Runs on	Proves
1	Tests/WebRTCme.Tests	any machine, net10.0	The logic is right. Tests the source
2	Tests/WebRTCme.PackageTests	PC + Mac	The package restores and compiles for a consumer, per TFM
3	Tests/WebRTCme.DeviceTests	Windows	The native half loads and negotiates
4	Tests/WebRTCme.DeviceTests.Runner	Android, iOS, Mac Catalyst	The same, on a real device
5	Tests/WebRTCme.BlazorTestHost + …BlazorTests	headless Chromium	The same, in a browser

Stack: xUnit v3 + NSubstitute + FluentAssertions.

Invoke the test projects directly, not with dotnet test. Since the xunit.v3 4.0 / Microsoft.Testing.Platform change, dotnet test fails outright on the .NET 10 SDK — "Testing with VSTest target is no longer supported". Use dotnet run --project …. --filter "X" becomes -filterVSTest "X", and --logger trx becomes -result-trx.

The central idea: loopback

A WebRTC smoke test needs no second machine, no signalling server and no network. Two RTCPeerConnection objects in one process, with the offer, the answer and the ICE candidates handed between them in code — which is all signalling ever does.

That one test proves, on whichever platform is running it:

• the native half loaded at all — libwebrtc.aar, WebRTC.xcframework, WebRTC.framework, WebRtcInterop.dll, or the browser's own WebRTC;
• the binding marshals SDP in both directions;
• ICE candidates survive the round trip;
• DTLS and SCTP work.

A package with a missing or flat framework fails it immediately, which is the failure this project keeps shipping. It needs no camera, so it runs anywhere — including on a CI runner.

Tests/WebRTCme.DeviceTests.Core holds those scenarios once and all three runtime tiers execute them, so Windows, a phone and a browser are running literally the same code. Hello world is a reduction of it.

Media is a separate group. GetUserMedia needs a real capture device, and that is where nearly every bug in KnownGaps.md actually lived — rotation, hot-plug, backgrounding, device-died-under-a-call. Those tests skip cleanly when no device is present, so the negotiation core stays runnable everywhere.

Tier 1 — unit and integration

No package, no device, no second peer. Runs in about two seconds.

dotnet run --project Tests/WebRTCme.Tests

=== TEST EXECUTION SUMMARY ===
   WebRTCme.Tests  Total: 114, Errors: 0, Failed: 0, Skipped: 0, Not Run: 0, Time: 1.979s

Unit/OrtcTests	Capability negotiation — GetExtendedRtpCapabilites, GetSendingRtpParameters, ReduceCodecs, CanSend / CanReceive, and the validators' rejection cases
Unit/H264Tests	ParseProfileLevelId, round-trips, GenerateProfileLevelIdForAnswer, IsSameProfile — table-driven from the spec's own examples
Unit/ScalabilityModesTests	L1T3, S3T3, malformed input, empty string
Unit/CommonUtilsTests	Including ExtractDtlsParameters, reached through InternalsVisibleTo
Unit/ModelExtensionsTests	ToStringOrNumber, whose in-place dictionary mutation was a real fixed bug
Unit/SpeakingDetectorTests	The noise floor adapting, the clamp, the hangover holding through a pause, and explicitly a comma-decimal culture — the fault that once made one peer report speaking permanently
Integration/ServiceRegistrationTests	AddMiddleware() and AddMediaSoup() build a container that resolves
Integration/SignalingServerTests	The real SignalR hub in-process via WebApplicationFactory, driven by a real SignalR client: join, peer-joined, leave, dropped connection

This tier is where most of the mediasoup port lives, and the port is the largest body of borrowed logic here — the one most likely to drift from its JavaScript original.

SpeakingDetectorTests exists because a production refactor was done for it: the threshold rule was lifted out of SignalingConnection's async loop into a pure class taking (level, now). That was an explicit decision rather than a surprise in a diff.

Tier 2 — consumption

Does the package work for a consumer? A project per target framework whose only job is to restore the packed .nupkg and compile against it — PackageReference, never ProjectReference.

gh run download <run-id> -n nupkg -D artifacts
./Tests/Test-Package.ps1 -Version 26.9.21

Compiling is not the assertion, and this is the subtlety that makes the tier worth having. NuGet falls back: a net10.0-android project with no Android slice resolves lib/net10.0/ instead and compiles perfectly, because the common API is identical across slices. What it would not have is any Android binding, and nothing says so until an app runs on a phone. So the script reads project.assets.json back and requires every target framework to have resolved its own slice.

It is deliberately not in WebRTCme.sln: there is no package to restore until someone downloads a CI artifact, and a test that breaks the ordinary build gets deleted rather than fixed.

This tier alone would have caught the packaging faults this repository has actually shipped. It also found the MAUI version pin that every consumer now has to write — see Platform prerequisites.

Tier 3 — runtime on Windows

The loopback scenarios plus device enumeration, in a plain xUnit process against the package.

Mac Catalyst was originally planned for this tier and cannot be: Catalyst builds an .app rather than an executable, xUnit v3 refuses to build a test project without an app host, and there is no app host for maccatalyst-x64. Setting UseAppHost produces the second error, so the two cannot both be satisfied. Mac Catalyst moved to tier 4.

Tier 4 — runtime on device

Android, iOS and Mac Catalyst cannot run a plain test process either, for the same reason. So the scenarios run inside Tests/WebRTCme.DeviceTests.Runner, a MAUI app that executes them and prints one line per result, and Tests/Test-Device-Phase4.ps1 builds it, deploys it, runs it twice and reads the lines back.

./Tests/Test-Device-Phase4.ps1 -Platform android

Not XHarness, which the original plan assumed. The newest build on the dotnet-eng feed is from September 2023: it predates .NET 10, and it drives Apple devices through mlaunch, which hangs with "Please connect the device" on a locked phone. adb and devicectl work, and the harness drives them directly — so the extra NuGet feed the plan worried about is not needed either.

All three run from a PC, including the two Apple ones, over SSH to a Mac.

iOS targets a real iPhone, not the Simulator. The Simulator works and is one command, but it runs the simulator slice of WebRTC.xcframework; the arm64 slice is the one that ships.

Four things here cost real time and none is discoverable from an error message:

• Codesigning a device build must go through the desktop session. From SSH it fails with errSecInternalComponent, because an SSH session's keychain is not the GUI session's. The harness asks Terminal.app to run the build via osascript, with a completion file to carry the exit code back.
• devicectl's command-line arguments never reach the app — passed plainly or after --, the runner silently ran all scenarios instead of the one asked for. The filter travels as an environment variable instead.
• macOS has no timeout, and its absence is silent: the app appears to have run and reported nothing.
• A harness authored on Windows sends CRLF to zsh, and the failure is indistinguishable from an app that crashed on launch.

The last three all fail in the same direction — a green-looking or empty result rather than a loud one.

Tier 5 — Blazor

The odd one out: the Blazor binding only exists inside a browser.

./Tests/Test-Blazor.ps1 -Version 26.9.21

WebRTCme.BlazorTestHost is a Blazor WebAssembly app that runs the shared scenarios and writes each result into the DOM; WebRTCme.BlazorTests drives headless Chromium with Playwright and asserts on what the page reported. It waits on the results element's data-state, so a page that never finished fails rather than reading as zero failures.

Chrome's --use-fake-device-for-media-stream supplies synthetic media, so Blazor is the one platform where the media group needs no hardware at all — and the one tier that can run while another app is holding the machine's only camera.

What this tier found on its first run: that a Blazor consumer had to reconfigure JSInterop's JSON serialiser by reflection or no peer connection could be created at all. It was the first consumer ever written without copying the demo app's startup, and it failed immediately. Fixed in the binding; see Bindings.

In CI

.github/workflows/ci.yml is manual (workflow_dispatch) and runs the Apple slice build, the pack and verify, then every tier a hosted runner can honestly run: unit, package consumption, Windows runtime, Blazor, Mac Catalyst, iOS Simulator and the Android emulator.

The cost of being manual is real and worth stating: nothing verifies a commit automatically, so a packaging fault sits undetected until somebody runs it or pushes a tag. Run it before tagging a release, not after.

The uploaded nupkg artifact is a publishable package, and it is what the test jobs consume — and what publish.yml later promotes unchanged.

One job is knowingly unreliable: the Android emulator fails about two runs in three with a native SIGABRT inside libwebrtc's signalling thread. It has never once happened on an arm64 phone, the emulator runs x86_64 and no user does, and the .aar is stripped so the failing assertion is unreachable from here. It is kept for the managed regressions it does catch — both faults it found in September 2026 were managed-code faults — and it is not worth failing a release over.

Configuration

Every host-specific value is an environment variable with a sensible default, so anyone with a PC and a Mac can run the suite by setting a handful.

variable	default
WEBRTCME_PACKAGE_SOURCE	./artifacts	Folder feed holding the .nupkg under test
WEBRTCME_PACKAGE_VERSION	the csproj <Version>	Which version to restore
WEBRTCME_TESTS_REQUIRED	unset	When 1, a missing device fails instead of skipping
WEBRTCME_TEST_ARTIFACTS	TestResults/	Where logs and captured SDP go on failure
WEBRTCME_ANDROID_UDID	first attached	Which Android device

Note what is absent: no Appium URL, no per-host IP, no signalling server address. The loopback design means each host runs its own suite against itself, so the two machines never have to find each other.

Where the packages come from

The tests consume CI-built packages, not packages built on the test machine — because a package built on a PC has that machine's Apple slices, and on Windows those are flat and unsignable. Testing a locally built package would test something that will never be published.

What stays manual, honestly

WebRTCme.Middleware ships the video tile, and that is UI. A headless test can prove frames arrive; it cannot prove they are drawn, the right way up, at the right size — which is precisely the frame-rotation bug class that took four commits across three platforms.

So a short manual checklist survives for rendering, and the demo apps are how it is checked. Also manual: real multi-party SFU behaviour against mediasoup, TURN relay paths, and anything needing two physically separate networks.

The record of one call across all five platforms at once is in doc/KnownGaps.md under "One call, five platforms" — what it covered, and what it cost to set up.