Testing

Testing

How Open Wallpaper Engine is tested: the unit test target, targeted runs, library sweeps, asset-gated tests, benchmarks and reference comparisons, CI, and the Swift 6.2 type-check traps.

The test target

• OpenWallpaperEngineTests: unit tests hosted in the app (over 500 files). Under XCTest the app starts without its delegate.
• Fixtures live in Tests/Fixtures/ (outside the target): scenes, effects, particles, models, timelines, SceneScript, Workshop, NowPlaying, video and more. Read them with Fixtures.url(_:). They are written for the project; never WE files.
• Every fix or feature comes with a test. Format and value tests decode fixtures; rendering checks go in RenderCheckTests.
• Known gaps are asserted with XCTExpectFailure("<snapshot id>: …"). It is strict: fixing a gap makes its test fail until you delete the expectation.
• Tests never touch the user's state. Under XCTest, AppStorageLocation switches to the suite app.openwallpaperengine.isolated.tests, Open Wallpaper Engine (isolated tests) folders and isolated Keychain services. AppStorageIsolationTests guards this.

Running tests

All tests (as CI's asset job runs them):

Scripts/fetch-we-assets.sh     # downloads or refreshes the WE assets, see below
TEST_RUNNER_OWE_ASSETS=<assets folder> xcodebuild test \
  -project OpenWallpaperEngine.xcodeproj -scheme OpenWallpaperEngine

Targeted runs (one class or method) are much faster while iterating:

xcodebuild test -project OpenWallpaperEngine.xcodeproj -scheme OpenWallpaperEngine \
  -only-testing:OpenWallpaperEngineTests/ShaderVariantCacheTests

Environment variables reach the test host through xcodebuild with the TEST_RUNNER_ prefix. The full list is on Environment variables.

Test tiers

Tier	Needs	Examples	On CI
Unit and fixture	Nothing	Format decoding, values, the clock, particles, SceneScript, Workshop filters, settings	Run
Asset-gated	OWE_ASSETS (an assets folder or WE install)	Shader translation of built-ins, WE labels, effect galleries, model tests using WE models. They start with _ = try Fixtures.assets()	Skipped on PRs; run on main and nightly
Library	A wallpaper library, found in the usual places or via OWE_LIBRARY (:-separated)	LibrarySweepTests, ImageMaterialSweepTests, ParticleSimulationSweepTests, TimelineLibrarySweepTests, ModelLibraryRenderTests, BloomLibrarySweepTests, HDRLibrarySweepTests	Skipped
Benchmarks	Opt-in variables	SceneFrameBenchmarkTests (OWE_SCENE_BENCH), SceneDetailEquivalenceTests, SceneSharedInstanceBenchmarkTests, SceneTextureLoadBenchmarkTests, ParticleLibraryBenchmarkTests, LightingFrameBenchmarkTests, SceneScriptLibraryCostTests, TimelineCostTests	Skipped
Reference comparison	WE captures	WEReferenceComparisonTests (OWE_WE_REFERENCE), WEExtrasComparisonTests, ModelGroundTruthLibraryTests, ScenePuppetReferenceTests	Skipped

Reference comparisons

WEReferenceComparisonTests draws every captured wallpaper headlessly with the real loader and renderer and compares it with Wallpaper Engine's stills: one 1920×1080 display, the capture's settings, default properties, seeded particles, a warm-up until pipelines compile, then the scene clock stepped to each still's time. Metrics per 8×6 cell and for the frame: mean colour and luma differences, SSIM, and Sobel edge alignment.

Guards on the rules

Test	Guards
ShaderVariantCacheTests	Translated output changes need a ShaderVariantTranslator.revision bump
LocalizationCatalogTests, LocalizationLintTests	Every string is translated; localizing APIs get literals
AppStorageIsolationTests	Tests never touch the user's defaults, folders or Keychain
WEAuthoredValuesTests	Values come from WE's data, not invented constants

WE assets from Steam

Scripts/fetch-we-assets.sh downloads WE's assets/, locale/ui_*.json and projects/defaultprojects/ with DepotDownloader 3.4.0 (pinned, SHA-256 verified), logged in as the dedicated CI Steam account that owns Wallpaper Engine. It skips the download while the cached .build matches Steam's current build, and --dry-run shows the plan without logging in. Credentials come from OWE_CI_STEAM_USER, OWE_CI_STEAM_PASSWORD and OWE_CI_STEAM_SHARED_SECRET, or on a Mac from the login Keychain. Each command prompts for its value:

security add-generic-password -U -s owe-ci-steam -a user -w
security add-generic-password -U -s owe-ci-steam -a password -w
security add-generic-password -U -s owe-ci-steam -a shared-secret -w

The default output is ~/Library/Caches/owe-we-assets ($OWE_WE_ASSETS_DIR, or the first argument, overrides it); point TEST_RUNNER_OWE_ASSETS at it. Exit codes: 3 missing credentials, 4 login failed (password, Steam Guard, clock), 5 rate limited, 6 account doesn't own WE, 7 download failed, 8 DepotDownloader fetch or checksum failed, 9 build ID lookup failed (--build-id). Setup and troubleshooting: docs/ci-assets.md in the repository. Scripts/fill-assets-cache.sh still copies from a WE install on disk.

CI

.github/workflows/ci.yml runs on pushes to main, on pull requests and on manual runs. build-and-test (the required check) runs everywhere (forks included) without assets: a plan job picks the test classes (on a PR, only those the changed files select through .github/test-map.yml), a build job builds the tests once, three test shards run them with Scripts/ci-run-tests.sh, and a packages job runs swift test on Packages/. asset-tests runs only on pushes to main and manual runs from main (the Steam secrets live in the steam-ci environment): asset-cache restores or refreshes the encrypted asset cache, then four shards run every class with TEST_RUNNER_OWE_ASSETS. .github/workflows/nightly.yml runs the whole suite with the assets and the slow tier (OWE_SLOW_TESTS=1) every night. Library-gated tests skip everywhere. The no-asset build:

Step	Details
Runner	macos-26 (actool on macOS 15 crashes compiling the layered AppIcon.icon)
Xcode	26.3 (the macOS 26 SDK), falling back to the oldest Xcode 26 present
Build	xcodebuild build-for-testing, Debug with Scripts/test-build-settings.txt (optimised), CODE_SIGNING_ALLOWED=NO, no assets (asset tests skip)
Deployment target check	Fails if a linked library was built for a newer macOS than the app
Toolchain symbols	Scripts/check-toolchain-symbols.sh checks glslang and SPIRV-Cross for clashes
On failure	Uploads xcodebuild.log

Swift 6.2 type-check traps

CI's Swift 6.2 type checker is slower and stricter than a local build and has failed on expressions that compile locally. Past fixes in tests:

• Chained SIMD/matrix products: Swift 6.2 couldn't pick SIMD4's scalar in a chained matrix product. Split the chain and annotate the intermediate types.
• Inferred optional columns: it read inferred first/last columns as optional in SIMD2's initializer. Give the values explicit types.
• Large literal expressions (expected values, fixture arrays): build them with loops or split them into typed lets. One took 108 ms to type-check locally, and longer on CI.

Rule of thumb: in tests, give expected values explicit types and keep arithmetic expressions short.

Before pushing

Run Scripts/ci-local.sh (the local gate, the suite the way CI runs it) with OWE_ASSETS set, so the asset-dependent tests run too (CI runs them only on main and nightly).

User guide: Troubleshooting