Add support for Compose Preview Screenshot Testing

[dfabulich]

https://github.com/dfabulich/skipapp-compose-snapshot-sample

This is a Skip Lite dual-platform app project, written mostly in Swift that transpiles to Kotlin, demonstrating how to setup and run Compose Preview Screenshot Testing.

What's Compose Preview Screenshot Testing?

In Compose Preview Screenshot Testing, you write @Preview components (like SwiftUI #Preview, but in Jetpack Compose) and decorate them with @PreviewTest. An update build plugin captures screenshots of all of your @PreviewTest components and writes them to disk; you commit them with your code.

Later, you run your tests again in validate mode. The tests fail if any of your components render differently. You then review the updated screenshots in a generate HTML report; if you approve of all of the changes, rerun update, which wil regenerate all of the snapshots.

Writing a test

@PreviewTest
@Preview(showBackground = true)
@Composable
fun WelcomeViewPreview() {
    ProcessInfo.launch(LocalContext.current)
    Surface(
        modifier = Modifier.fillMaxSize(),
        color = MaterialTheme.colorScheme.background
    ) {
        WelcomeView(welcomeName = Binding.constant("Skipper"))
            .Compose(context = ComposeContext())
    }
}

Running tests

Generate screenshot test results:

gradle -p Android -Pandroid.experimental.enableScreenshotTest=true updateDebugScreenshotTest

Validate that screenshot test results are unchanged:

gradle -p Android -Pandroid.experimental.enableScreenshotTest=true valdateDebugScreenshotTest

(We have to use -P because setting this in Android/app/gradle.properties change doesn't apply to all of the projects that Skip generates; see below.)

[dfabulich]

Since I was able to get it set up, debatably, Skip already kinda does support this, but I think Skip could go a long way toward making these easier to run.

1. xcodebuild fails in this sample project. Add "android.experimental.enableScreenshotTest=true" to gradle.properties
   The Skip Gradle build needs to be updated to be aware of screenshot tests, how to enable them, and how to avoid building/running them when they're not needed.
2. There could be a SkipScreenshotTest library, wrapping https://github.com/pointfreeco/swift-snapshot-testing for iOS snapshots and Compose Snapshot Testing for Android snapshots.
3. skip init/skip create could add an example snapshot test. (Maybe just for Android, or maybe one for iOS and one for Android.)
4. I'd like the screenshot test plugin to be installed in generated build.gradle.kts files automatically; users shouldn't have to mess around with skip.yml to turn them on.
5. Skip could generate snapshot tests from SwiftUI previews somehow. See also Transpile Compose @Previews from SwiftUI #Preview #588. Note that the plugin currently doesn't support running tests directly on @Previews in production source; this is a top-voted feature request. https://issuetracker.google.com/issues/340578712

[marcprux]

Very interesting! Thanks for all the good research. I wasn't aware of the Compose Preview Screenshot Testing library.

I had previously been musing on a higher-level screenshot testing solution like Maestro, but it seems like this could be a valid lower-level solution with potentially much higher performance and reliability. In either case, having functional parity on both the Android and iOS sides would be very nice to have. I.e., it would be great to just be able to write a single test and have it work on both iOS and Android without needing to do too much extra work on each platform.

Additionally, being able to reuse such a solution for automated screenshot testing with the ability to vary based on locale to be used for fastlane distribution would be ideal (they have their own iOS capture_screenshots and Android capture_android_screenshots, but those don't fulfill any sort of interface testing).

We don't have any short-term plans to work on this, but it is something that would be an awesome addition!

[dfabulich]

I think my main request in the short term is #587. That will unbreak xcodebuild when screenshot tests are enabled.

[dfabulich]

I spent some time investigating this, and I found some bugs that I wanted to share.

kotlin-reflect hackery

https://issuetracker.google.com/issues/482433854 "Screenshot testing fails when composable uses kotlin-reflect"

This caused the screenshot tests to fail whenever I (for example) tried putting a ScrollView in a screenshot, because it would try to read a preference, and PreferenceKey would use key.companionObjectInstance, which requires kotlin-reflect.

The problem is that the plugin brings an enormous blob of bundled classes in compose-preview-renderer. compose-preview-renderer-0.0.1-alpha13.jar contains 26,245 files, including a copy of org/objectweb/asm, org/checkerframework, and its own copy of a bunch of kotlin/* classes, including the stub version of kotlin/reflect.

I found a hacky workaround where I wrote an afterEvaluate Gradle rule to forcibly inject kotlin-reflect into the plugin's classpath; then, it worked normally.

gotcha: clean your .build folder or validate might wrongly claim "build successful"

https://issuetracker.google.com/issues/482664893 "validate incorrectly succeeds with stale screenshot when test throws exception"

This one's just a gotcha… the update command and the validate command put a copy of the screenshot in .build that doesn't automatically get cleaned up. So if you successfully validate, then update the code in a way that throws an exception, the exception appears logged in the gradle log output, but gradle returns "build successful", relying on the stale screenshot in .build.

Compose @Previews have some inherent limitations (no filesystem, no network)

https://developer.android.com/develop/ui/compose/tooling/previews#best-practices

Limitations and best practices

Android Studio executes previews code directly in the preview area. It doesn't require running an emulator or physical device because it leverages a ported part of the Android framework called Layoutlib. Layoutlib is a custom version of the Android framework designed to run outside of Android devices. The goal of the library is to provide a preview of a layout in Android Studio that is very close to its rendering on devices.

Previews limitations

Because of the way previews are rendered within Android Studio, they are lightweight and don't require the whole Android framework to render them. However, this comes with the following limitations:

• No network access
• No file access
• Some Context APIs may not be fully available

As a result:

• The SQLPlayground is basically untestable under screenshot testing, because it requires filesystem access.
• Even StoragePlayground fails, as android.content.SharedPreferences.getAll() returns null (not empty).

There is a way for Compose code to detect that it's running in Preview mode:

@Composable
fun GreetingScreen(name: String) {
    if (LocalInspectionMode.current) {
        // Show this text in a preview window:
        Text("Hello preview user!")
    } else {
        // Show this text in the app:
        Text("Hello $name!")
    }
}

I thought about trying to use that, to update SQLPlayground to use an in-memory database in Preview mode. Maybe I'll do that some day.

You get one composition and no recompositions

The plugin will capture its screenshot immediately after the first composition.

This means that anything in a GeometryReader simply won't show up at all. GeometryReader renders nothing until its onGloballyPositionedInRoot handler fires, updating its state, and then rendering the content based on the updated state. On first composition, the handler hasn't fired yet, and the result is blank.

None of the images work either, not even the local resources, because they're all using coil to load them asynchronously from the jar.

WebView is untestable

Perhaps this is to be expected, since there's no network access, but I couldn't even test locally embedded resources/HTML, because layoutlib provides a broken mock implementation of WebView, missing methods like getSettings() that have been present since API level 1.