TestPilot

Overview

TestPilot enables testing any Android APK on standard JVM without emulators or physical devices. By converting DEX bytecode to JVM bytecode and providing an Android API shim layer, developers can run fast, reliable UI tests directly on their development machines or CI servers.

Core Idea

"Test any APK, anywhere JVM runs"

Input: APK file → Output: Test results on pure JVM

val session = TestPilot.load("app.apk").launch()

// Tap by coordinates
session.tap(100f, 200f)

// Tap by view ID
session.tap(R.id.login_button)

// Lifecycle control
session.pause().resume().stop().destroy()

// Screenshot (using layoutlib for pixel-perfect rendering)
val screenshot = session.takeScreenshot("""
    <LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
        android:layout_width="match_parent"
        android:layout_height="match_parent">
        <TextView
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"
            android:text="Hello World" />
    </LinearLayout>
""")
ImageIO.write(screenshot, "PNG", File("screenshot.png"))

Product Direction

Replace Maestro + Emulator with pure JVM execution for faster, more reliable testing.

Architecture Evolution

Current State:

┌─────────────┐     ┌─────────┐     ┌───────────────────┐
│ Test Cases  │ ──▶ │ Maestro │ ──▶ │ Emulator / Device │
└─────────────┘     └─────────┘     └───────────────────┘
                                              │
                              Slow startup, resource heavy, CI struggles

Target State:

┌─────────────┐     ┌───────────────┐     ┌───────────────┐
│ Test Cases  │ ──▶ │ TestPilot SDK │ ──▶ │ App Simulator │
└─────────────┘     └───────────────┘     └───────────────┘
                            │                     │
                      APK as input      Pure JVM, instant startup

Value Proposition

Pain Point	Current State	With TestPilot
CI Startup Time	Emulator cold start 30-60s	JVM process < 2s
Parallelization	Limited by KVM/machine resources	Dozens of instances on single machine
Flakiness	System animations, popup interference	Fully controlled environment
Debug Capability	adb logcat log retrieval	Direct breakpoints, same process
Input	Source code + build	Just APK

How It Works

┌─────────┐    ┌─────────────┐    ┌─────────────┐    ┌─────────────┐    ┌─────────┐
│   APK   │ ──▶│   Unpack    │ ──▶│ DEX → JVM   │ ──▶│  Bytecode   │ ──▶│   Run   │
│         │    │             │    │  (dex2jar)  │    │  Rewrite    │    │ on JVM  │
└─────────┘    └─────────────┘    └─────────────┘    └─────────────┘    └─────────┘
                    │                                      │
                    ▼                                      ▼
              resources/                          android.view.View
              AndroidManifest.xml                        ↓
                                               io.johnsonlee.testpilot.simulator.View

Pipeline Steps

1. APK Unpacking: Extract classes.dex, resources, AndroidManifest.xml
2. DEX → JVM Conversion: Convert Dalvik bytecode to JVM bytecode using dex2jar/enjarify
3. Bytecode Rewriting: Replace android.* references with TestPilot shim classes
4. Execution: Load transformed classes and run on standard JVM

Technical Challenges

1. DEX → JVM Bytecode Conversion

• Android uses DEX/ART bytecode format, JVM uses class files
• Solution: Use proven tools (dex2jar, enjarify) for conversion
• Risk: Some edge cases may not convert perfectly

2. Android Framework Dependencies

• android.* packages rely heavily on native implementations
• Solution: Implement shim layer that mimics Android API behavior
• Priority: Focus on UI-related APIs first (View, Activity, Resources)

3. Native Code (JNI) - Biggest Risk

• Many APKs contain .so native libraries
• These cannot run on standard JVM
• Solution: Stub/mock JNI calls, or provide pure-Java alternatives
• Scope limitation: Apps heavily dependent on native code may not be fully testable

4. Resources System

• Binary XML parsing (AndroidManifest, layouts)
• resources.arsc qualifier resolution (density, locale, night mode)
• R.java constant mapping

Architecture

┌───────────────────────────────────────────────────────────────────┐
│                          TestPilot SDK                            │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────────┐    │
│  │ APK Loader  │  │  Assertions │  │  UI Actions             │    │
│  │             │  │             │  │  (tap/swipe/input)      │    │
│  └─────────────┘  └─────────────┘  └─────────────────────────┘    │
├───────────────────────────────────────────────────────────────────┤
│                         App Simulator                             │
│  ┌─────────────────────────────────────────────────────────────┐  │
│  │                    Transformed APK Code                     │  │
│  │       (android.* → io.johnsonlee.testpilot.simulator.*)     │  │
│  └─────────────────────────────────────────────────────────────┘  │
│  ┌─────────────────────────────────────────────────────────────┐  │
│  │                    Android API Shim Layer                   │  │
│  │         Activity | View | Resources | Intent | ...          │  │
│  └─────────────────────────────────────────────────────────────┘  │
├───────────────────────────────────────────────────────────────────┤
│                       Rendering (Layoutlib)                       │
│  ┌─────────────────────────────────────────────────────────────┐  │
│  │                  Android Official Layoutlib                 │  │
│  │              Pixel-perfect rendering on JVM                 │  │
│  └─────────────────────────────────────────────────────────────┘  │
├───────────────────────────────────────────────────────────────────┤
│                            JVM (JDK 21+)                          │
└───────────────────────────────────────────────────────────────────┘

Dual-Layer Design

TestPilot uses a dual-layer architecture to achieve both accurate behavior simulation and pixel-perfect rendering:

Layer	Technology	Purpose
Behavior Simulation	Bytecode Rewriting + Shim	Activity lifecycle, touch events, view hierarchy traversal
Rendering	Android Layoutlib	Screenshot capture, visual regression testing

Why this approach?

• Shim Layer: Fast, lightweight simulation for behavior testing (lifecycle, touch dispatch)
• Layoutlib: Android's official rendering library used by Android Studio, ensures pixel-perfect screenshots that match real device rendering

This is similar to how Paparazzi (by Cash App) works for screenshot testing.

Key Components

Component	Approach
APK Processing	Unzip + dexlib2 + ASM bytecode rewriting
Activity Lifecycle	State machine + callback chain
View System	measure/layout/draw pipeline implementation
LayoutInflater	Binary XML parsing + reflection-based construction
Resources	resources.arsc parsing + qualifier resolution
Event Dispatch	TouchEvent simulation via View hierarchy
Rendering	Android Layoutlib (official Android rendering library)

Scope & Limitations

Supported

• Pure Kotlin/Java APKs
• Standard UI components (View, ViewGroup, common widgets)
• Activity lifecycle testing
• UI interaction testing (tap, swipe, text input)
• Layout verification

Not Supported (Initially)

• Native code (JNI/.so libraries)
• Hardware features (Camera, Bluetooth, sensors)
• System services requiring real Android (ContentProvider with system data)
• Compose UI (future consideration)

Testing Strategy

1. Unit Tests - Shim API Contract Verification

class ActivityLifecycleTest {
    @Test
    fun `onCreate should be called before onStart`() {
        val calls = mutableListOf<String>()
        val activity = TestActivity { calls += it }

        activityController.create().start()

        assertThat(calls).containsExactly("onCreate", "onStart")
    }
}

2. Integration Tests - Real APK Execution

class ApkLoadingTest {
    @Test
    fun `should load and launch simple APK`() {
        val app = TestPilot.load("test-fixtures/simple-app.apk")

        app.launch("com.example.MainActivity")

        assertThat(app.currentActivity).isNotNull()
        assertThat(app.findView<TextView>(R.id.title).text).isEqualTo("Hello")
    }
}

3. Visual Regression - Screenshot Comparison

class LayoutRenderingTest {
    private val snapshots = SnapshotManager(File("src/test/snapshots"))

    @Test
    fun `LinearLayout vertical should match Android rendering`() {
        val session = TestPilot.load("test-fixtures/layout-test.apk").launch()

        val screenshot = session.takeScreenshot("""
            <LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
                android:layout_width="match_parent"
                android:layout_height="match_parent"
                android:orientation="vertical">
                <TextView
                    android:layout_width="wrap_content"
                    android:layout_height="wrap_content"
                    android:text="Hello World" />
            </LinearLayout>
        """)

        // Assert against golden image with 1% tolerance
        screenshot.assertMatchesSnapshot(snapshots, "layout_activity", tolerance = 0.01)
    }

    @Test
    fun `record new golden image`() {
        val session = TestPilot.load("app.apk").launch()
        val screenshot = session.takeScreenshot(layoutXml)

        // Auto-record if golden doesn't exist
        screenshot.assertMatchesSnapshot(snapshots, "new_screen", recordIfMissing = true)
    }
}

4. Behavioral Tests - Golden Data Comparison

class ViewMeasureSpecTest {
    @Test
    fun `MeasureSpec packing should match Android`() {
        val androidResults = loadGoldenData("measure_spec_cases.json")

        androidResults.forEach { case ->
            val jvmResult = MeasureSpec.makeMeasureSpec(case.size, case.mode)
            assertThat(jvmResult).isEqualTo(case.expected)
        }
    }
}

Roadmap

Phase 1: POC (1-2 days) ✅ COMPLETE

• Basic Activity lifecycle state machine
• Basic View/ViewGroup implementation
• Simple Canvas rendering
• APK unpacking (extract DEX, resources, manifest)
• DEX → JVM conversion (using dexlib2 + ASM)
• Basic bytecode rewriting framework (android.* → shim mapping)

Result: Successfully loaded a real APK (7.5MB, 3713 classes) with 0 conversion errors

Phase 2: MVP (1-2 weeks) ✅ COMPLETE

• DEX instruction to JVM bytecode translation
• AndroidManifest.xml binary parsing
• Resources.arsc parsing (resource ID mapping)
• LayoutInflater with binary XML
• Common widgets (TextView, Button, ImageView, EditText, ScrollView, ProgressBar, etc.)
• Touch event dispatch
• TestPilot SDK basic API

Result: Full APK loading pipeline with touch event dispatch. Supports tap interactions and event listeners.

Phase 3: Real App Support (3-4 weeks) 🚧 IN PROGRESS

• Layoutlib Integration - Android official rendering library
  • Add layoutlib dependencies
  • Create renderer module
  • Implement takeScreenshot() API
  • Visual comparison utilities (ImageComparator, SnapshotManager, assertions)
• Complete Resources system with qualifiers
• Fragment support
• RecyclerView
• ViewPager
• More widgets coverage

Goal: Test medium-complexity real-world APKs with pixel-perfect screenshots

Phase 4: Production Ready (4-6 weeks)

• android.view.Window API support (statusBarColor, navigationBarColor, flags, decorView, etc.)
• Performance optimization (caching, incremental processing)
• Comprehensive widget support
• CI/CD integration guide
• Documentation & examples
• Edge case handling

Effort Estimation

Module	Estimated Effort
APK Processing	1-2 days
DEX → JVM Conversion	1 day
Bytecode Rewriting	2-3 days
Android API Shim	15-25 days
Rendering Backend	3-5 days
TestPilot SDK	3-5 days
Total	25-40 days

Note: Shim layer is the long tail - core 20% APIs cover 80% of use cases.

Use Cases

1. Fast UI Testing: Test any APK on pure JVM, no emulator needed
2. CI/CD Optimization: Parallel test execution without KVM overhead
3. Quick Verification: Instant APK testing without device deployment
4. Reliable Tests: Eliminate flakiness from system animations and popups
5. Better Debugging: Same-process debugging with standard IDE tools

Prior Art & Inspiration

• Robolectric: Shadow-based Android testing (requires source, TestPilot takes APK)
• Paparazzi: Screenshot testing using Layoutlib (requires source, TestPilot takes APK)
• Android Layoutlib: Official Android rendering library used by Android Studio
• dex2jar/dexlib2: DEX to JAR conversion
• Maestro: Test orchestration API inspiration
• ASM: Bytecode manipulation library

Alternatives Considered

Approach	Pros	Cons	Decision
DEX → JVM conversion	Mature tools, cacheable	Some conversion edge cases	✅ Chosen
DEX interpreter	100% fidelity	Huge effort, slow execution	❌ Too complex
Source-based (like Robolectric)	Simple	Can't test arbitrary APK	❌ Different goal
Optimized emulator	Real Android	Still slow, needs KVM	❌ No differentiation

---

TestPilot - Test any Android APK, anywhere JVM runs