Gemma 3n E4B IT cannot run with `LlmInference.Backend.GPU` on Pixel 7

[nmrenyi]

Doesn't work. The APP would stuck on the LLM loading page, and crash

private val mediaPipeLanguageModelOptions: LlmInferenceOptions =
    LlmInferenceOptions.builder().setModelPath(
        baseFolder + GEMMA_MODEL
    ).setPreferredBackend(LlmInference.Backend.GPU).setMaxTokens(4096).build()

Though on CPU it works normally

private val mediaPipeLanguageModelOptions: LlmInferenceOptions =
    LlmInferenceOptions.builder().setModelPath(
        baseFolder + GEMMA_MODEL
    ).setPreferredBackend(LlmInference.Backend.CPU).setMaxTokens(4096).build()

[nmrenyi]

Investigation findings:

Root cause: This is a known regression in tasks-genai v0.10.13+. The MediaPipe GPU path uses OpenCL shader compilation that exhausts host (CPU) RAM during compilation — not a GPU VRAM issue. Google's official docs now explicitly state: "The LLM Inference API does not reliably support Pixel 7." See MediaPipe Issue #5406.

Why AI Edge Gallery works on the same device: Gallery uses a completely different inference library — com.google.ai.edge.litertlm:litertlm-android:0.9.0-alpha04 — which has a newer MLDrift-based GPU backend that avoids the shader compilation OOM entirely. Our app uses com.google.mediapipe:tasks-genai:0.10.25 (via localagents-rag), which has the broken GPU path.

What we tried:

• Switching to LlmInference.Backend.GPU → crashes during init (subgraph 2/4)
• Deferring Gecko + SQLite loading until after LLM GPU init → still crashes, confirming memory competition is not the issue

GPU works on: Pixel 8+, Samsung S23+ (Snapdragon 8 Gen 2+).

Fix options:

1. Migrate to litertlm-android — would fix GPU on Pixel 7, but requires replacing the LLM backend in RagPipeline.kt and dropping the localagents-rag inference chain (embeddings/vector store can stay). Model format changes from .task to .litertlm.
2. Stay on CPU — works fine, no action needed for now.

Current status: Reverted to CPU. Leaving for later unless a GPU-capable test device is available.

[nmrenyi]

Investigation findings:

Root cause: This is a known regression in tasks-genai v0.10.13+. The MediaPipe GPU path uses OpenCL shader compilation that exhausts host (CPU) RAM during compilation — not a GPU VRAM issue. Google's official docs now explicitly state: "The LLM Inference API does not reliably support Pixel 7." See MediaPipe Issue #5406.

Why AI Edge Gallery works on the same device: Gallery uses a completely different inference library — com.google.ai.edge.litertlm:litertlm-android:0.9.0-alpha04 — which has a newer MLDrift-based GPU backend that avoids the shader compilation OOM entirely. Our app uses com.google.mediapipe:tasks-genai:0.10.25 (via localagents-rag), which has the broken GPU path.

What we tried:

• Switching to LlmInference.Backend.GPU → crashes during init (subgraph 2/4)
• Deferring Gecko + SQLite loading until after LLM GPU init → still crashes, confirming memory competition is not the issue

GPU works on: Pixel 8+, Samsung S23+ (Snapdragon 8 Gen 2+).

Fix options:

1. Migrate to litertlm-android — would fix GPU on Pixel 7, but requires replacing the LLM backend in RagPipeline.kt and dropping the localagents-rag inference chain (embeddings/vector store can stay). Model format changes from .task to .litertlm.
2. Stay on CPU — works fine, no action needed for now.

Current status: Reverted to CPU. Leaving for later unless a GPU-capable test device is available.

The comment is written by Claude Code, based on our discussion to solve this issue.