Skip to content

feat(vlm): Vision/VLM screen + full modality model catalog (0.20.9 parity) - #4

Merged
sanchitmonga22 merged 1 commit into
mainfrom
feat/vlm-vision-parity
Jul 13, 2026
Merged

feat(vlm): Vision/VLM screen + full modality model catalog (0.20.9 parity)#4
sanchitmonga22 merged 1 commit into
mainfrom
feat/vlm-vision-parity

Conversation

@sanchitmonga22

Copy link
Copy Markdown
Contributor

Summary

Brings the React Native starter to full SDK-feature + model-catalog parity with the monorepo reference example (examples/react-native/RunAnywhereAI), using only published @runanywhere/*@0.20.9 public APIs.

The headline change is the new Vision / VLM screen (the one capability the starter was missing) plus a curated model for every modality the starter exposes. This PR is stacked on the 0.20.9 integration baseline, so it also carries those preceding commits.

What's new (this work)

👁 Vision / VLM screen (src/screens/VisionScreen.tsx)

  • On-device image understanding via RunAnywhere.processImageStream — streaming, token-by-token descriptions.
  • Image input adapted to the starter's dependency set (no new native modules): tap a sample image or paste a custom image URL / local path. Remote sources are fetched to the app cache with react-native-fs and the on-disk path is fed to the VLM (VLM_IMAGE_FORMAT_FILE_PATH), mirroring the reference example's file-path flow.
  • Editable prompt, live preview, and cancel mid-generation.
  • Hermes-safe: manual iterator.next() loop over the NitroModules async iterable (no for await...of).

src/services/VLMService.ts

Thin wrapper over RunAnywhere.processImageStream / cancelVLMGeneration / currentModel(MULTIMODAL), mirroring the reference example's VLMService.

Model catalog — one curated model per modality (src/services/ModelService.tsx)

Registered with structured proto enums (INFERENCE_FRAMEWORK_*, MODEL_CATEGORY_*, MODEL_ARTIFACT_TYPE_*) — ids/urls/framework/category/artifactType copied from the reference ModelCatalogBootstrap:

Modality Model Framework / Category
LLM lfm2-350m-q8_0 (+ smollm2-360m-q8_0 alt) LLAMA_CPP / LANGUAGE
VLM smolvlm-500m-instruct-q8_0 LLAMA_CPP / MULTIMODAL (tar.gz)
STT sherpa-onnx-whisper-tiny.en SHERPA / SPEECH_RECOGNITION
TTS vits-piper-en_US-lessac-medium SHERPA / SPEECH_SYNTHESIS
VAD silero-vad ONNX / VOICE_ACTIVITY_DETECTION

Also added the VLM (MULTIMODAL) download/load lifecycle to ModelService and its unload path.

Wiring

Vision added to the navigator (App.tsx), RootStackParamList, HomeScreen grid + model-info panel, screens/index.ts, and the FeatureCard / ModelLoaderWidget icon maps. README refreshed with the Vision feature and the full model table.

Not included (deferred, per scope)

  • Diffusion / image generation — intentionally out of scope.

Verification

  • npx tsc --noEmitclean (0 errors).
  • eslint on new/changed source — no new errors.
  • No native run / pod install attempted (tsc is the gate).

🤖 Generated with Claude Code

Bring the RN starter to feature + catalog parity with the monorepo
reference example, using only published @runanywhere/*@0.20.9 public APIs.

- Add VisionScreen (image input + RunAnywhere.processImageStream) with
  streaming, token-by-token descriptions, sample images + custom URL/path
  input, editable prompt, and cancel. Manual iterator.next() loop per the
  Hermes NitroModules async-iterable caveat.
- Add VLMService wrapping RunAnywhere.processImageStream /
  cancelVLMGeneration / currentModel (MULTIMODAL), mirroring the reference
  example's VLMService.
- ModelService: add VLM download/load lifecycle (MULTIMODAL) and register
  a curated model per modality using structured proto enums:
    LLM  -> lfm2-350m-q8_0 (+ smollm2-360m-q8_0 alt)
    VLM  -> smolvlm-500m-instruct-q8_0 (LLAMA_CPP, MULTIMODAL, TAR_GZ)
    STT  -> sherpa-onnx-whisper-tiny.en
    TTS  -> vits-piper-en_US-lessac-medium
    VAD  -> silero-vad (ONNX, VOICE_ACTIVITY_DETECTION)
  ids/urls/framework/category/artifactType copied from the reference
  ModelCatalogBootstrap.
- Wire Vision into the navigator, HomeScreen grid + model-info, screens
  index, FeatureCard/ModelLoaderWidget icon maps; refresh README.

Gate: npx tsc --noEmit clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T8PFRCwpmNzKLyMwFPGzCd
@sanchitmonga22
sanchitmonga22 force-pushed the feat/vlm-vision-parity branch from c963c3f to ff80512 Compare July 13, 2026 22:45
@sanchitmonga22
sanchitmonga22 merged commit d1ee3d3 into main Jul 13, 2026
1 check passed
@sanchitmonga22
sanchitmonga22 deleted the feat/vlm-vision-parity branch July 13, 2026 22:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant