feat(vlm): Vision/VLM screen + full modality model catalog (0.20.9 parity) - #4
Merged
Merged
Conversation
Bring the RN starter to feature + catalog parity with the monorepo
reference example, using only published @runanywhere/*@0.20.9 public APIs.
- Add VisionScreen (image input + RunAnywhere.processImageStream) with
streaming, token-by-token descriptions, sample images + custom URL/path
input, editable prompt, and cancel. Manual iterator.next() loop per the
Hermes NitroModules async-iterable caveat.
- Add VLMService wrapping RunAnywhere.processImageStream /
cancelVLMGeneration / currentModel (MULTIMODAL), mirroring the reference
example's VLMService.
- ModelService: add VLM download/load lifecycle (MULTIMODAL) and register
a curated model per modality using structured proto enums:
LLM -> lfm2-350m-q8_0 (+ smollm2-360m-q8_0 alt)
VLM -> smolvlm-500m-instruct-q8_0 (LLAMA_CPP, MULTIMODAL, TAR_GZ)
STT -> sherpa-onnx-whisper-tiny.en
TTS -> vits-piper-en_US-lessac-medium
VAD -> silero-vad (ONNX, VOICE_ACTIVITY_DETECTION)
ids/urls/framework/category/artifactType copied from the reference
ModelCatalogBootstrap.
- Wire Vision into the navigator, HomeScreen grid + model-info, screens
index, FeatureCard/ModelLoaderWidget icon maps; refresh README.
Gate: npx tsc --noEmit clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T8PFRCwpmNzKLyMwFPGzCd
sanchitmonga22
force-pushed
the
feat/vlm-vision-parity
branch
from
July 13, 2026 22:45
c963c3f to
ff80512
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Brings the React Native starter to full SDK-feature + model-catalog parity with the monorepo reference example (
examples/react-native/RunAnywhereAI), using only published@runanywhere/*@0.20.9public APIs.The headline change is the new Vision / VLM screen (the one capability the starter was missing) plus a curated model for every modality the starter exposes. This PR is stacked on the
0.20.9integration baseline, so it also carries those preceding commits.What's new (this work)
👁 Vision / VLM screen (
src/screens/VisionScreen.tsx)RunAnywhere.processImageStream— streaming, token-by-token descriptions.react-native-fsand the on-disk path is fed to the VLM (VLM_IMAGE_FORMAT_FILE_PATH), mirroring the reference example's file-path flow.iterator.next()loop over the NitroModules async iterable (nofor await...of).src/services/VLMService.tsThin wrapper over
RunAnywhere.processImageStream/cancelVLMGeneration/currentModel(MULTIMODAL), mirroring the reference example'sVLMService.Model catalog — one curated model per modality (
src/services/ModelService.tsx)Registered with structured proto enums (
INFERENCE_FRAMEWORK_*,MODEL_CATEGORY_*,MODEL_ARTIFACT_TYPE_*) — ids/urls/framework/category/artifactType copied from the referenceModelCatalogBootstrap:lfm2-350m-q8_0(+smollm2-360m-q8_0alt)smolvlm-500m-instruct-q8_0sherpa-onnx-whisper-tiny.envits-piper-en_US-lessac-mediumsilero-vadAlso added the VLM (
MULTIMODAL) download/load lifecycle toModelServiceand its unload path.Wiring
Vision added to the navigator (
App.tsx),RootStackParamList,HomeScreengrid + model-info panel,screens/index.ts, and theFeatureCard/ModelLoaderWidgeticon maps. README refreshed with the Vision feature and the full model table.Not included (deferred, per scope)
Verification
npx tsc --noEmit— clean (0 errors).eslinton new/changed source — no new errors.🤖 Generated with Claude Code