feat(desktop): move to new native vnc render pipeline#18
Merged
Conversation
- Add @pi-ui/client/vnc package with full RFB protocol implementation:
- protocol.ts: type definitions (PixelFormat, Rect, Event, Adapter)
- rfb-protocol.ts: RFB state machine + all encoding decoders
(Raw, CopyRect, RRE, Hextile, ZRLE, DesktopSize)
- vnc-auth.ts: pure-JS DES cipher for VNC password auth
- vnc-input.ts: pointer/keyboard event encoders + keysym table
- vnc-transport.ts: WebSocket boundary with message classification
- wasm-pipeline.ts: WASM decoder loader (web-only, JS fallback on native)
- Add WASM pipeline infrastructure:
- AssemblyScript source for framebuffer-owning decoder
- Pre-built WASM binary at public/wasm/
- Build script for recompilation
- Add useVncSession hook:
- Orchestrates WebSocket transport + RFB protocol
- Auto-loads WASM on web, JS fallback on native
- Encoding fallback (timeout -> Raw-only reconnect)
- Exposes sendPointerEvent/sendKeyEvent/sendWheelEvent
- Rewrite web viewer (vnc-viewer.web.tsx):
- HTML canvas with WASM pipeline (single putImageData per frame)
- JS fallback with per-rect putImageData
- ResizeObserver aspect-ratio scaling
- Full pointer/keyboard/wheel input handling
- Rewrite native viewer (vnc-viewer.native.tsx):
- Skia Canvas + Skia.Image.MakeImage from RGBA pixels
- JS-side framebuffer with rect blitting
- Trackpad-style drag-to-move with visible cursor overlay
- Tap=click, long-press=right-click, two-finger=scroll
- Hidden TextInput keyboard bridge
- Dependencies: @shopify/react-native-skia, fflate, @assemblyscript/loader
Replace useRef with useSharedValue for all values accessed inside gesture callbacks (pan, tap, long-press, two-finger scroll). Use runOnJS to bridge worklet -> JS thread for setState and session.sendPointerEvent/sendWheelEvent calls. Fixes Reanimated crash: 'Tried to modify key current of an object which has been already passed to a worklet'
- Back press while keyboard is open → dismiss keyboard - Back press while fullscreen/immersive → exit fullscreen - Otherwise → default back navigation
- Slide-out panel on left edge with Keyboard, Fullscreen, Reconnect - Small arrow tab toggle (▶/◀) at left center - Animated slide in/out with spring animation - Back button closes menu before exiting fullscreen - Shows resolution info at bottom of panel - Same pattern as the original noVNC slide-out menu
- Remove KeyboardAvoidingView wrapping (was resizing canvas container on keyboard open causing layout recalc + image scale jump) - Remove declarative StatusBar (was causing layout shift on toggle) - Use imperative StatusBar.setHidden with 'fade' animation instead - Canvas container now stays full size regardless of keyboard state
Use KeyboardAvoidingView with behavior='position' on iOS (translates content up without resizing container) and no behavior on Android (system handles it). This shifts the canvas up when keyboard opens so VNC content remains visible, without triggering layout recalc.
Switch KeyboardAvoidingView back to padding/height behavior so the canvas area shrinks when keyboard opens. Debounce onLayout (150ms) to prevent rapid setCanvasSize calls during keyboard animation, eliminating the flicker while still updating the scale after the keyboard settles.
0xHexE
changed the title
This was referenced Mar 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.