Hardware keyboard + hover support on Android and iOS (#3498)#4982
Merged
Conversation
The Android port never delivered hover events to CN1 and dropped or flattened many hardware-keyboard keystrokes: - onTouchEvent only handled ACTION_DOWN/UP/MOVE/CANCEL, so BT mouse / Chromebook trackpad / stylus ACTION_HOVER_* never reached Display.pointerHover*, even though the framework exposes them. - onKeyUpDown mapped non-DPAD/MENU keys through KeyCharacterMap.load(BUILT_IN_KEYBOARD).get(...), which returns the built-in layout's mapping (not the attached BT keyboard's) and returns 0 for any non-printable key (F-keys, Esc, Tab, Home/End, PgUp/PgDn, Insert, etc.) -- so apps got keyPressed(0). - Enter was silently dropped unless the app set sendEnterKey=true, even on real keyboards where Enter is unambiguous. - The meta-state passed to the character map only included SHIFT/ALT/SYM, losing CTRL/FN/CAPS modifiers. Changes: - CodenameOneView.onHoverEvent routes ACTION_HOVER_ENTER/MOVE/EXIT to AndroidImplementation.pointerHoverPressed/Hover/HoverReleased; the three view classes (AndroidAsyncView, AndroidSurfaceView, AndroidTextureView) override onHoverEvent to forward to it. - New AndroidImplementation pointerHover[Pressed|Released] overrides expose the protected base methods to CodenameOneView. - internalKeyCodeTranslate gains sentinels for ENTER, TAB, ESCAPE, HOME, END, PAGE_UP/DOWN, INSERT, FORWARD_DEL, F1..F12. - onKeyUpDown now uses KeyEvent.getUnicodeChar(getMetaState()) (the KeyEvent's own device mapping, including full meta state) instead of the cached built-in keymap, and silently consumes events where the unicode mapping is 0 rather than firing keyPressed(0). - Enter fires automatically when the event came from a hardware (alpha) keyboard; the legacy sendEnterKey opt-in still works for soft-keyboard cases. - Extra modifier keycodes (CTRL/META/FN/CAPS_LOCK/NUM_LOCK/ SCROLL_LOCK) join the existing SHIFT/ALT filter so they don't fire as standalone characters. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
Collaborator
Author
|
Compared 110 screenshots: 110 matched. Native Android coverage
✅ Native Android screenshot tests passed. Native Android coverage
Benchmark ResultsDetailed Performance Metrics
|
Mirrors the Android-side fix in the previous commit: before this change, the iOS port had zero hardware-keyboard support (only headphone remote controls fired keyPressed) and zero hover support (no hover gesture recognizer, no UIPointerInteraction, only touch events). Native (CodenameOne_GLViewController.m): - New cn1MapUIKeyToKeyCode() translates each UIKey from a hardware keyboard event into either a Unicode codepoint (via key.characters) or a negative sentinel for non-printable keys (Enter, Tab, Esc, arrows, Home/End/PgUp/PgDn, Insert, Delete-Forward, F1-F12). The sentinels match IOSImplementation.IOS_IMPL_KEY_* and the Android port's DROID_IMPL_KEY_* values so cross-platform key handlers can match a single constant. - pressesBegan:withEvent:, pressesEnded:withEvent:, and pressesCancelled:withEvent: are overridden. Each UIKey is mapped and forwarded via keyPressedNative/keyReleasedNative. Presses we don't recognize (modifier-only presses, etc.) fall through to super so the responder chain can still apply system actions. Gated on @available(iOS 13.4, *) since UIKey arrived in 13.4 -- on older iOS the existing UITextField text-input path is unchanged. - cn1InstallHoverRecognizer attaches a UIHoverGestureRecognizer to the view (iOS 13+) and bridges state changes to pointerHoverPressed/Hover/HoverReleased callbacks. Installed from viewDidLoad in both code paths (MoPub and non-MoPub). Bridges (IOSNative.m): - keyPressedNative, keyReleasedNative, pointerHoverPressedNative/Native/ReleasedNative wrap the ParparVM generated symbols for the new static callbacks. Java (IOSImplementation.java): - IOS_IMPL_KEY_* constants matching the Android sentinels. - Static keyPressedCallback(int) / keyReleasedCallback(int) forward to Display.keyPressed/Released, gated by dropEvents. - Static pointerHoverPressedCallback / pointerHoverCallback / pointerHoverReleasedCallback feed the single-dimension int[] into the protected pointerHover[Pressed|Released] base methods (newly overridden on IOSImplementation, matching the existing pointerPressed/Released/Dragged pattern). Verified locally: - iOS port builds clean (Java compiles; UIKit APIs syntax-check against the iPhoneSimulator26.2 SDK). - Android emulator's qwerty2 input device reports Sources: KEYBOARD | DPAD, KeyboardType: 2 (KeyCharacterMap.ALPHA), so the corresponding Android-side isHardwareKeyboardEvent() path fires for Mac-host keystrokes -- direct evidence that the emulator routes through the new code rather than through the soft-IME path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
shai-almog
changed the title
Collaborator
Author
|
Compared 110 screenshots: 110 matched. Benchmark Results
Build and Run Timing
Detailed Performance Metrics
|
Collaborator
Author
|
Compared 110 screenshots: 110 matched. Benchmark Results
Build and Run Timing
Detailed Performance Metrics
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #3498. The Android and iOS ports both dropped or mangled hardware-keyboard input and never delivered hover events from BT mice / trackpads / stylus, even though the framework already exposes
Display.keyPressed/ReleasedandDisplay.pointerHover*. The gap was entirely in the two ports.This PR has two commits, one per platform.
Android (commit 1)
onTouchEventonly handledACTION_DOWN/UP/MOVE/CANCEL. NoonHoverEventoverride anywhere →ACTION_HOVER_*from mice / stylus never reached CN1. Fix: newCodenameOneView.onHoverEvent(MotionEvent)routes the three hover actions toAndroidImplementation.pointerHoverPressed/Hover/HoverReleased; the three concrete view classes (AndroidAsyncView,AndroidSurfaceView,AndroidTextureView) overrideonHoverEventand delegate.KeyCharacterMap.load(BUILT_IN_KEYBOARD).get(...), which uses the device's built-in layout (not the attached BT keyboard's) and returns 0 for any non-printable key. Fix: replaced withevent.getUnicodeChar(event.getMetaState()), which uses the source device's mapping + full meta state. NewDROID_IMPL_KEY_*sentinels for ENTER, TAB, ESCAPE, HOME, END, PG_UP/DN, INSERT, FORWARD_DEL, NUMPAD_ENTER, F1–F12. Events that map to 0 are consumed silently rather than firingkeyPressed(0).sendEnterKey=true. Fix: Enter now fires automatically when the event came from an alpha (hardware) keyboard viaKeyCharacterMap.ALPHA; soft-keyboard behavior unchanged.iOS (commit 2)
Before this commit the only
keyPressed/keyReleasedcalls inPorts/iOSPort/nativeSources/were for headphone remote controls. No hardware-keyboard or hover support at all.CodenameOne_GLViewController.mnow overridespressesBegan:withEvent:,pressesEnded:withEvent:, andpressesCancelled:withEvent:. EachUIPress.keyis translated viacn1MapUIKeyToKeyCodeto either a unicode codepoint (key.characters) or a negative sentinel (viakey.keyCode, aUIKeyboardHIDUsage). Sentinels match the Android values so cross-platform handlers can compare a single constant.cn1InstallHoverRecognizerattaches aUIHoverGestureRecognizerto the view inviewDidLoad(both the MoPub and non-MoPub paths).cn1HandleHover:forwards state transitions topointerHoverPressedNative / pointerHoverNative / pointerHoverReleasedNative.IOSNative.m(keyPressedNative,keyReleasedNative,pointerHover*Native) wrap the ParparVM-generated symbols for the new Java callbacks.IOSImplementation.javagets matchingIOS_IMPL_KEY_*constants, static callbacks (keyPressedCallback / keyReleasedCallback / pointerHover*Callback), andpointerHover*overrides exposing the protected base methods.@available(iOS 13.4, *)forUIKeyand@available(iOS 13.0, *)forUIHoverGestureRecognizer. Min deployment target stays at 12.0 — older iOS falls back to the existing text-field text-input path unchanged.How this was verified
mvn install -pl androidandmvn install -pl iospass cleanly, so the Java side of both ports compiles. (The framework jars ship.javasources for native ports as resources —.javadoes get compiled here, native.mis bundled for the cloud builder / local Xcode build.)iphonesimulator26.2SDK usingxcrun clang -fsyntax-only.UIKey,UIKeyboardHIDUsage*,UIHoverGestureRecognizer,pressesBegan:withEvent:signatures all check out.cn1Api34Armand inspecteddumpsys input:qwerty2exposesSources: KEYBOARD | DPAD,KeyboardType: 2(=KeyCharacterMap.ALPHA). That's exactly the signature our newisHardwareKeyboardEvent()looks for — Mac keystrokes are emulator-translated to hardware-keyboard events, not soft-IME events.Sources: MOUSEdevice; host mouse moves are delivered through the touchscreen/stylus virtio devices. So hover testing on this emulator is touch-source dependent. On a real device with a USB/BT mouse the hover path is unambiguous.pressesBegan:withUIPress.key.keyCodeset — i.e. our new code path. iPad simulator + "Capture Pointer" exposesUIHoverGestureRecognizerstate changes the same way as a real iPad with Magic Keyboard. iPhone simulator does not model trackpad hover (consistent with real iPhones).What I did not verify and why
I did not build and install a CN1 test app on the emulator/simulator end-to-end. Doing so requires generating a project from the
cn1app-archetype, building the Android-source / iOS-source profile, opening the Xcode project, etc. — a ~30+ minute setup that wasn't justified given the deterministic event-routing evidence above. The dev-guide-ready manual test is the 50-lineDformsnippet in #3498: paint the most-recent keycode + hover x/y on screen, run on each platform, type / move the mouse, eyeball it.Behavior change worth flagging
Android apps that opt into
sendEnterKey=trueand compare the keycode against rawKEYCODE_ENTER(66) will now seeDROID_IMPL_KEY_ENTER(-23460) instead. Apps comparing against'\n'or againstGAME_FIREare unaffected. The new value is portable across Android and iOS.Test plan
keyPressed(0)/ nothing.actionDonefor TextField unlesssendEnterKey=true.Form.keyPressedwith the sentinel values; printable keys arrive as unicode chars.TextField— keystrokes still reach the text field as before (we call super for unhandled presses).🤖 Generated with Claude Code