feat(recording): full AX element extraction, modifier tracking, scroll phase, Win32 fixes#29
Merged
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
…l phase, Win32 element inspection - macOS: extract all 6 AX attributes (role, title, description, value, identifier, role_description) per click; resolve app name via PID→AXUIElementCreateApplication; switch tap to kCGEventTapOptionListenOnly so slow AX lookups never block the event queue - macOS: extract modifier flags via CGEventGetFlags (Shift/Ctrl/Alt/Cmd/CapsLock bitmask) and actual characters via CGEventKeyboardGetUnicodeString on key events - macOS: read kCGScrollWheelEventScrollPhase (field 99) and kCGScrollWheelEventMomentumPhase (field 123) for accurate scroll phase values - Windows: implement get_element_at via WindowFromPoint + GetClassNameA/GetWindowTextA/ GetAncestor(GA_ROOTOWNER) for role, name, and app extraction without new dependencies - Windows: extract modifier state via GetKeyState (VK_SHIFT/CONTROL/MENU/LWIN/RWIN) and translate characters via GetKeyboardState + ToUnicode on key events - Windows: add WM_MOUSEWHEEL and WM_MOUSEHWHEEL handling (HIWORD delta / WHEEL_DELTA=120) - Windows: fix pre-existing bug where GLOBAL_TX was never set in WindowsRecorder::start(), making all Windows recording completely broken (events never reached the bridge channel) - core/events.rs: add value, description, identifier, role_description optional fields to ElementInfo for full attribute capture - README: mark all four previously unchecked recording features as complete https://claude.ai/code/session_01GAPteYe8ztzpa1mdh2M2m6
mohabbis
force-pushed
the
feat/recording-improvements
branch
from
5350659 to
dc98511
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
AXUIElementCopyElementAtPosition; app name is resolved via PID →AXUIElementCreateApplication→ AXTitle. Tap switched tokCGEventTapOptionListenOnlyso slow AX lookups never block the system event queue.CGEventGetFlags(Shift/Ctrl/Alt/Cmd/CapsLock) and the actual characters viaCGEventKeyboardGetUnicodeString.kCGScrollWheelEventScrollPhase(field 99) with fallback tokCGScrollWheelEventMomentumPhase(field 123) for accurate phase values.get_element_atimplemented viaWindowFromPoint+GetClassNameA/GetWindowTextA/GetAncestor(GA_ROOTOWNER)— no new dependencies required.GetKeyStateextracts modifier state;GetKeyboardState+ToUnicodetranslates virtual key codes to characters.WM_SYSKEYDOWN/UPnow handled.WM_MOUSEWHEELandWM_MOUSEHWHEELnow recorded using HIWORD(mouseData)/WHEEL_DELTA normalisation — previously missing entirely.GLOBAL_TXwas declared but never assigned inWindowsRecorder::start(), making all Windows recording completely broken (hooks installed but events never reached the bridge channel). Fixed.ElementInfo: addedvalue,description,identifier,role_descriptionoptional fields for full attribute capture.Test plan
ElementInfoin the event timeline shows role, name, app, and at least some of the new optional fieldsmodifiersbitmask andcharsare populated correctlyphasefield is non-zero where applicableElementInforole/name/app are populatedScrollevents appear in the timelineinspect_elementcommand returns populatedElementInfo[x]https://claude.ai/code/session_01GAPteYe8ztzpa1mdh2M2m6
Generated by Claude Code