Skip to content

Refactor accessibility command queue and utilities; add keyboard/media helpers and multimodal serialization#63

Merged
Android-PowerUser merged 4 commits intocodex/analyze-technical-debt-and-complexity-o6bm9pfrom
identify-technical-debt-in-app
Mar 29, 2026
Merged

Refactor accessibility command queue and utilities; add keyboard/media helpers and multimodal serialization#63
Android-PowerUser merged 4 commits intocodex/analyze-technical-debt-and-complexity-o6bm9pfrom
identify-technical-debt-in-app

Conversation

@Android-PowerUser
Copy link
Copy Markdown
Owner

Motivation

  • Improve reliability and concurrency of accessibility command processing by replacing ad-hoc queues and flags with a dedicated AccessibilityCommandQueue abstraction.
  • Factor platform-specific helpers (media projection starter, accessibility state checker, keyboard visibility observer) out of MainActivity to simplify lifecycle management and reduce duplicated code.
  • Centralize command model and parsing, and add utilities for app name mapping, string similarity, image handling and serialization to support robust multimodal/photo reasoning features.

Description

  • Added AccessibilityCommandQueue, AccessibilityServiceStateChecker, KeyboardVisibilityObserver, and MediaProjectionServiceStarter to encapsulate queueing, state checks, keyboard detection, and service start logic respectively, and wired them into MainActivity and ScreenOperatorAccessibilityService.
  • Reworked ScreenOperatorAccessibilityService to use AccessibilityCommandQueue and AccessibilityServiceStateChecker and updated queue processing logic to use the new API (enqueue, tryAcquireProcessing, releaseProcessing, etc.).
  • Introduced a shared Command sealed class in util/Command.kt and substantially refactored CommandParser to be more robust (pattern metadata, overlap handling, single-instance command deduplication and logs).
  • Added multimodal helpers: PhotoReasoningSerialization for streaming JSON and bitmap->Base64 conversion, and MainActivityBridge to safely access MainActivity context/API key from view models; updated PhotoReasoningViewModel to use these utilities and to delegate AI calls to the foreground service.
  • Reworked app mapping logic (AppMappings + AppNamePackageMapper) to use structured definitions and string-similarity matching via a new StringSimilarity util, improved normalization and caching behavior.
  • Various utility and preferences improvements: ImageUtils cleanup, CoordinateParser simplification, GenerationSettingsPreferences, ChatHistoryPreferences, SystemMessagePreferences, SystemMessageEntryPreferences, and UserInputPreferences refactored for safer SharedPreferences access, default population, and clearer keys/serializers.

Testing

  • Built the Android app with ./gradlew assembleDebug to verify compilation with the refactor; build succeeded.
  • Ran unit tests with ./gradlew test (project unit tests) and static checks; they completed successfully.

Codex Task

Copy link
Copy Markdown

@amazon-q-developer amazon-q-developer bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR implements significant refactoring to improve accessibility command processing, lifecycle management, and code organization. However, critical thread-safety bugs have been identified in the new AccessibilityCommandQueue class that must be fixed before merge.

Critical Issues (Must Fix):

  • 🛑 Thread-safety bugs in AccessibilityCommandQueue: The clearAndUnlock, enqueue, peek, poll, size, and isEmpty methods lack synchronization, which can cause race conditions, lost commands, and ConcurrentModificationExceptions when accessed by multiple threads simultaneously.

Summary:
The refactoring successfully:

  • Introduces dedicated abstractions for command queue, service state checking, keyboard visibility, and media projection starting
  • Extracts Command sealed class and improves CommandParser with better pattern handling and deduplication
  • Adds multimodal helpers for streaming JSON, bitmap serialization, and MainActivity bridge
  • Refactors app mapping with structured definitions and string similarity matching
  • Improves SharedPreferences handling across utility classes

Once the thread-safety issues are resolved with proper synchronization, this refactoring will significantly improve code maintainability and reliability.


You can now have the agent implement changes and create commits directly on your pull request's source branch. Simply comment with /q followed by your request in natural language to ask the agent to make changes.

Android-PowerUser and others added 3 commits March 29, 2026 19:15
…ueue.kt

Co-authored-by: amazon-q-developer[bot] <208079219+amazon-q-developer[bot]@users.noreply.github.com>
…ueue.kt

Co-authored-by: amazon-q-developer[bot] <208079219+amazon-q-developer[bot]@users.noreply.github.com>
…ueue.kt

Co-authored-by: amazon-q-developer[bot] <208079219+amazon-q-developer[bot]@users.noreply.github.com>
@Android-PowerUser Android-PowerUser merged commit e0681f9 into codex/analyze-technical-debt-and-complexity-o6bm9p Mar 29, 2026
@Android-PowerUser Android-PowerUser deleted the identify-technical-debt-in-app branch March 29, 2026 17:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant