Skip to content

Add AI Agent Android app: accessibility automation, planner, Gemini API client and UI#2

Open
Alex72-py wants to merge 2 commits intomainfrom
codex/build-android-ai-agent-app-from-scratch-wjt8x9
Open

Add AI Agent Android app: accessibility automation, planner, Gemini API client and UI#2
Alex72-py wants to merge 2 commits intomainfrom
codex/build-android-ai-agent-app-from-scratch-wjt8x9

Conversation

@Alex72-py
Copy link
Copy Markdown
Owner

Motivation

  • Add a self-contained Android app that provides an AI-driven accessibility automation agent capable of planning and executing UI actions and exposing a simple UI for configuration and status.
  • Integrate with a generative model (Gemini) to generate step plans, enforce rate limits, and provide recovery and safety checks for risky actions.
  • Provide core services required for running on-device: accessibility service, overlay, foreground notification, secure storage for API keys, and telemetry.

Description

  • Introduces a full :app Android module with Gradle configuration files (build.gradle.kts, settings.gradle.kts, gradle.properties, and wrapper) and Android manifest with permissions and services defined.`
  • Adds application scaffolding and UI: AiAgentApp, MainActivity, resources (layout, themes, strings, colors, launcher drawables) and a settings/ checklist screen with API key management.
  • Implements core automation engine components: AccessibilityAgentService, AgentOverlayService, AgentLoop, AdaptiveWaiter, ActionVerifier, TaskPlanner, GeminiApiClient (wraps the generative model client and prompt/parse logic), and RateLimiter for per-model throttling.`
  • Adds support utilities and models: UIExtractor, ElementRanker, ScreenshotProvider, SecurePrefs, NotificationHelper, Telemetry, TaskMemory, MacroManager, SafetyLayer, AppStateDetector, and data models in AgentModels.kt.
  • Adds accessibility configuration in res/xml/accessibility_service_config.xml and minimal proguard, resource icons and launcher configuration.

Testing

  • No automated unit or integration tests were added as part of this change.
  • No automated test suite was executed inside this rollout; please run ./gradlew build or ./gradlew assembleDebug locally or in CI to validate project compilation and resource linkage.

Codex Task

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: caefe49712

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread app/src/main/java/com/aquib/aiagent/agent/AgentLoop.kt
Comment on lines +27 to +29
val modelTier = if (complex) RateLimiter.Model.FLASH else RateLimiter.Model.FLASH_LITE
val allowed = rateLimiter.awaitAndAcquire(modelTier)
if (!allowed) throw IllegalStateException("Daily limit reached. Resets at midnight.")
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Avoid consuming quota before API key validation

getPlan acquires a rate-limit slot before checking whether active_api_key is present, so calls with no key still increment daily usage and eventually hit the hard limit even though no Gemini request is sent. In that state, users can start receiving Daily limit reached errors while only using fallback behavior.

Useful? React with 👍 / 👎.


val actionSucceeded = when (step.action) {
ActionType.TAP -> tapWithRetries(service, step.targetText, onLog)
ActionType.TYPE -> service.performType(step.value.orEmpty())
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve target field when executing TYPE steps

TYPE execution drops step.targetText and sends only the text value to performType, which then writes to the first editable node it finds. On screens with multiple inputs, this can type into the wrong field and derail the plan even when the model provided the intended target.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant