Add physical interface accessibility aid tool#120
Conversation
…ove test coverage
|
Occasionally it will tell me no tool output when there should be some degree of tool output according to server logs, why does this happen? Also, when I am off to the side of a button it just tells me I’m on that button instead of giving direction I need to move to get there |
- Add REPEAT_INTERVAL=10: re-announce guidance every 10 frames even when the scene is stable, fixing the "no tool output" symptom where server logs show a valid LLM response but TTS stays silent - Revise Stage 2 spatial_reasoning prompts to explicitly distinguish "finger DIRECTLY ON (overlapping) a button" vs "finger NEAR/ADJACENT but not overlapping", fixing the false "on the button" confirmations when the user is off to the side of a button - Revise Stage 3 navigation prompt to prioritise directional guidance over on-button confirmation unless Stage 2 explicitly says the finger overlaps the button - Add test_periodic_repeat_re_announces_at_interval test - Add test_module_has_repeat_interval_constant check
Both issues are addressed in the latest commit. No output despite valid server logs: The streaming deduplication check was returning "On the button" when off to the side: The Stage 2 spatial-reasoning prompt previously asked for "which button the finger is touching or nearest to", which conflated proximity with contact. Stage 3 also listed the "on the button" confirmation before directional guidance, biasing the model toward it. Both prompts are revised: Stage 2 now explicitly asks whether the finger directly overlaps the button area vs is merely nearby/adjacent, and Stage 3 now prioritises directional guidance unless Stage 2 explicitly reports an overlap. |
|
Since this tool is for a blind user, instructions that say things like “touch the nearest button” or “touch the red start button” aren’t helpful. You need to either tell them a button they are already touching or give them directions (ex: move left, move right, move up, move down) to reach the nearest button |
…rompt The Stage 3 nav prompt now requires exactly one of two forms: - "Your finger is on the [label] button." (finger overlapping) - "Move [direction] to the [label] button." (finger nearby) Explicitly bans 'touch', 'tap', 'press', 'reach', 'find', and 'locate'. Updates the copilot_llm_call goal string to match.
…e 3 prompt - Promote equidistant-button guidance to an explicit third allowed form (C) to avoid contradicting the "EXACTLY ONE of these forms" constraint - Expand banned-verb list to include gerund forms: touching, tapping, pressing, reaching, finding, locating - Update goal string to match three-form contract
Fixed. The Stage 3 navigation prompt now enforces exactly three output forms and nothing else:
The verbs |
|
Restrict responses to the form of “your finger is on [insert interface element]” when they are on a specific button , “move [left, right, up, or down] towards [interface element]” when they are not on a button but near one, or “[interface element a] is slightly [above/below, left/right] of your finger, [interface element b] is slightly [opposite direction of previous] your finger” when they are equidistant to two elements |
- Form A: 'your finger is on [element]' - Form B: 'move [direction] towards [element]' (cardinal only: left/right/up/down) - Form C: '[element a] is slightly [direction] of your finger, [element b] is slightly [opposite direction] your finger' - Restrict directions to left/right/up/down only (no diagonals) - Update test mock strings to reflect new forms
'[element b] is slightly [direction] of your finger' now consistent with '[element a] is slightly [direction] of your finger' in both navigation prompt variants and the goal string.
Done. The Stage 3 navigation prompt now enforces exactly these three forms:
Committed in |
- Add _fix_diagonals(): replaces diagonal directions (up-left, up-right, down-left, down-right, upper/lower variants) with cardinal equivalents - Add _enforce_output_format(): applies _fix_diagonals then suppresses responses containing banned verbs (touch/tap/press/reach/find/locate) - Apply _enforce_output_format() in main() after the word-cap truncation so format restrictions hold regardless of backend planner/routing state - Add 22 new tests covering helpers and end-to-end enforcement via main()
- Restore missing `class TestPhysicalInterfaceAidReturnTypes` declaration that was lost during insertion of TestOutputFormatEnforcement - Simplify _replace inner function: combine paired conditions into `if g1 or g2: return 'up'` / `if g3 or g4: return 'down'`
…lity
Single pattern with named groups 'vert' and 'horiz' replaces four
separate alternation groups. _replace() now reads `m.group("vert")`
instead of unpacking four captured groups.
|
Stop describing elements that are not related to what the user is touching. For example, if a user is navigating a microwave and they are touching the power level button, you should not also include the time on the microwave display as it is not relevant |
Fixed in the latest two commits. Stage 1 now explicitly excludes non-interactive elements — the system prompt says "do NOT include displays, screens, digital readouts, clocks, timers, or status indicators; list only elements the user can physically activate." The goal string repeats the same exclusion. Stage 2 now asks for the single nearest button only, not an enumeration of all buttons, so irrelevant context doesn't propagate forward. Stage 3 now carries an explicit rule in both the system prompt and the goal: "do NOT mention other buttons, display elements, status readouts, or any part of the interface not immediately relevant to where the finger is or needs to go." The |
|
Limit responses to 10 words. Emphasize the interface elements they are interacting with and do not describe surrounding context. Also respond at most every 1.5 seconds, not faster |
Done in the latest commit.
|
|
@copilot do this all in one step, no step 1, step 2, step 3. enforce all rules at every step |
Done. The 3-stage pipeline is replaced with a single |
|
Put your prompt to the model in a variable called |
Done. The system prompt is now stored in a module-level |
Live-mode tool that helps blind users locate and navigate buttons on physical interfaces (microwave keypads, thermostats, kiosks, checkout terminals) via real-time directional audio guidance.
Type of contribution
Required Information if Visual Assistive Tool contribution
What does this tool do?
Identifies physical interface elements from the camera frame, detects the user's finger position, and speaks directional guidance to move their finger to the correct button — e.g. "Move right to the Start button" or "Your finger is on the 7 button". Uses a 3-stage LLM pipeline:
structured_visual_understanding— maps the interface type and all visible button labels/positions; notes finger visibilityspatial_reasoning— determines which button the finger is nearest to and the direction needednavigation— produces the final spoken instructionResponses are capped at 15 words (enforced in code and via system prompt). Returns
""when the scene is unchanged to prevent repetitive TTS. Handles lighting uncertainty, button occlusion, and ambiguous adjacent buttons.What mode or modes do you anticipate the tool will run in?
Live mode — every ~1 second, or immediately on scene change.
What has your own testing process been so far?
Unit tested with mocked
copilot_llm_callcalls covering: streaming deduplication, code-enforced word truncation (25-word input → ≤15 words), empty response passthrough, exception error-dict path, and word-boundary error message truncation. Guardrail validation (validate_generated_tools.py) passes cleanly.What eronious behavior, if any, do you expect from the tool?
Why are you merging this tool?
Addresses a concrete navigation barrier: blind users cannot independently interact with unlabeled or inaccessible physical interfaces. This tool fills that gap with interface-agnostic, real-time audio guidance.
Required Information if Core Code Base contribution
Link to approved issue
Contribution Checklist
Code of Conduct:
https://github.com/program-at/ProgramAT-opensource/blob/main/CODE_OF_CONDUCT.md