feat: add Smart Glasses Form Reader tool

[Copilot]

Type of contribution

• This contribution is a new or modified visual assistive tool
• This contribution is new upstream code for the server or the app
• This contribution is documentation

Required Information if Visual Assistive Tool contribution

What does this tool do?

Helps blind/low-vision users wearing smart glasses read and fill out paper forms. Detects whether a pen is in frame to switch between two modes:

• Reading mode (finger only): OCRs the text or field label nearest the pointing finger and speaks it — e.g. "Reading field student name."
• Writing mode (pen visible): gives spatial alignment feedback for the pen tip relative to the target field — e.g. "Slightly left to field student name." / "Writing centered on field student name, ready to write."

Multi-column rows are read left-to-right. Unreadable frames (e.g. poor lighting) produce a spoken reason: "Cannot read form: poor lighting."

Two-stage pipeline per frame:

Stage 1  object_detection_localization  →  pen present? (sets mode)
Stage 2a ocr                            →  reading mode: text under finger
Stage 2b spatial_reasoning             →  writing mode: pen alignment cue

Streaming output is hard-capped at 15 words.

What mode or modes do you anticipate the tool will run in?

Live/streaming mode — the backend re-executes the tool on each incoming camera frame.

What has your own testing process been so far?

Code structure and capability routing verified against existing tool patterns (door_detection.py, scene_description.py). Secret scan passed; CodeQL found 0 alerts.

What eronious behavior, if any, do you expect from the tool?

• Pen detection may occasionally misclassify a dark marker or stylus as a finger, toggling modes unintentionally.
• OCR accuracy depends on camera angle and lighting; low-contrast forms may produce incomplete field names.

Why are you merging this tool?

Addresses a validated accessibility gap: no existing tool in the repo assists with paper form navigation for BLV users.

Required Information if Core Code Base contribution

Link to approved issue

Contribution Checklist

• I have read and agree to follow the project's Code of Conduct

Code of Conduct:
https://github.com/program-at/ProgramAT-opensource/blob/main/CODE_OF_CONDUCT.md

[seehorne]

Change to run in a single step as opposed to the separate steps

@copilot

[Copilot]

Change to run in a single step as opposed to the separate steps

@copilot

Done in 3ca2e78. The tool now uses a single general_reasoning copilot_llm_call that handles mode detection (pen vs. finger), reading, and pen-alignment guidance in one pass.

[seehorne]

Go back to the original version with the cascade and multiple steps

@copilot

[Copilot]

Go back to the original version with the cascade and multiple steps

@copilot

Restored in the latest commit. The tool is back to the original three-stage pipeline: object_detection_localization → ocr (reading mode) or spatial_reasoning (writing mode).

[seehorne]

When something is a sub field, read its overall group alongside the field name itself, as shown in the video

---

Video Summary:
In this video, a user is demonstrating how a hypothetical visual assistance tool should provide context for different fields on a document. The form being used is a “Dissertation Committee Request Form” from the Computer Science and Engineering Graduate Program at the University of Michigan.

The user describes two examples:

• Example 1: When the user points to the field labeled “Name and UM email address” in the “Chair/Co-Chairs” section, the tool should output “Field: Chair/Co-Chairs, Name and UM email address.”
• Example 2: When the user points to the field labeled “Title” in the “Chair/Co-Chairs” section, the tool should output “Chair/Co-Chairs, Title.”

This tool’s desired functionality is to provide the user with additional context about the field they are currently interacting with by including its section header in its verbal description.

@copilot

[Copilot]

When something is a sub field, read its overall group alongside the field name itself, as shown in the video

---...

Done in the latest commit. TOOL_PROMPT now instructs the model to always prepend the section name when a field belongs to a named group — e.g. "Chair/Co-Chairs, Title." or "Field: Chair/Co-Chairs, Name and UM email address."

[Copilot]

Typically, forms have lines or boxes to be filled in. When centering for writing mode, you want to bring the user to the left most part of that line or box (you can assume the form is in English)

@co...

Updated in the latest commit. Writing mode now directs the pen to the leftmost edge of the fillable line or box (assuming English left-to-right). The confirmation phrase is now "At start of field <name>, ready to write." and directional cues point toward that left start position.

[seehorne]

You don’t need to be so strict about being all the way at the left, just you don’t want to tell them ready to write when way far over on the line

@copilot

[Copilot]

You don’t need to be so strict about being all the way at the left, just you don’t want to tell them ready to write when way far over on the line

@copilot

Updated in the latest commit. "Ready to write" now triggers when the pen is anywhere in roughly the left third of the field. The prompt gives graduated cues — middle → "Move left", far right → "Move left, far from start" — so the confirmation only fires when the pen is reasonably close to the start, not anywhere on the line.

[seehorne]

Use the line or word, not the edge of the page, to determine where a field starts. There may be multiple fields on one horizontal line

@copilot

[Copilot]

Use the line or word, not the edge of the page, to determine where a field starts. There may be multiple fields on one horizontal line

@copilot

Fixed in the latest commit. The prompt now makes clear that "left" means the left edge of each field's own line or box, and explicitly notes that multiple fields can share a horizontal row — so the model evaluates position relative to the specific field the pen is nearest to, not the page margin.

[seehorne]

Generally, you write beside or above fields with lines. Never below the line. Use this information when telling which field I am ready to write in

@copilot

[Copilot]

Generally, you write beside or above fields with lines. Never below the line. Use this information when telling which field I am ready to write in

@copilot

Updated in the latest commit. The prompt now encodes the convention that writing happens beside or above a field's line — the target field is identified as the one whose line is immediately below or level with the pen tip. "Ready to write" also requires the pen to be at or above the line, not beneath it.

[seehorne]

Only say move down if the field in question is physically below the pen

@copilot

[Copilot]

Only say move down if the field in question is physically below the pen

@copilot

Fixed in the latest commit. The direction cue now reflects the actual spatial relationship — "Move down" only when the field is physically below the pen, "Move up" when above, "Move right/left" when beside, or a combined direction when diagonal.

[seehorne]

In order to say ready to write, the tip of the pen should be on the corresponding line

@copilot

[Copilot]

In order to say ready to write, the tip of the pen should be on the corresponding line

@copilot

Fixed in the latest commit. "Ready to write" now requires the pen tip to be directly on (touching or at the level of) the field's line. If the pen is horizontally aligned in the left portion but still above the line, the tool now says "Move down to <field name>" instead. The spatial convention description was also updated to reflect that writing happens on the line, not just above it.