You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Cursor overlay: follow the element under the cursor, add a text image, and vary where it lands
#687
I record product videos with the cursor overlay and custom images (--image, --pointer-image). The purpose of this proposal is to make the drawn cursor move and look more like a person's: a viewer follows a product video more naturally when the pointer behaves the way their own does. Two things currently make it read as scripted. I've built both on top of 1.3.1 and would like to know whether you want them before I open pull requests.
The clip is one scripted take on macOS with both features on. The cursor lands off-center on the picture button as a hand, moves off after the click and becomes the arrow, shows the I-beam in the text field, and is a hand again on Continue.
1. Show the image for what is under the cursor
Today on macOS the pointer image appears when a travel arrives at a pressable control, and the cursor returns to the arrow after the action's effect. Text fields keep the arrow.
A system cursor behaves differently. It stays a hand while it rests on a button, becomes an I-beam over a text field, and turns back into an arrow when a click opens a dialog underneath it.
The proposal:
--text-image PATH and --text-hotspot X,Y add a third image, shown over a text field, combo box, or date field.
After each move and effect, the macOS renderer hit-tests the element under the resting cursor, walks up to the nearest pressable or text ancestor, and shows the image for it. It checks 0.35 s, 1 s, and 2.5 s after the action, so it also catches a view that appears late.
The instruction carries a shape (arrow, pointer, or text) in place of the pointer flag. Windows reads the shape and keeps its current behavior, showing the arrow and the pointer image.
2. Vary the landing point and move off a clicked control
Today every ref action lands on the exact center of its control, and the cursor stays there after the click. A person rarely hits the exact center, and usually moves the pointer away once the click has registered.
The proposal adds two motion settings to cursor-overlay enable, both off by default and macOS only:
--aim-spread N, from 0 to 1, lands the cursor at a point inside the central part of the control, weighted toward the middle and different on every move. A cursor that already rests in that area clicks where it is.
--drift-off moves the cursor off a clicked control 0.4 to 0.9 s after the click, unless the next action arrives first.
Both affect the drawn cursor only. A physical click still lands on its verified point, and both draw from --motion-seed when it is set, so a recording stays reproducible. The travel instruction carries the control's bounds in a new landing field only when aiming is configured, so nothing changes on the wire otherwise.
Implementation
Both are on one branch, as two commits on top of main: https://github.com/ok/agent-desktop/tree/feat/cursor-shape-and-aim. The first two commits are these features. The third is the Windows fix in #686. CI passes on my fork for macOS, Linux, Windows, and Windows ARM64. I've run fifteen scripted takes with the branch on macOS, some with three cursors at once, and checked on a Windows 11 machine that the pointer image still appears on arrival.
Questions
Do you want either feature in agent-desktop? I can open them as two separate pull requests, the second stacked on the first.
Is replacing the pointer flag with shape acceptable, given that an old renderer has to be restarted after an upgrade anyway, or would you rather keep pointer and add a separate field?
For aiming, should Windows ignore the settings as it does the other motion settings, or would you want it implemented there before this merges?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
I record product videos with the cursor overlay and custom images (
--image,--pointer-image). The purpose of this proposal is to make the drawn cursor move and look more like a person's: a viewer follows a product video more naturally when the pointer behaves the way their own does. Two things currently make it read as scripted. I've built both on top of 1.3.1 and would like to know whether you want them before I open pull requests.The clip is one scripted take on macOS with both features on. The cursor lands off-center on the picture button as a hand, moves off after the click and becomes the arrow, shows the I-beam in the text field, and is a hand again on Continue.
1. Show the image for what is under the cursor
Today on macOS the pointer image appears when a travel arrives at a pressable control, and the cursor returns to the arrow after the action's effect. Text fields keep the arrow.
A system cursor behaves differently. It stays a hand while it rests on a button, becomes an I-beam over a text field, and turns back into an arrow when a click opens a dialog underneath it.
The proposal:
--text-image PATHand--text-hotspot X,Yadd a third image, shown over a text field, combo box, or date field.shape(arrow,pointer, ortext) in place of thepointerflag. Windows reads the shape and keeps its current behavior, showing the arrow and the pointer image.2. Vary the landing point and move off a clicked control
Today every ref action lands on the exact center of its control, and the cursor stays there after the click. A person rarely hits the exact center, and usually moves the pointer away once the click has registered.
The proposal adds two motion settings to
cursor-overlay enable, both off by default and macOS only:--aim-spread N, from 0 to 1, lands the cursor at a point inside the central part of the control, weighted toward the middle and different on every move. A cursor that already rests in that area clicks where it is.--drift-offmoves the cursor off a clicked control 0.4 to 0.9 s after the click, unless the next action arrives first.Both affect the drawn cursor only. A physical click still lands on its verified point, and both draw from
--motion-seedwhen it is set, so a recording stays reproducible. The travel instruction carries the control's bounds in a newlandingfield only when aiming is configured, so nothing changes on the wire otherwise.Implementation
Both are on one branch, as two commits on top of
main: https://github.com/ok/agent-desktop/tree/feat/cursor-shape-and-aim. The first two commits are these features. The third is the Windows fix in #686. CI passes on my fork for macOS, Linux, Windows, and Windows ARM64. I've run fifteen scripted takes with the branch on macOS, some with three cursors at once, and checked on a Windows 11 machine that the pointer image still appears on arrival.Questions
pointerflag withshapeacceptable, given that an old renderer has to be restarted after an upgrade anyway, or would you rather keeppointerand add a separate field?All reactions