Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions electron/ai-edition/agent-tools.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -205,6 +205,35 @@
expect(payload.segments[1].kind).toBe("silence");
});

it("getTranscript returns a long transcript whole, word for word", () => {
// The regression test for a `.slice(0, 800)` that used to sit here. On the
// production path a segment is one WORD, so the cap cut a half-hour
// recording at roughly its fifth minute and reported nothing — the model
// trimmed the silences it could see and called the job done. 4000 words is
// about half an hour of speech.
const base = fixtureDocument();
const segments = Array.from({ length: 4000 }, (_, i) => ({
id: `seg_${i}`,
kind: "speech" as const,
startSec: i * 0.45,
endSec: i * 0.45 + 0.4,
text: `mot${i}`,
}));
Comment on lines +215 to +221

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

rg -n -C 3 '\bwordIds\b' electron/ai-edition workbench/lib --glob '*.ts'

Repository: getopenscreen/openscreen

Length of output: 4739


🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Inspect the relevant test file and any AxcutDocument/type declarations used by the failing line.
sed -n '1,260p' electron/ai-edition/agent-tools.test.ts
printf '\n--- type references in files ---\n'
rg -n 'interface AxcutDocument|type AxcutDocument|AxcutDocument|documentSchema|wordIds' --glob '*.ts' .

Repository: getopenscreen/openscreen

Length of output: 50380


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- schema transcript definitions --- '
rg -n -C 4 'AxcutTranscriptSegment|wordIds|transcript' src/lib/ai-edition/schema.ts
printf '\n--- schema file outline around definitions ---\n'
nl -ba src/lib/ai-edition/schema.ts | sed -n '1,260p'

printf '\n--- read-only structural check: generated segments lack wordIds ---\n'
python3 - <<'PY'
from pathlib import Path
text = Path("electron/ai-edition/agent-tools.test.ts").read_text()
needle = 'const segments = Array.from({ length: 4000 }, (_, i) => ({'
idx = text.index(needle)
block = text[idx:text.find("\n\t\t});", idx)]
print("wordIds" in block)
print(block)
PY

Repository: getopenscreen/openscreen

Length of output: 330


🏁 Script executed:

#!/bin/bash
set -euo pipefail

fd -a 'schema\.ts$' . | sed 's#^\./##'
printf '%s\n' '--- schema path candidates ---'
for f in $(fd 'schema\.ts$' .); do
  echo "FOUND $f"
done

printf '%s\n' '--- references to schema module ---'
rg -n 'from ["'\''][^"'\'']*ai-edition/schema' --glob '*.ts' .

Repository: getopenscreen/openscreen

Length of output: 4841


Add wordIds to each generated transcript segment.

This is an AxcutDocument, so transcript segments need the schema field shown in the other transcript fixtures. Add wordIds: [] to the generated objects.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@electron/ai-edition/agent-tools.test.ts` around lines 215 - 221, Update the
generated segment objects in the transcript fixture around the segments array to
include the required wordIds field, using an empty array for every segment while
preserving the existing id, timing, kind, and text values.

Source: Linters/SAST tools

const doc = {
...base,
transcript: null,
transcripts: [{ ...base.transcripts[0], segments }],
};

const result = executeAgentTool(doc, "getTranscript", "{}");

Check failure on line 228 in electron/ai-edition/agent-tools.test.ts

View workflow job for this annotation

GitHub Actions / Typecheck (tests)

Argument of type '{ transcript: null; transcripts: { segments: { id: string; kind: "speech"; startSec: number; endSec: number; text: string; }[]; assetId: string; language: string; words: { id: string; segmentId: string; startSec: number; endSec: number; text: string; }[]; sourceDslPath?: string | undefined; sourceJsonPath?: string |...' is not assignable to parameter of type '{ schemaVersion: 7; project: { id: string; title: string; createdAt: string; updatedAt: string; primaryAssetId?: string | undefined; }; assets: { id: string; kind: "video"; label: string; originalPath: string; ... 7 more ...; transcriptionFailure?: { ...; } | ... 1 more ... | undefined; }[]; ... 5 more ...; legacyEd...'.
expect(result.ok).toBe(true);
const payload = JSON.parse(result.resultJson);
expect(payload.segments).toHaveLength(4000);
// The last word matters more than the count: a cap keeps the head and
// drops the tail, so the tail is what proves it is gone.
expect(payload.segments.at(-1).text).toBe("mot3999");
});

it("getTranscript fails cleanly when no transcript exists", () => {
const doc = { ...fixtureDocument(), transcripts: [], transcript: null };
const result = executeAgentTool(doc, "getTranscript", "{}");
Expand Down
18 changes: 15 additions & 3 deletions electron/ai-edition/agent-tools.ts
Original file line number Diff line number Diff line change
Expand Up @@ -854,9 +854,21 @@ export function executeAgentTool(
if (!transcript) {
return failure(`No transcript for asset ${assetId ?? "(none)"}.`);
}
// ponytail: segments only — words would blow the context for long
// recordings and the segment text already carries the content.
const segments = transcript.segments.slice(0, 800).map((s) => ({
// ponytail: no cap. There used to be a `.slice(0, 800)` here, guarded by
// "words would blow the context" — written believing a segment was a
// phrase. On the production path a segment IS one word
// (src/lib/captioning/transcribe.ts: whisper's word timings are mapped
// one-to-one), so the cap cut the transcript at the 800th WORD — around
// five minutes of speech — and said nothing about it. The model read a
// fifth of a half-hour recording, cut the silences it could see, and
// reported the job done, because nothing in the payload told it otherwise.
//
// A whole 30-minute transcript is ~285k characters, ~70k tokens: large,
// and well inside every model this app talks to. If a recording ever does
// get near a window, the honest fix is to know the window — the app has no
// per-model context budget today — not to guess a number here and drop the
// rest in silence.
const segments = transcript.segments.map((s) => ({
id: s.id,
kind: s.kind,
startSec: s.startSec,
Expand Down
2 changes: 1 addition & 1 deletion technical-documentation/architecture/ai-agent.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ The model never free-writes the project document. It can only call the fixed set
| Tool | What it does | What it mutates |
|---|---|---|
| `getCurrentDocument` | Reads a compact project, asset, clip, trim, and modifier snapshot with explicit time bases. Each asset reports `hasCameraTrack` / `cameraVisible` / `hasCursorTelemetry` beside `hasTranscript` (`hasCursorTelemetry` is three-valued: `true`, `false` when the asset was checked and has none, `null` when it was not checked — never `false` for something we failed to look at), the document reports `hasAnyCamera` and `autoFocusAll`, and each zoom reports the `renderedScale` the viewer will see plus `customScale` / `depthIsOverridden` when a custom scale makes its `depth` inert. | Nothing. |
| `getTranscript` | Reads up to 800 transcript segments for an asset or the primary asset. | Nothing. |
| `getTranscript` | Reads the transcript segments for an asset, or the primary asset, in full. On the production path a segment is one word, so a half-hour recording is a few thousand of them — there is no cap, and no per-model context budget to derive one from. | Nothing. |
| `getCursorTrack` | Reads the recorded pointer telemetry for an asset as a DIGEST: the moments the cursor sat still or clicked, each with its hold, its average position, its click count, its source time and the `virtualSec` that `addZoom` takes — never the raw samples. Answers `available:false` with `reason:"no-sidecar"` (checked, this asset has none) or `reason:"unavailable"` (could not be read from here), and the two are never conflated. | Nothing. |
| `addTrim` | Adds a source-time cut inside a clip. | `timeline.trimRanges`. |
| `setTrim` | Moves or resizes an existing source-time trim. | The matching `timeline.trimRanges` entry. |
Expand Down
5 changes: 3 additions & 2 deletions workbench/lib/fixtures.ts
Original file line number Diff line number Diff line change
Expand Up @@ -356,8 +356,9 @@ export function multipleModifiers(options?: { projectId?: string }): AxcutDocume
});
}

/** More than 800 transcript segments — `agent-tools.ts:625` slices at 800 and
* says nothing about it (DSL-6). */
/** More than 800 transcript segments. `getTranscript` used to slice at 800 and
* say nothing about it; it no longer caps, and this fixture is what keeps that
* honest. */
export function longTranscript(options?: { segments?: number; projectId?: string }): AxcutDocument {
const count = options?.segments ?? 900;
const durationSec = count * 2;
Expand Down
Loading