Media Audio and Video

Media: Audio & Video

A recording is a document you can't read. SenseTree gives it two independent sources of meaning — what is said and what is seen — and then treats the result exactly like any other document: chunked, qualified, keyword-indexed, contextually enriched.

Both are opt-in and disabled by default, because each assumes a server you have to run.

	Transcription	Video description
Answers	What is said	What is shown
Applies to	Audio and video	Video only
Endpoint	POST {base}/audio/transcriptions (multipart)	POST {base}/chat/completions (video_url part)
Typical server	speaches, whisper.cpp, LocalAI, vLLM, OpenAI/Groq	vLLM serving Qwen-VL or similar
Config section	transcription	video

Ollama does neither. Its default port is deliberately not used for these slots — the defaults point at http://localhost:8000/v1 (speaches; whisper.cpp listens on 8080).

What gets indexed

If both are enabled on a video, the two outputs are concatenated:

Description visuelle : <what the model saw>

Transcription : <what was said>

That text then goes through the standard document path — chunking, LLM qualification (gated by qualify_media), BM25 indexing, contextual retrieval. The consequence matters: an hour-long recording is searchable on any of its passages, not just on a summary. Ask for a phrase from the middle of a meeting and you land on the right file.

If only one is enabled, only that one runs. If neither produced anything usable, the file falls back to contextual indexing — name, folder, neighbours — so it never vanishes from the index.

Which files are routed here

Roughly 40 audio and video container extensions (mp3, wav, flac, m4a, opus, mp4, mkv, mov, webm, avi…), plus anything whose magic bytes say audio/* or video/*.

The app never decides what your server can read. A routed file is sent as-is; the server accepts or refuses. A refusal is a permanent error and the file falls back to context — it does not block the queue. Video-vs-audio is determined by MIME first, extension second, and only decides whether a visual description is worth attempting.

.ts is excluded from the extension list on purpose: it is TypeScript far more often than MPEG-TS, and shipping source code to a transcription server would be a worse failure than missing a rare video. Genuine MPEG-TS files are still caught by their magic bytes.

Configuring transcription

Settings → Transcription (audio / video)

Field	Default	Notes
base_url	http://localhost:8000/v1	speaches' port. whisper.cpp: :8080.
model	Systran/faster-whisper-large-v3	A locally served model id. whisper-1 is the hosted-OpenAI identifier and only works there.
enabled	false	Opt-in.
endpoint_path	/audio/transcriptions	For servers that deviate from the convention.
language	(empty)	ISO-639-1 (fr, en…). Empty = auto-detect. Setting it noticeably improves speed and quality on a monolingual corpus.
response_format	(empty)	Empty = the server's own default. Set to text, json, verbose_json, srt… if yours requires it.
extra_fields	(empty)	JSON object {"key": "value"}, sent as extra multipart parts. The escape hatch for exotic parameters (temperature, prompt, diarize…). Invalid JSON is ignored with a warning rather than failing indexing.
max_file_mb	0	0 = no limit. The upload is streamed, so size isn't bounded by RAM. The cap exists for metered APIs.
timeout_secs	1800	Whisper on CPU is often slower than real time.

Responses are accepted as {"text": "..."} or as raw text — several local servers do the latter, and losing a successful transcription over a serialization detail would be absurd.

Configuring video description

Settings → Video description (image)

Field	Default	Notes
base_url	http://localhost:8000/v1	e.g. vLLM.
model	Qwen/Qwen2.5-VL-7B-Instruct	Must actually read video, not just images.
enabled	false	Opt-in.
endpoint_path	/chat/completions
delivery	base64	How the video reaches the server — see below.
max_file_mb	0	0 = no limit, same streaming rationale.
timeout_secs	1800

Delivery modes

base64 (default, universal). The video is inlined as data:video/mp4;base64,…. The JSON body is assembled around a stream of the file — prefix, streamed base64, suffix — with an exactly computed Content-Length. The video is therefore never held in memory, and servers that refuse chunked encoding still work. Cost: the payload is ~4/3 the file size, over the network.

file_uri (fastest). The URL is file:///C:/Users/…/film.mp4 and nothing transits — the server opens the file itself. Requires a server on the same machine (or seeing the same filesystem) and configured to allow local media, e.g. vLLM's --allowed-local-media-path. On a local setup this is by far the better choice.

Cost control

Transcription is the single most expensive call in the pipeline. Three things keep it in check:

• Content hashing. An unchanged media file is never re-transcribed. Hashes are computed in 16 KB blocks, so a multi-gigabyte file never enters memory.
• qualify_media. Separate from qualify_documents on purpose: an hour-long transcript is long and costly to qualify, and you may want to skip that step without touching your PDFs.
• max_file_mb on either slot, if you're billed per minute or want to skip your movie collection.

Also worth knowing: batch pipeline mode groups all LLM work for a slice of files before switching to embedding, which avoids reloading a large multimodal model once per file. See Indexing Pipeline.

Failure handling

Situation	Result
Server refuses the format (4xx)	Permanent — logged, contextual fallback, no retry.
Server busy, timeout, 5xx, 408, 429	Transient — retried, up to 3 attempts, then contextual fallback.
Empty transcription	Not an error. The media has no speech (music, ambience, silence). If the video description produced something, that alone is indexed.
Something succeeded, something else failed transiently	The successful part is indexed rather than replaying everything to complete it.

Verifying your setup

1. Enable the slot, set base_url and model, save. Each media slot has its own indicator in the sidebar's AI status panel, so you can see immediately whether the server you just plugged in answers — and which of the two is down when one is.
2. Drop one short file into an indexed folder.
3. Watch the Indexing queue modal — a media file shows a media stage before reasoning and embedding.
4. Open the file's detail drawer: the extract should hold the transcription and/or description, and the sense its qualification.

The logs (run the app from a terminal) print 🎧 N characters transcribed or 🎬 video described, and the exact error otherwise.

A media indicator reading "connected (no /models inventory)" is not a problem: several transcription servers, whisper.cpp included, simply don't expose /models. The 404 proves the server is answering, so it counts as reachable.

The precise wire format for both calls is in AI Server Protocol.