AI Server Protocol

AI Server Protocol

Exactly what SenseTree sends over the network, to whom, and with which parameters. If you run your own inference server, this page is the contract you have to satisfy.

Everything here is inference traffic to endpoints you configured yourself. See What leaves the machine at the bottom for the complete outbound list, including the non-inference metadata calls the model catalog makes.

Ground rules

• Protocol: HTTP/1.1 + JSON, OpenAI-compatible shapes. Any server speaking that dialect works — Ollama, LM Studio, vLLM, llama.cpp server, LocalAI, a gateway, or a hosted API.
• base_url includes the version prefix. You configure http://localhost:11434/v1; SenseTree appends /embeddings, /chat/completions, etc. A trailing / is trimmed.
• Auth: if api_key is non-empty, Authorization: Bearer <api_key>. If empty, no header at all — most local servers reject an empty bearer.
• No streaming for inference. Every completion request carries "stream": false and is parsed as a single JSON body. (The only streamed responses SenseTree reads are Ollama's NDJSON pull progress and MCP's SSE.)
• Two native, non-OpenAI escapes, both Ollama-only and both optional — the app degrades to the standard path if they fail: /api/embed (to bound context) and /api/generate with keep_alive: 0 (to free VRAM). See Ollama native calls.

Timeouts

Call	Timeout	Why
Embeddings, reasoning chat, agent chat	120 s (shared client)	Long enough for a cold model load on a busy GPU.
Vision (describe_image)	300 s	A multimodal model can take tens of seconds just to swap into VRAM.
Transcription	transcription.timeout_secs — 1800 s default	Whisper on CPU is often slower than real time.
Video description	video.timeout_secs — 1800 s default	Same reasoning, larger payloads.
Health ping (GET /models)	5 s	A liveness check, not a request.
Model unload, GET /api/ps	10 s	Fire-and-forget housekeeping.
Settings "Test connection"	6 s (chat) / 20 s (embedding)	Interactive: must answer fast.
Model pull (/api/pull)	3600 s	Downloading tens of gigabytes.
MCP tool calls	20 s	An external tool that hangs must not hang the chat.

Call map

#	Purpose	Request	Slot / config
1	Liveness + model list	GET {base}/models	reasoning, vision
2	Vectorize text	POST {base}/embeddings	embedding (mode: openai)
2b	Vectorize with bounded context	POST {root}/api/embed	embedding, Ollama only
3	Qualify / classify / plan	POST {base}/chat/completions	reasoning
4	Agent turn (tools)	POST {base}/chat/completions + tools	reasoning
5	Caption an image, OCR a page	POST {base}/chat/completions + image_url	vision
6	Transcribe speech	POST {base}{endpoint_path} (multipart)	transcription
7	Describe a video	POST {base}{endpoint_path} + video_url	video
8	Free VRAM / observe / install	/api/generate, /api/ps, /api/pull, /api/delete	Ollama only

---

1. Health check — GET {base}/models

Fired periodically by the header's AI-health indicator, and by Test connection in Settings. Reasoning and vision slots only; the embedding slot is reported from config without touching the network (loading the local model just to check it would keep it resident and spinning).

GET /v1/models HTTP/1.1
Authorization: Bearer <api_key>        # only if non-empty

Any 2xx counts as healthy. The model list is parsed from data[].id, used to populate the dropdowns and to verify a configured model exists — with tolerant matching, so llama3.1:8b matches llama3.1.

2. Embeddings — POST {base}/embeddings

Fired during indexing (documents) and on every search (query). Only in embedding.mode = "openai"; in local mode nothing leaves the process.

POST /v1/embeddings HTTP/1.1
Content-Type: application/json
Authorization: Bearer <api_key>        # only if non-empty

{
  "model": "<embedding.model>",
  "input": ["passage 1", "passage 2", "..."]
}

Expected response — only data[].embedding is read, in order:

{ "data": [ { "embedding": [0.013, -0.221] }, { "embedding": [0.004, 0.187] } ] }

Batching. input never holds more than indexing.batch_size texts (default 32). A file producing more chunks is split into several sequential requests. This is not cosmetic: a 1.7 MB log file produced a single request of ~1750 texts that servers refused, which failed the file, retried it, and stalled the whole queue.

Dimensions. SenseTree does not read a dimension from the response — the LanceDB table is built from embedding.dimensions in your config. If they disagree, upserts fail with a dimension mismatch. Test connection on the embedding slot embeds "test" and reports the real vector length so you can align the field.

Prefixes. In local mode, E5-family models get passage: / query: prefixes automatically. In openai mode SenseTree sends your text as-is — if your served model needs prefixes, the server must add them.

2b. Ollama native fast path — POST {root}/api/embed

Tried first, and only when both conditions hold: base_url ends with /v1 and api_key is empty (i.e. it looks like a local Ollama). {root} is base_url minus the /v1 suffix.

POST /api/embed HTTP/1.1
Content-Type: application/json

{
  "model": "<embedding.model>",
  "input": ["passage 1", "passage 2"],
  "options": { "num_ctx": 2048 }
}

Why bother: the OpenAI-compatible endpoint ignores num_ctx (verified), and Ollama allocates the KV cache for the full advertised context. Measured on a real server with a 32k-context embedding model: 5.78 GB of VRAM at the default context vs 2.13 GB at 2048 — 3.65 GB reserved for a window a chunk never uses.

num_ctx is computed from your chunk size, not hardcoded:

num_ctx = max(2048, ceil((chunk_size + 250) / 2)), rounded up to a multiple of 1024

(+250 for the contextual-retrieval prefix; /2 is a pessimistic 2 characters per token.) With the default chunk_size: 1000 this gives 2048.

Fallback is silent and total. Non-2xx, unparseable body, or a count of embeddings that doesn't match the count of inputs → SenseTree discards the attempt and re-issues the request on /embeddings. A non-Ollama server behind a /v1 URL therefore costs one wasted round-trip per batch, nothing more.

3. Reasoning chat — POST {base}/chat/completions

The workhorse: document qualification, folder classification, folder descriptions, context guessing, unknown-file extraction, and plan_reorganization.

POST /v1/chat/completions HTTP/1.1
Content-Type: application/json
Authorization: Bearer <api_key>        # only if non-empty

{
  "model": "<reasoning.model>",
  "messages": [
    { "role": "system", "content": "<the prompt for this task>" },
    { "role": "user",   "content": "<file name, folder, type, excerpt...>" }
  ],
  "temperature": 0.2,
  "stream": false,
  "response_format": { "type": "json_object" },
  "reasoning_effort": "none"
}

Only choices[0].message.content is read. A response without it is an error.

Field	When it appears
response_format	Only where the answer is parsed as strict JSON: plan_reorganization and the folder classifier. Omitted elsewhere.
reasoning_effort	Omitted entirely when the slot is set to Auto (the default), so servers that never heard of it behave exactly as before. Otherwise "none", "low", "medium" or "high".
temperature	Always 0.2 for reasoning, 0.1 for vision and video. Not configurable.

Reasoning effort is per-use, not global. Indexing qualifications default to "none" (indexing.qualify_effort) because they are thousands of calls whose answer is a handful of tokens; chat and action planning stay on Auto. Measured on a real server, classifying one folder: 24.4 s with reasoning vs 0.78 s without, identical answer — enough to time out in a loop and block indexing entirely.

4. Agent turn — POST {base}/chat/completions with tools

Same endpoint, plus native function-calling. Used only by the chat agent (see AI Chat & Agent).

{
  "model": "<reasoning.model>",
  "messages": [],
  "temperature": 0.2,
  "stream": false,
  "tools": [
    { "type": "function",
      "function": {
        "name": "search_files",
        "description": "Semantic hybrid search over the user's indexed files...",
        "parameters": { "type": "object",
                        "properties": { "query": { "type": "string" } },
                        "required": ["query"] }
      } }
  ],
  "tool_choice": "auto"
}

• tools and tool_choice are omitted when the array is empty — some servers reject an empty array.
• The response's choices[0].message is re-injected verbatim into the next request's messages, followed by one message per tool result:

  { "role": "tool", "tool_call_id": "<id from the model>", "content": "<observation text>" }

• function.arguments is read as a JSON string (per the spec), but a raw object is also accepted — servers differ.
• A model that cannot call tools simply returns content: the loop ends and the pre-RAG context still grounds the answer. Graceful degradation, no error.

5. Vision — POST {base}/chat/completions with an image

Fires for image files (caption) and for scanned-PDF pages (OCR). One request per image or page.

{
  "model": "<vision.model>",
  "messages": [{
    "role": "user",
    "content": [
      { "type": "text", "text": "<vision_caption or vision_ocr prompt>" },
      { "type": "image_url",
        "image_url": { "url": "data:image/jpeg;base64,/9j/4AAQSkZJRgABA..." } }
    ]
  }],
  "temperature": 0.1,
  "stream": false,
  "reasoning_effort": "low"
}

• The image is always inlined as a data URL, never a link — the server needs no filesystem access.
• MIME type comes from magic bytes, defaulting to image/png. PDF pages are rendered to JPEG at ~1600 px on the long side, quality 82.
• Only these raster formats are sent: jpg, jpeg, png, webp, gif, bmp. .svg, .ico, .cur are indexed by context instead — they used to come back as 400 invalid image input.
• The vision slot's own reasoning effort is honoured here. It matters: on a real image, 32.5 s with reasoning vs 6.9 s without, for 7 306 characters of thinking preceding a 226-character answer.

6. Transcription — POST {base}{endpoint_path} (multipart)

endpoint_path defaults to /audio/transcriptions. Everything that varies between servers is configurable, because the app deliberately assumes nothing about yours.

POST /v1/audio/transcriptions HTTP/1.1
Content-Type: multipart/form-data; boundary=...
Authorization: Bearer <api_key>        # only if non-empty
Content-Length: <exact>

--boundary
Content-Disposition: form-data; name="file"; filename="team meeting.mp3"
Content-Type: audio/mpeg
<raw bytes, streamed>
--boundary
Content-Disposition: form-data; name="model"

Systran/faster-whisper-large-v3
--boundary
Content-Disposition: form-data; name="language"

fr
--boundary
Content-Disposition: form-data; name="response_format"

verbose_json
--boundary--

Part	Source	Sent when
file	the media file itself, streamed	always
model	transcription.model	always
language	transcription.language (ISO-639-1)	only if non-empty
response_format	transcription.response_format	only if non-empty
anything else	transcription.extra_fields, a JSON object {"key": "value"}	one part per key

• Empty fields are omitted, never sent empty. Some servers reject language="", and their own default beats ours.
• The body is streamed with an exact Content-Length: a multi-gigabyte file is never held in memory, so media size is not bounded by RAM. transcription.max_file_mb defaults to 0 = no limit; the cap exists for people billed per minute.
• The filename is transmitted — several servers pick their demuxer from it.
• An unknown MIME type does not block the upload: if the HTTP client refuses the type string, the file is re-sent without a Content-Type and the server decides.
• Response: {"text": "..."} or raw text. Both are accepted; JSON without a text field is kept verbatim rather than discarded.

extra_fields is the escape hatch for servers expecting something exotic (temperature, prompt, diarize…). Invalid JSON there is ignored with a warning rather than failing the whole indexing run.

7. Video description — POST {base}{endpoint_path} with video_url

endpoint_path defaults to /chat/completions. Complementary to transcription: it reports what the video shows, where transcription reports what is said.

{
  "model": "<video.model>",
  "messages": [{
    "role": "user",
    "content": [
      { "type": "text", "text": "<video_describe prompt>" },
      { "type": "video_url", "video_url": { "url": "<see delivery below>" } }
    ]
  }],
  "temperature": 0.1,
  "stream": false
}

Two delivery modes (video.delivery):

Mode	url value	Trade-off
base64 (default)	data:video/mp4;base64,AAAAIGZ0...	Universal — every compatible server reads it. The JSON body is assembled around a base64 stream (prefix, streamed file, suffix) with an exactly computed Content-Length, so the video never enters memory and servers that refuse chunked encoding still work.
file_uri	file:///C:/Users/.../film.mp4	Nothing transits at all: the server opens the file itself. By far the most efficient, but requires a server seeing the same filesystem and configured to allow local media (vLLM: --allowed-local-media-path).

The model name, prompt and MIME type are escaped through the JSON serializer even in the hand-assembled streaming path, so a quote inside your prompt cannot break the body.

Ollama native calls

These four only make sense against Ollama and are best-effort everywhere: a server that doesn't implement them returns an error that SenseTree logs at debug level and ignores.

Call	Request	When
Unload	POST {root}/api/generate with {"model": "...", "keep_alive": 0}	On app shutdown (Ollama is a separate process and would otherwise keep gigabytes resident), and in batch pipeline mode between the LLM phase and the embedding phase.
Observe	GET {root}/api/ps	The Settings panel showing what is really loaded, its size, VRAM share and expiry. Ollama exposes no configuration API — so SenseTree observes instead of assuming.
Pull	POST {root}/api/pull with {"name": "...", "stream": true}	Catalog download button. Reads Ollama's NDJSON progress and re-emits it to the UI as model-pull-progress events.
Delete	DELETE {root}/api/delete with {"model": "..."}	Catalog delete button.

{root} is base_url with a trailing /v1 removed — only a trailing one, so http://api.example.com/v1/proxy is left intact.

For LM Studio (detected from the URL), install and delete go through the lms CLI instead, since it exposes no HTTP download API.

Errors and retries

Failures are classified, because "the server is busy" and "this file will never work" deserve opposite treatments:

Class	Triggered by	Consequence
Transient	Network error, timeout, 5xx, 408, 429	The task is re-queued. Up to 3 attempts per file; only on the last one does it fall back to contextual indexing.
Permanent	Any other 4xx (invalid media, rejected format, unknown model)	Immediate contextual fallback. Retrying would only reproduce it.

This is why a vision model swapping in and out of a shared GPU no longer permanently downgrades an image — see Troubleshooting.

What leaves the machine

Inference traffic goes only to the base_urls you set. In the default configuration that is localhost, and the app is fully offline.

Two other categories of traffic exist, both unrelated to your files:

Model provisioning (on demand, once):

• ONNX Runtime 1.20.0 (CPU or GPU build) from github.com/microsoft/onnxruntime, on first use of the local engine.
• Embedding / reranker / CLIP weights from Hugging Face, cached in %APPDATA%\com.virgi.sensetree\models.
• App updates from github.com/Eligrive/SenseTree/releases (signature-verified; see Installation).

Model catalog metadata (cached, TTL in brackets):

• mteb-leaderboard-backend.hf.space — embedding benchmarks (7 days).
• opencompass.openxlab.space and opencompass.oss-cn-shanghai.aliyuncs.com — vision and reasoning benchmarks (7 days).
• ollama.com/library — the live Ollama library and per-model tags (24 h).
• huggingface.co/api — GGUF repository resolution for install names (7 days).

None of these carry file names, contents, queries, or identifiers. They are plain public GETs, they answer from cache when offline, and the catalog degrades to stale-but-usable data if they can't be reached.

MCP servers you configure yourself are the remaining outbound path — see MCP Servers. Note that tool arguments are chosen by the model, so an MCP server receives whatever the agent decides to pass it.