Releases: anishmoncivarghese/docsift
Release list
v0.5.7
-
PowerPoint answers can now be cited by slide. A deck's chunks came back
with no attribution at all — no slide number, no title, and MarkItDown's raw
slide markers sitting in the chunk text — because the pipeline recognises
<!-- page: N -->and MarkItDown writes<!-- Slide number: N -->, so the
marker was never seen as one and fell through as ordinary paragraph text.Translating it at the engine boundary fixed the attribution but not the
citation: slide content is sparse, so the token-budgeted chunker packed 55 of
a 60-slide deck into one chunk. Accurate, and no use to anyone. A deck is now
chunked one slide at a time, the same way Docling supplies its own chunks for
PDFs, so 60 slides yield 60 chunks each naming its slide and title. An
oversized slide still splits normally.Speaker notes were already extracted and are searchable. Diagrams are not: a
slide with three boxes and two arrows gives three labels, and the arrows are
lost. Images inside slides are not read at all. Both are documented.
v0.5.6
-
Listed in the official MCP Registry. Adds
server.json(validated against
the published2025-12-11schema) and themcp-name:ownership marker the
registry checks against the README published on PyPI — which is why this
needed a release rather than just a commit.The listing installs with
uvx --from "docsift[mcp,docling,markitdown]" docsift mcp, so the extras come along; a bareuvx docsiftwould install the
CLI without an MCP SDK.
v0.5.5
Documentation only; no code changes. Released so the page on PyPI matches the
repository, since that is where most people meet the project first.
-
Says which MCP clients this works with, and which it does not. "MCP
server" reads to a lot of people as "works with ChatGPT". It does not:
claude.ai in the browser and ChatGPT both accept only a remote server at a
public HTTPS URL, and DocSift speaks stdio. There is no configuration that
changes that, so the README now says so before anyone spends 1.6 GB finding
out. Reaching those clients would mean running DocSift on a server and
uploading documents to it, which is the opposite of what it is for. -
Adds the VS Code and Codex CLI configurations, which were missing —
including the startup timeout Codex needs. Its default allows ten seconds, and
DocSift loads PyTorch on the way up, so the default reports a server that
failed to start when it was only still starting.
v0.5.4
-
Fixed: two more lines of engine output reached the terminal on Linux.
0.5.3 silenced the Python logging, which is all that macOS produces. A first
run on Linux still showed onnxruntime announcing its PCI bus scan, and — while
the model cache was still cold — a warning from the HuggingFace Hub about
unauthenticated requests.The Hub one is a logger and simply joins the list. onnxruntime's could not be
fixed that way: it is C++ writing straight to file descriptor 2, and the
message is emitted while the module is importing, which defeats importing it
early to lower its log severity. That import now runs with the file descriptor
detached — only the import, so everything the conversion does afterwards keeps
a live stderr and real failures still surface. -
Internal:
tests/integrationnow runs in CI. It never had, because the normal
test job installs only themarkitdownextra — which is how 0.5.2 shipped a
progress indicator buried under 107 lines of engine logging with every test
passing. There is also a newscanned.pdffixture of rasterised text, so
docling's OCR path is exercised; every previous PDF fixture was born-digital
and skipped it entirely.
v0.5.3
-
Fixed: the conversion engines' own logging buried everything else.
Converting a 34-page PDF wrote 113 lines to stderr, 107 of them from docling
and its model stack — torch dynamo graph-break notices, and one warning per
page reporting that OCR found no text, which is the normal case for a
born-digital PDF rather than a problem.It made the progress output added in 0.5.2 effectively invisible, and the wall
of yellow WARNING lines read like a failure on a conversion that had actually
succeeded. Same document now writes 7 lines.Anything at ERROR or above still comes through.
docsift convert --verbose,
orDOCSIFT_VERBOSE=1for the MCP server and HTTP API, restores every line —
which is what a bug report needs.
v0.5.2
-
docsift convertnow shows progress instead of going silent. A cold PDF
conversion printed nothing for two and a half minutes — on a clean Linux
machine a three-page, 1.8 KB fixture took 186 seconds, 154 of them with no
output at all. There was no way to tell a running conversion from a hung one,
and the reasonable response to that is Ctrl-C.The CLI now shows a spinner with the current phase and elapsed time: loading
the engine, the one-time model download, converting, chunking, writing. It
writes to stderr, so piping stdout is unaffected; when stderr is not a
terminal it degrades to one plain line per phase, and--quietturns it off.The wait itself is unchanged, and it is worth being clear about where it goes:
almost all of it is Docling fetching its layout and table models the first
time it ever runs, plus loading PyTorch. It is startup cost, not page count —
a three-page file costs about the same as a thirty-page one, and only the
first conversion pays it. -
DocSift now says when you are carrying an unused CUDA build of PyTorch.
On Linux the default install resolves to the CUDA build: 5.3 GB on disk,
roughly 2 GB of itnvidia-*wheels that a machine without an NVIDIA GPU
never loads. No published wheel can prevent this — the CPU builds live on a
separate package index, and package metadata cannot redirect an installer —
so instead the first conversion on such a machine now reports it, with the
command that fixes it.For uv users,
uv tool install --torch-backend autoavoids the problem up
front and brings the install to 1.6 GB, while leaving CUDA in place for people
who do have a GPU. The README documents it. -
richis now a direct dependency rather than one inherited fromtyper.
v0.5.1 — fix MCP search budget
- Fixed: the MCP server's default token budget was a third of the CLI's, so
search_documentreturned a single chunk per call. Chunks routinely reach
~1,000 tokens and the budget was 2,000, which left no room for a second one.
In practice one question against a 34-page report cost six tool calls, each
handing back one fragment — spending more tokens in total than the wider
budget would have. The default is now 5,000, matchingdocsift search, and a
test holds the two surfaces together.
v0.5.0 — local MCP server
-
New
docsift mcpruns DocSift as a local MCP
server over stdio, for Claude Desktop, Claude Code, Codex, Cursor and other MCP
clients. Install it with the newmcpextra.Two tools:
search_documenttakes a file path and a question and returns only
the matching passages with page and section metadata, converting the file the
first time it is seen;convert_documentconverts and indexes a file and
returns a summary rather than its text. Neither returns a whole document —
that would put it in the model's context and undo the point.The server runs in your own process. Nothing listens on a port and no document
content crosses the network.
v0.4.0
Makes DocSift usable from Copilot Studio, Power Automate and n8n.
docsift openapi --format swagger2 -o docsift-connector.json
Swagger 2.0 export. Power Platform custom connectors do not accept the OpenAPI 3.1 document the service serves at /openapi.json. This command emits the Swagger 2.0 they need, with host and scheme taken from DOCSIFT_PUBLIC_URL.
Optional API key. Set DOCSIFT_API_KEY and every route except /health, /version, /openapi.json, /docs and /redoc requires an X-API-Key header. Off by default — a deployment that sets nothing behaves as before. This is one shared secret, not per-user identity.
Agent-oriented API descriptions. Every operation now carries a summary and a description written for tool selection, including when to prefer search over retrieving a whole document.
Examples. examples/n8n/ has an importable workflow (upload → poll → search) with failure and timeout branches; examples/copilot-studio/ covers connector setup; examples/power-automate/ documents the Do until flow that waits for conversion.
A Copilot Studio action cannot poll. Search works as a direct connector call. Uploading needs a Power Automate flow, because an action calls a connector once and conversion can take minutes. The guides explain the split rather than pretending one action can wait.
Upgrading from 0.3.0. No behaviour change unless you set the new variables. The OpenAPI document now advertises a servers entry only when DOCSIFT_PUBLIC_URL is set, so existing clients are unaffected.
Known limitations. No per-user identity, rate limiting or multi-tenancy. The bundled Dockerfile is still not build-tested. The connector file has been validated against the Swagger 2.0 schema but not yet imported into a live Power Platform tenant.
v0.3.0
Adds fully local keyword search over converted documents.
curl -sS --get --data-urlencode 'q=operational risk' \
http://127.0.0.1:8000/v1/documents/doc_xxxxxxxxxxxx/search
docsift search doc_xxxxxxxxxxxx "operational risk"
Successful API conversions index their chunks in SQLite FTS5. Search returns ranked chunks with section path, page numbers and token counts — never the document's complete Markdown. Quoted phrases are supported.
Controls. limit (default 5, max 20) sets direct matches; context (0–2) includes adjacent chunks, marked separately from direct matches; max_tokens (default 5000) caps the whole response. Queries are capped at 1024 characters and 64 terms.
Also in this release
- Invalid search syntax returns a stable 422 that never exposes SQLite details and never echoes the submitted query back.
- Genuine database faults surface as server errors rather than being reported to the caller as an invalid query.
- A document with no index rows returns
409telling you to re-upload it, instead of an empty result set indistinguishable from no matches. - FTS matching is scoped to the requested document, so search cost no longer grows with the rest of the corpus.
- The search endpoint uses the metadata row for its existence check rather than parsing the whole stored document — roughly half the previous latency.
- If a SQLite build lacks the FTS5 module, search returns
503and the rest of the service still runs. - Deletion removes a document's index rows together with its record; failed and cancelled jobs are never indexed.
Upgrading from 0.2.0. The index is created automatically — no migration step. But documents converted before this release are not in it and will return 409; re-upload them to index them. The version bump also invalidates 0.2.x cache entries, so the first conversion of each document after upgrading reconverts.
Read Known limitations before deploying. Search is lexical, not semantic — it does not understand synonyms. It is scoped to one document at a time. Page-number filtering is not supported. Scores order results within one response and are not comparable across requests. The API still has no authentication, rate limiting or multi-tenancy — run it behind your own access control. The bundled Dockerfile remains untested by a real build.