Skip to content

Releases: anishmoncivarghese/docsift

v0.5.7

Choose a tag to compare

@anishmoncivarghese anishmoncivarghese released this 12 Aug 20:23
  • PowerPoint answers can now be cited by slide. A deck's chunks came back
    with no attribution at all — no slide number, no title, and MarkItDown's raw
    slide markers sitting in the chunk text — because the pipeline recognises
    <!-- page: N --> and MarkItDown writes <!-- Slide number: N -->, so the
    marker was never seen as one and fell through as ordinary paragraph text.

    Translating it at the engine boundary fixed the attribution but not the
    citation: slide content is sparse, so the token-budgeted chunker packed 55 of
    a 60-slide deck into one chunk. Accurate, and no use to anyone. A deck is now
    chunked one slide at a time, the same way Docling supplies its own chunks for
    PDFs, so 60 slides yield 60 chunks each naming its slide and title. An
    oversized slide still splits normally.

    Speaker notes were already extracted and are searchable. Diagrams are not: a
    slide with three boxes and two arrows gives three labels, and the arrows are
    lost. Images inside slides are not read at all. Both are documented.

v0.5.6

Choose a tag to compare

@anishmoncivarghese anishmoncivarghese released this 11 Aug 03:27
  • Listed in the official MCP Registry. Adds server.json (validated against
    the published 2025-12-11 schema) and the mcp-name: ownership marker the
    registry checks against the README published on PyPI — which is why this
    needed a release rather than just a commit.

    The listing installs with uvx --from "docsift[mcp,docling,markitdown]" docsift mcp, so the extras come along; a bare uvx docsift would install the
    CLI without an MCP SDK.

v0.5.5

Choose a tag to compare

@anishmoncivarghese anishmoncivarghese released this 10 Aug 09:03

Documentation only; no code changes. Released so the page on PyPI matches the
repository, since that is where most people meet the project first.

  • Says which MCP clients this works with, and which it does not. "MCP
    server" reads to a lot of people as "works with ChatGPT". It does not:
    claude.ai in the browser and ChatGPT both accept only a remote server at a
    public HTTPS URL, and DocSift speaks stdio. There is no configuration that
    changes that, so the README now says so before anyone spends 1.6 GB finding
    out. Reaching those clients would mean running DocSift on a server and
    uploading documents to it, which is the opposite of what it is for.

  • Adds the VS Code and Codex CLI configurations, which were missing —
    including the startup timeout Codex needs. Its default allows ten seconds, and
    DocSift loads PyTorch on the way up, so the default reports a server that
    failed to start when it was only still starting.

v0.5.4

Choose a tag to compare

@anishmoncivarghese anishmoncivarghese released this 10 Aug 08:48
  • Fixed: two more lines of engine output reached the terminal on Linux.
    0.5.3 silenced the Python logging, which is all that macOS produces. A first
    run on Linux still showed onnxruntime announcing its PCI bus scan, and — while
    the model cache was still cold — a warning from the HuggingFace Hub about
    unauthenticated requests.

    The Hub one is a logger and simply joins the list. onnxruntime's could not be
    fixed that way: it is C++ writing straight to file descriptor 2, and the
    message is emitted while the module is importing, which defeats importing it
    early to lower its log severity. That import now runs with the file descriptor
    detached — only the import, so everything the conversion does afterwards keeps
    a live stderr and real failures still surface.

  • Internal: tests/integration now runs in CI. It never had, because the normal
    test job installs only the markitdown extra — which is how 0.5.2 shipped a
    progress indicator buried under 107 lines of engine logging with every test
    passing. There is also a new scanned.pdf fixture of rasterised text, so
    docling's OCR path is exercised; every previous PDF fixture was born-digital
    and skipped it entirely.

v0.5.3

Choose a tag to compare

@anishmoncivarghese anishmoncivarghese released this 10 Aug 05:28
  • Fixed: the conversion engines' own logging buried everything else.
    Converting a 34-page PDF wrote 113 lines to stderr, 107 of them from docling
    and its model stack — torch dynamo graph-break notices, and one warning per
    page reporting that OCR found no text, which is the normal case for a
    born-digital PDF rather than a problem.

    It made the progress output added in 0.5.2 effectively invisible, and the wall
    of yellow WARNING lines read like a failure on a conversion that had actually
    succeeded. Same document now writes 7 lines.

    Anything at ERROR or above still comes through. docsift convert --verbose,
    or DOCSIFT_VERBOSE=1 for the MCP server and HTTP API, restores every line —
    which is what a bug report needs.

v0.5.2

Choose a tag to compare

@anishmoncivarghese anishmoncivarghese released this 10 Aug 04:22
  • docsift convert now shows progress instead of going silent. A cold PDF
    conversion printed nothing for two and a half minutes — on a clean Linux
    machine a three-page, 1.8 KB fixture took 186 seconds, 154 of them with no
    output at all. There was no way to tell a running conversion from a hung one,
    and the reasonable response to that is Ctrl-C.

    The CLI now shows a spinner with the current phase and elapsed time: loading
    the engine, the one-time model download, converting, chunking, writing. It
    writes to stderr, so piping stdout is unaffected; when stderr is not a
    terminal it degrades to one plain line per phase, and --quiet turns it off.

    The wait itself is unchanged, and it is worth being clear about where it goes:
    almost all of it is Docling fetching its layout and table models the first
    time it ever runs, plus loading PyTorch. It is startup cost, not page count —
    a three-page file costs about the same as a thirty-page one, and only the
    first conversion pays it.

  • DocSift now says when you are carrying an unused CUDA build of PyTorch.
    On Linux the default install resolves to the CUDA build: 5.3 GB on disk,
    roughly 2 GB of it nvidia-* wheels that a machine without an NVIDIA GPU
    never loads. No published wheel can prevent this — the CPU builds live on a
    separate package index, and package metadata cannot redirect an installer —
    so instead the first conversion on such a machine now reports it, with the
    command that fixes it.

    For uv users, uv tool install --torch-backend auto avoids the problem up
    front and brings the install to 1.6 GB, while leaving CUDA in place for people
    who do have a GPU. The README documents it.

  • rich is now a direct dependency rather than one inherited from typer.

v0.5.1 — fix MCP search budget

Choose a tag to compare

@anishmoncivarghese anishmoncivarghese released this 08 Aug 19:58
  • Fixed: the MCP server's default token budget was a third of the CLI's, so
    search_document returned a single chunk per call. Chunks routinely reach
    ~1,000 tokens and the budget was 2,000, which left no room for a second one.
    In practice one question against a 34-page report cost six tool calls, each
    handing back one fragment — spending more tokens in total than the wider
    budget would have. The default is now 5,000, matching docsift search, and a
    test holds the two surfaces together.

v0.5.0 — local MCP server

Choose a tag to compare

@anishmoncivarghese anishmoncivarghese released this 08 Aug 19:07
  • New docsift mcp runs DocSift as a local MCP
    server over stdio, for Claude Desktop, Claude Code, Codex, Cursor and other MCP
    clients. Install it with the new mcp extra.

    Two tools: search_document takes a file path and a question and returns only
    the matching passages with page and section metadata, converting the file the
    first time it is seen; convert_document converts and indexes a file and
    returns a summary rather than its text. Neither returns a whole document —
    that would put it in the model's context and undo the point.

    The server runs in your own process. Nothing listens on a port and no document
    content crosses the network.

v0.4.0

Choose a tag to compare

@anishmoncivarghese anishmoncivarghese released this 06 Aug 10:11

Makes DocSift usable from Copilot Studio, Power Automate and n8n.

docsift openapi --format swagger2 -o docsift-connector.json

Swagger 2.0 export. Power Platform custom connectors do not accept the OpenAPI 3.1 document the service serves at /openapi.json. This command emits the Swagger 2.0 they need, with host and scheme taken from DOCSIFT_PUBLIC_URL.

Optional API key. Set DOCSIFT_API_KEY and every route except /health, /version, /openapi.json, /docs and /redoc requires an X-API-Key header. Off by default — a deployment that sets nothing behaves as before. This is one shared secret, not per-user identity.

Agent-oriented API descriptions. Every operation now carries a summary and a description written for tool selection, including when to prefer search over retrieving a whole document.

Examples. examples/n8n/ has an importable workflow (upload → poll → search) with failure and timeout branches; examples/copilot-studio/ covers connector setup; examples/power-automate/ documents the Do until flow that waits for conversion.

A Copilot Studio action cannot poll. Search works as a direct connector call. Uploading needs a Power Automate flow, because an action calls a connector once and conversion can take minutes. The guides explain the split rather than pretending one action can wait.

Upgrading from 0.3.0. No behaviour change unless you set the new variables. The OpenAPI document now advertises a servers entry only when DOCSIFT_PUBLIC_URL is set, so existing clients are unaffected.

Known limitations. No per-user identity, rate limiting or multi-tenancy. The bundled Dockerfile is still not build-tested. The connector file has been validated against the Swagger 2.0 schema but not yet imported into a live Power Platform tenant.

v0.3.0

Choose a tag to compare

@anishmoncivarghese anishmoncivarghese released this 06 Aug 04:14

Adds fully local keyword search over converted documents.

curl -sS --get --data-urlencode 'q=operational risk' \
  http://127.0.0.1:8000/v1/documents/doc_xxxxxxxxxxxx/search

docsift search doc_xxxxxxxxxxxx "operational risk"

Successful API conversions index their chunks in SQLite FTS5. Search returns ranked chunks with section path, page numbers and token counts — never the document's complete Markdown. Quoted phrases are supported.

Controls. limit (default 5, max 20) sets direct matches; context (0–2) includes adjacent chunks, marked separately from direct matches; max_tokens (default 5000) caps the whole response. Queries are capped at 1024 characters and 64 terms.

Also in this release

  • Invalid search syntax returns a stable 422 that never exposes SQLite details and never echoes the submitted query back.
  • Genuine database faults surface as server errors rather than being reported to the caller as an invalid query.
  • A document with no index rows returns 409 telling you to re-upload it, instead of an empty result set indistinguishable from no matches.
  • FTS matching is scoped to the requested document, so search cost no longer grows with the rest of the corpus.
  • The search endpoint uses the metadata row for its existence check rather than parsing the whole stored document — roughly half the previous latency.
  • If a SQLite build lacks the FTS5 module, search returns 503 and the rest of the service still runs.
  • Deletion removes a document's index rows together with its record; failed and cancelled jobs are never indexed.

Upgrading from 0.2.0. The index is created automatically — no migration step. But documents converted before this release are not in it and will return 409; re-upload them to index them. The version bump also invalidates 0.2.x cache entries, so the first conversion of each document after upgrading reconverts.

Read Known limitations before deploying. Search is lexical, not semantic — it does not understand synonyms. It is scoped to one document at a time. Page-number filtering is not supported. Scores order results within one response and are not comparable across requests. The API still has no authentication, rate limiting or multi-tenancy — run it behind your own access control. The bundled Dockerfile remains untested by a real build.