docs_parser

docs_parser is a FastAPI-based document ingestion and search service. It creates indexing tasks for documents stored in Google Drive, parses content from PDF/DOC/DOCX files, stores page-level text in Typesense, and provides a search API over indexed pages.

A browser-based operator UI is available separately: docs-parser-dashboard (search and task submission against this API; point its VITE_API_URL at this service and reuse API_TOKEN).

Project Stack

• Python 3.13
• FastAPI + Starlette (HTTP API)
• Pydantic / pydantic-settings (config and models)
• Typesense (document storage and full-text search)
• Google Drive API (file/folder ingestion source)
• PDF and OCR:
  • pypdf
  • pdf2image
  • pytesseract
• Word parsing: python-docx
• Docker + Docker Compose (local orchestration)

Functionality

• Auth middleware checks bearer token from Authorization header.
• POST /v1/tasks creates an investigate task for a Google Drive folder/file.
• Worker investigate scans a Google Drive folder and creates upload tasks per file.
• Worker upload downloads each file, selects parser by extension, extracts text by pages/chunks, and saves documents to Typesense collection for the project.
• POST /v1/search performs full-text search in the selected project collection and returns matching pages with snippets/highlights.

Repository Structure

• main.py - FastAPI app initialization
• middlewares.py - token authentication middleware
• api/ - API routes, schemas, dependencies
• services/ - business logic (FileUploader, TextSearch)
• tasks/ - background task runners (investigate, upload)
• lib/converter/ - PDF/Word converters
• lib/typesense/client.py - async wrapper around Typesense Python client
• db/typesense/models.py - Typesense documents/task models
• run_task.py - starts worker loop
• create_task.py - quick script to enqueue investigate task

How It Works

1. Client creates investigate task (/v1/tasks).
2. investigate worker reads folder content from Google Drive and creates upload tasks in tasks collection.
3. upload worker processes waiting upload tasks:

• downloads file from Google Drive
• chooses parser by extension (pdf, doc, docx)
• detects PDF text-vs-image path (OCR if image-based)
• saves extracted pages into project collection in Typesense

4. Client searches indexed data via /v1/search.

Environment Variables

Copy env.example to .env and fill values:

TYPESENSE_API_KEY=your_typesense_key
GOOGLE_SERVICE_ACCOUNT_FILE=/absolute/path/to/service-account.json
API_TOKEN=your_api_bearer_token

Also used in Docker Compose:

• TYPESENSE_HOST (defaults to localhost, compose uses docs_parser_typesense)
• TYPESENSE_PORT (default 8108)
• TYPESENSE_PROTOCOL (default http)
• TYPESENSE_DATA (host directory for persistent Typesense data)

Local Run (without Docker)

1. Install dependencies:

pip install -r requirements.txt

1. Make sure these services/tools are available:

• running Typesense instance
• Tesseract OCR + Poppler binaries (required for image PDF OCR flow)

2. Export env vars or create .env.
3. Start API:

uvicorn main:app --host 0.0.0.0 --port 8000 --reload

1. Start workers in separate terminals:

python run_task.py --task=investigate
python run_task.py --task=upload

Docker Run

1. Set env vars (.env file recommended), including:

• TYPESENSE_API_KEY
• TYPESENSE_DATA
• GOOGLE_SERVICE_ACCOUNT_FILE
• API_TOKEN

2. Start all services:

docker compose up --build

This starts:

• docs_parser_typesense
• docs_parser_backend
• docs_parser_investigate_task
• docs_parser_upload_task

API will be exposed on http://localhost:8083.

API

Base prefix: /v1

Create investigate task

POST /v1/tasks

Request:

{
  "lang": "ukr",
  "file_path": "https://drive.google.com/drive/folders/<folder_id>",
  "project_name": "kgb_project",
  "provider": "google"
}

Response:

{}

Search pages

POST /v1/search

Request:

{
  "query": "архів",
  "project_name": "kgb_project"
}

Response (example):

[
  {
    "file_path": "...",
    "book_name": "...",
    "page_number": 12,
    "page_content": "...",
    "snippets": ["..."]
  }
]

Authentication

Send bearer token header for protected routes:

Authorization: Bearer <API_TOKEN>

If token is missing/invalid, API returns 401 Unauthorized.

Useful Scripts

• python run_task.py --task=investigate - run investigate worker
• python run_task.py --task=upload - run upload worker
• python create_task.py <google_file_or_folder_id_or_url> <lang> - enqueue quick investigate task (uses hardcoded project name kgb_project)

Notes

• Collections in Typesense are created dynamically (tasks plus per-project collections).
• Current provider implementation is Google Drive only.
• Supported file formats by uploader mapping: pdf, doc, docx.