Skip to content

v1.2.0

Choose a tag to compare

@wzdavid wzdavid released this 15 Jul 08:15
· 18 commits to main since this release

Production hardening: keep Redis as a slim Celery queue (no parse bodies), isolate Redis disk from temp/output, shorten temp cleanup defaults, and document single-host multi-GPU plus multi-node scale-out.

Added

  • shared/task_result.py: slim Celery/Redis results; hydrate markdown / images[] / JSON from storage for the status API (response shape unchanged for ThinkDoc / ThinkExtract)
  • docker/docker-compose.multi-gpu.yml: one worker per GPU (mineru-gpu-0 … mineru-gpu-7)
  • docker/docker-compose.worker-only.yml: disable local API/cleanup on dedicated GPU hosts
  • Docs: multi-node production guide (docs/PRODUCTION_MULTI_NODE.md / .zh.md); Redis REDIS_DATA_PATH guidance in deployment docs
  • TEMP_MAX_AGE_HOURS (default 6) wired through cleanup scheduler / Compose

Changed

  • Celery task results no longer store full markdown, base64 images[], or inline content_list / middle_json in Redis (AOF/memory grow only with queue metadata)
  • Default cleanup interval CLEANUP_INTERVAL_HOURS: 24 -> 6; temp orphan age: 24 -> 6 (above TASK_TIME_LIMIT=2h)
  • Docker Redis volume can bind via REDIS_DATA_PATH (keep off the temp/output disk)

Fixed

  • S3 download_to_local: preserve remote file suffix so get_file_type() still selects MinerU for PDFs/images

Upgrade notes

  • Rebuild/restart API + Worker + cleanup images/containers so slim-result + hydrate code is live
  • Production: set REDIS_DATA_PATH on a disk separate from mineru_temp / mineru_output
  • Multi-GPU single host: COMPOSE_FILE=docker-compose.yml:docker-compose.multi-gpu.yml and profiles mineru-gpu-0,... (do not also enable mineru-gpu)
  • Multi-node GPU hosts: also load docker-compose.worker-only.yml; point REDIS_URL / S3 at shared services
  • Configure S3 temp bucket lifecycle when using MINERU_STORAGE_TYPE=s3
  • Public HTTP API fields are unchanged; clients need no code change