feat: archive cloud MinerU raw ZIP to S3 - #238
Open
gdccyuen wants to merge 5 commits into
Open
Conversation
Add MINERU_LOCAL_MODE and supporting config (lang_list, backend,
timeout) to dispatch PDF parsing to a local MinerU instance's
synchronous /file_parse endpoint, avoiding the cloud-only batch APIs
that return 404 against local deployments.
parse_via_full now checks settings.MINERU_LOCAL_MODE first and routes
to a new parse_via_local that POSTs the PDF as multipart form data,
gets a ZIP back, extracts it, then flattens the nested {stem}/auto/
layout into the flat {output_dir}/full.md + images/* shape downstream
code expects.
Local MinerU is single-concurrency by default, so a separate requests
session with Retry(read=0) prevents ReadTimeout-driven urllib3 retries
from pushing requests to the back of the same queue. 429s surface as
UnavailableException without going through the cloud quota manager
(local mode has no API key).
Excerpted from ADR 0001 in knowhere-self-hosted; retires the
forked-image stopgap there once a knowhere release ships this natively.
Local MinerU's /file_parse endpoint expects the PDF under the 'files'
multipart field, not 'file'. The 422 error confirmed:
{'type':'missing','loc':['body','files'],'msg':'Field required'}
MinerU 3.4.0's /file_parse returns JSON with md_content and images (base64-encoded) inline, not a ZIP. Write md_content to full.md and decode images to images/ directory. No ZIP extraction or flattening needed for this API version.
- Request response_format_zip=true + return_original_file=true from
local MinerU /file_parse so the response body is a raw ZIP
- Upload the raw ZIP to results/{job_id}/mineru_raw{suffix}.zip before
extracting, writing the S3 key to _mineru_raw_s3_key.txt (sidecar)
- Keep inline JSON handling as fallback for MinerU builds that ignore
response_format_zip (no archival in that case)
- Thread job_id through parse_via_full; shard parses get suffixed keys
and their sidecars are merged into the main output dir
- Add optional on_zip_downloaded callback to download_and_extract_zip,
fired after download completes but before extract+delete
- Thread callback through poll_mineru_task
- parse_via_full (cloud mode) archives the raw ZIP to
results/{job_id}/mineru_raw{suffix}.zip and writes the S3 key sidecar
after polling completes (sidecar would be pruned by extraction cleanup)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Archives the raw MinerU ZIP in cloud mode (ADR 0002, step 4/6), so the complete raw MinerU output — including the original PDF with
return_original_file— is permanently retained in S3 for audit and re-processing.Changes
shared/utils/zip_download.pydownload_and_extract_zipgains an optionalon_zip_downloaded: Callable[[Path], None] | Noneparameter, invoked with the downloaded ZIP's path after the download completes but before extraction and temp-file deletion. The callback can therefore archive the raw artifact that the function would otherwise throw away.apps/worker/.../mineru/task_polling.pypoll_mineru_taskaccepts the same optionalon_zip_downloadedcallback and forwards it todownload_and_extract_zipwhen the task reachesdone.apps/worker/.../mineru/pdf_service.pyIn cloud mode,
parse_via_fullpasses a callback that uploads the downloaded ZIP to S3 atresults/{job_id}/mineru_raw{suffix}.zip(same helper + sidecar convention as local mode, step 3/6). The S3 key is written to the_mineru_raw_s3_key.txtsidecar after polling completes, because the extraction cleanup indownload_and_extract_zipwould prune a sidecar written inside the callback.Tests
download_and_extract_zipinvokes the callback before extraction and deletes the temp ZIP afterwards_archive_mineru_raw_zipend-to-end (upload + sidecar key capture)Notes
job_results.mineru_raw_s3_key) and the download API land in a follow-up PR (ADR steps 5/6 + 6/6).on_zip_downloadedare unaffected.