v1.4.2

Security

Only real image files are sent to the vision model (#387)

In 1.4.1, a Markdown image reference to any existing local file became an image block, and its bytes were base64-encoded and sent to the vision-model provider. Ingesting an untrusted .md containing ![x](/etc/hosts) or a reference to a key file could therefore send that file to the model API.

• A Markdown reference now becomes an image block only if the file has an image extension and starts with a PNG, JPEG, GIF, BMP, TIFF/BigTIFF or WebP signature. Anything else stays literal text.
• Both image encoders also check the bytes they are about to send. This covers parse-cache entries written by 1.4.1, content lists, files swapped after parsing, and Image Path: lines in retrieved text during VLM-enhanced queries.
• Referenced images are still sent wherever they are on disk, so only ingest Markdown whose image references you trust.

Upgrade note: if you ingested untrusted Markdown with 1.4.1, check which files it referenced. Any non-image files among them were already sent to your vision-model provider.

What's new

Page numbers on text chunks (#361)

Text chunks now record their source pages, as multimodal chunks already did. Text blocks are still joined into one string before LightRAG splits them. Once LightRAG has processed the document, each of its stored text chunks (text_chunks) is annotated with the page_idx of the blocks it came from, plus page_idx_end when it spans pages. Closes #330. Originally contributed by @kokhlo.

• Positions are computed by replaying LightRAG's default chunker, never by searching for chunk text.
• A chunk is left without page_idx rather than guessed when its position can't be established. That happens for: a custom chunking_func; a chunk cut into token windows by a tokenizer that doesn't decode losslessly position by position (LightRAG's default tiktoken does; pieces split at split_by_character are placed with any tokenizer); a re-insert or failed retry of a doc_id LightRAG already knows; a queued document chunked by another insert's run with different split_by_character settings; a document still queued when inserted through process_document_complete_lightrag_api (which does not wait); or a chunk whose text recurs on other pages.
• The fields are extra keys on the text_chunks record. The default JSON KV storage and other schemaless backends keep them; fixed-schema backends such as PostgreSQL may drop them. LightRAG stores one record per distinct chunk text, so a chunk shared by several documents carries the pages of the document its full_doc_id names. On backends that merge updates, such as MongoDB, a chunk re-stored by a document that was not annotated can keep the previous document's pages.

Queued text is no longer lost; failed text inserts now raise (#386)

When LightRAG's pipeline was busy with another insert, ainsert only queued the document and returned. RAG-Anything then overwrote its status, so LightRAG never picked it up: the text was never indexed while doc_status said processed. This affected process_folder_complete() (files run concurrently) and any concurrent process_document_complete() / insert_content_list() callers. In the PR's reproduction, three of four staggered concurrent inserts lost all their text.

process_document_complete() and insert_content_list() now wait after the text insert until LightRAG has finished the document's text. If a pipeline run ended without reaching it, they run the pipeline themselves.

Behaviour changes:

• If LightRAG fails a document's text and no pipeline run is left to retry it, both methods now raise RuntimeError: LightRAG failed to process the text of <doc_id>: <error>. Previously the failure was overwritten and the document was reported processed. The document stays failed in doc_status, and LightRAG retries it on its next pipeline run (for example, when the file is processed again). process_folder_complete() logs such a file as failed and continues with the other files.
• If the user cancelled LightRAG's pipeline before the queued text was processed, both methods raise instead of restarting the pipeline.
• Both methods can take longer to return when queued behind another insert. A warning is logged every five minutes while they wait.
• Before each text insert, a doc_status record that RAG-Anything wrote without LightRAG having stored the text is removed (for example, failed after a parse error with an explicit doc_id). Such a record no longer causes LightRAG to drop later inserts of that doc_id as duplicates.
• process_document_complete_lightrag_api() keeps its asynchronous behaviour.

Fixes

Parsing & Markdown

• Markdown code fences now close only on a bare fence of the same marker that is at least as long as the opener (CommonMark). A shorter fence, or a fence line with an info string such as ```python, no longer ends a code block early (#356, @er-s-an).
• Percent-encoded local Markdown image paths are decoded before resolving, so figure%201.png finds figure 1.png. If the decoded name doesn't resolve, the path is tried as written (#359, @lllleolin-max).
• Markdown image paths containing parentheses now resolve, in both the <Screenshot (1).png> and image(1).png forms (one level of balanced parentheses). A bare target with an unbalanced (, such as ![a](image(1.png), is now literal text, as CommonMark specifies; 1.4.1 resolved it when such a file existed. For such names, use the angle-bracket form ![a](<image(1.png>) (#373, @huiq777).
• ![alt](path) inside an inline code span or after an escaping backslash stays literal text instead of becoming an image block (#379, @KennyMcSimpson).
• Docling: page_idx comes from the item's source page (prov[].page_no, converted to zero-based) instead of a running block count divided by ten. Items without provenance keep 0 (#363, @lllleolin-max).
• PaddleOCR 3.x: result metadata such as the input filename, min and general no longer leaks into recognized text, and blank pages yield no text (#385, @er-s-an).
• OMML equations: converted commands are separated from the letters that follow them, so a≤b becomes a\leq b instead of the undefined \leqb (#366, @lllleolin-max).

Multimodal processors

• The image, table, equation and generic processors now return three values on their error fallback, as they do on success. Previously an item that hit the fallback raised ValueError: not enough values to unpack in the caller, which hid the original error. The README modal-processor examples now unpack three values as well (#367, @lihongyuan99).
• String captions and footnotes (as Docling emits them) are no longer joined one character at a time (F, i, g, u, r, e) in image chunks or in image/table context extraction (#368, @lihongyuan99).
• Multimodal chunks keep their document's full_doc_id in chunks_vdb during entity extraction. Before this fix, the field was overwritten with the chunk id (#360, @lllleolin-max).

Query

• With a vision model configured, aquery() now honours only_need_context / only_need_prompt and returns LightRAG's context or prompt. Previously these flags raised a duplicate-keyword TypeError or triggered a vision call. Direct aquery_vlm_enhanced() calls are fixed too (#362, @lllleolin-max).
• aquery_with_multimodal(..., stream=True) with multimodal content now skips the multimodal result cache entirely: no cache key, read, write or flush. Previously a streaming call could get back an earlier cached text answer instead of a stream. The single-use token iterator was also written to the LLM response cache, so a repeated streaming query got an exhausted iterator, and the next flush could leave a truncated kv_store_llm_response_cache.json that failed to load on the next start. Only string answers are cached now (#376, @tuanzirwar).

Reliability

• Concurrent first calls to _ensure_lightrag_initialized() are serialised by an instance-level lock, so stores are not initialized twice and no caller returns while another is still initializing. A cache whose initialization fails or is cancelled is retried on the next call instead of being skipped for good (#364, @lllleolin-max).
• CircuitBreaker.async_call: cancelling a half-open probe, directly or through an asyncio.wait_for timeout, now releases its trial slot. Previously the breaker stayed half-open and rejected every later call (#377, @er-s-an).

Batch

• incremental=True now reparses a file when the parser, parse_method or a result-affecting parser option (for example lang or the page range) changes. Previously the unchanged file was skipped and its old results were kept. Passing a default explicitly or changing the parser name's case does not trigger a reparse, and options that can't be serialised to JSON disable reuse for that run. Manifests written by older versions reparse every file once, and a file whose reparse fails is removed from the manifest so the next run retries it (#365, @lllleolin-max).

Offline setup

• scripts/create_tiktoken_cache.py now caches o200k_base (the encoding of LightRAG 1.4.x's default gpt-4o-mini tokenizer) as well as cl100k_base, so the offline procedure in docs/offline_setup.md works. When TIKTOKEN_CACHE_DIR is already set, the script now creates and reports that directory; previously it created and printed ./tiktoken_cache even though tiktoken wrote to the variable's directory. env.example now documents WORKING_DIR (#384, @roozjalali, cherry-picked from #358).

Docs

• docs/batch_processing.md has a new section on multimodal ingestion concurrency. It explains how max_parallel_insert, BatchParser.max_workers and llm_model_max_async differ, and how per-call limits multiply across concurrent files and video scenes. docs/multimodal_rag_failure_modes.md links to it. Runtime behaviour is unchanged (#375, @er-s-an).
• docs/batch_processing.md now matches the shipped API: the process_batch example passes the required output_dir, the environment variable is OUTPUT_DIR (not PARSER_OUTPUT_DIR), and success_rate is shown as a property (#370, @lihongyuan99).
• docs/batch_processing.md also documents the incremental configuration digest (#365), and how text queued behind a busy LightRAG pipeline is indexed and how text failures are reported (#386). The README (EN/ZH) now explains the #386 behaviour change next to the end-to-end processing example (#387).

Build & tests

• requirements.txt now pins lightrag-hku<1.5, and setup.py now requires Python >=3.10, matching pyproject.toml and the published wheel. pip install -r requirements.txt previously resolved to LightRAG 1.5.x, which breaks multimodal extraction. The <1.5 pin is permanent because RAG-Anything was merged upstream into LightRAG 1.5+ (#381).
• .gitignore: test_* is now anchored to the repo root, so new tests/test_*.py files are no longer silently ignored. test_output/ from the example scripts is now ignored as well (#369, @lihongyuan99).
• Restored the regression suites described in #352 (Markdown image blocks) and #354 (media file routing), which the old .gitignore pattern had kept out of the repository (30 tests) (#380).
• Version set to 1.4.2 (#383).

Validation

Beyond the unit suites (684 tests; Python 3.10 with LightRAG 1.4.15 and Python 3.12 with LightRAG 1.4.16, both file orders, plus CI on Linux 3.10–3.12), this release was checked as follows.

Packaging

• twine check --strict passes for the wheel and the sdist.
• A clean git archive rebuild has identical file sets and metadata.
• Compared with 1.4.1, the wheel adds only parse_options.py and image_signature.py. Requires-Dist and extras are unchanged. Requires-Python is now >=3.10, which LightRAG 1.4.x already required.
• Fresh installs on Python 3.10 and 3.12 resolve lightrag-hku 1.4.16, pip check is clean, and the test suite passes against the installed wheel.

Clean Linux end-to-end (python:3.11-slim, wheel only; live gpt-4o-mini for LLM and VLM, text-embedding-3-small via OpenRouter):

• a 4-page content list with text, table, equation and image: every text chunk carries the correct page_idx/page_idx_end, and hybrid, mix and naive queries answer from all three modalities;
• 5 concurrent inserts and a 4-file process_folder_complete: no document lost, all processed;
• an invalid model raises LightRAG failed to process the text …, leaves the document failed, and the re-run recovers it;
• streaming multimodal queries are never cached and the cache file stays valid;
• only_need_context / only_need_prompt make no VLM call;
• the offline tiktoken cache (o200k_base) works with no network.

Real parsers on macOS:

• PaddleOCR 3.7.0: exactly the recognised lines, no metadata, blank pages empty.
• Docling 2.134: page_idx taken from source pages.
• Markdown: images with parentheses, spaces and percent-encoding become VLM-described blocks; fenced, inline-code, escaped, missing and remote references stay literal.
• Batch incremental=True: re-parses when a result-affecting option changes, skips otherwise.

Security (#387):

• An adversarial bypass hunt found nothing: symlinks to files and devices, FIFOs, hard links, case and encoding tricks, NUL bytes, reference-style and HTML images, other ingest paths.
• A 1.4.1 parse cache reused by 1.4.2 now sends 0 payloads, where 1.4.1 sent /etc/hosts.
• 63 real image files across all supported formats still pass.
• A live end-to-end run sends only the real images, never the fake one.

Full history: v1.4.1...v1.4.2