Repository navigation
Replies: 1 comment
|
Hi Dr @ceberam, Thank you very much for writing this up, and for the time you spent reading both of my PRs. You are right that #3978 and #4716 disagree on almost every row of your table, and I think that is a fair reason to pause both. I am happy to rework both of them to follow the contract we agree on here. My suggestion in short Backends only discover attachments, and the converter decides, converts and records them. For the limits I would start from #4716:
But I now think the conversion itself belongs in the converter, not in the backend. Each attachment would get its own record and its own A rough sketch (the names are only placeholders): # Backend side: discovery only, no parsing.
class AttachmentSource:
name: str # sanitized, never used as a path
container_path: str # e.g. word/embeddings/Microsoft_Excel_Worksheet.xlsx
declared_mimetype: str | None
size: int
relation: AttachmentRelation # attached | embedded | linked | associated
anchor: str | None # self_ref of the host item, e.g. the preview picture
def open(self) -> bytes: ...
# DeclarativeDocumentBackend.get_attachments() -> list[AttachmentSource] (default: [])
# Converter side: one switch and one budget for the whole tree.
class AttachmentOptions:
mode: Literal["off", "list", "convert"] = "off"
max_depth: int = 1
max_count: int = 32 # whole tree, not per level
max_item_bytes: int = ...
max_total_bytes: int = ...
strict: bool = False
# Result: one flat list for the whole tree, so the schema is not recursive.
class AttachmentRecord:
id: str
parent_id: str | None
depth: int
name: str
container_path: str
mimetype: str # detected from the content
size: int
sha256: str
relation: AttachmentRelation
anchor: str | None
status: Literal["converted", "listed", "skipped", "failed", "duplicate"]
reason: str | None
errors: list[ErrorItem]
document: DoclingDocument | None
# ConversionResult.attachments: list[AttachmentRecord]
About your open questions
What would happen to the two PRs #4716 would become a I may be wrong on any of these points, so please feel free to change them. Thank you again for your guidance! |
Uh oh!
There was an error while loading. Please reload this page.
1. Summary
mainthat touches real attachments is the email backend (names only, opt-in) and the calendar backend (names or inline-data markers only). The Office backends keep only the preview picture of an embedded object. The PDF and HTML backends have no code for embedded files.DoclingDocument. The main questions are what an attachment is in the data model, who decides to parse it, and which limits protect the user and the service.DoclingDocumenthas no concept of a child document. Merging an attachment into the body of its host weakens the boundary between the two documents, risks page number collisions, and has no place for per-document origin and status. Pull request feat(docx): convert embedded Word, Excel and PowerPoint objects #4716 reduces some of these effects with a named group and by removing page provenance, but at the cost of data (section 4.3). A first-class representation avoids these problems but needs a schema change indocling-coreand in the DocLang archive.2. Why guidelines are needed
2.1 The demand is real
Issue #2051 asked for OLE object extraction from Word files, and it described the loss of data in RAG pipelines. It was closed for lack of reply. Pull request #3978 has a comment from a user who says their team waits for recursive attachment parsing. Email, invoices with embedded XML, reports with embedded workbooks, and archives of mixed files are common in enterprise inputs.
2.2 Each backend decides alone
The two open pull requests are not a review target here. They are useful as evidence of what happens when the contract is missing:
process_attachmentsprocess_embedded_objectsmax_attachments, default 32max_embedded_objects, default 32max_embedding_depth, default 3 (first version: fixed 1).docx,.xlsx,.pptx)GroupLabel.SECTION, namedembedded: <part name>SUCCESSdocument_converter.pyEvery row is a decision that users will see as inconsistent behavior if more backends follow. The same contributor wrote both pull requests and asked maintainers for guidance on the contract in the #4716 discussion. Even with one author, the two designs differ, which shows the need for a published contract.
2.3 Security and cost
Attachments multiply the risk of every input:
max_total_bytes,max_file_bytes,max_member_count), and a nested archive must pass the same checks before any parser opens it.document_timeoutis per document, andDocumentLimitsdefaults (max_num_pages,max_file_size) aresys.maxsize, so the library trusts its input unless the caller sets limits.ATTACHURIs in calendars, and<iframe src>or<object data>in HTML point to resources outside the file. Fetching them is a server-side request forgery risk. The existing switchesenable_remote_fetchandenable_local_fetch(both defaultFalse) should govern them.vbaProject.bin) and scripts must never run. Parsing must stay passive. The contract should say that unsupported or active content is listed but not opened.2.4 Precedents in the project become hard to change
Option names, default values, and output shapes become part of the public API and of reference data in
tests/data. The first merged implementation sets the pattern for PDF, PPTX, XLSX, ODF, HTML, and others.3. Terminology
The word "attachment" is overloaded. In the Docling code base it already means different things. For example, the iWork backend uses "attachment" for inline drawables, and the calendar backend uses it for the
ATTACHproperty. This document uses these terms:Packageobjectword/embeddings/*.xlsx, ODF sub-object<iframe src>, linked OLE object, calendarATTACHURI, Markdown link to a filePictureItemObjectReplacementspreview imagesThis paper deals with the first three. Embedded resources and internal parts are out of scope. Composite inputs are the existing precedent (section 4.2).
4. State of Docling on
main4.1 Backend inventory
maindoesdocling_parse_backend.py,pypdfium2_backend.py,pdf_backend.py)msword_backend.py)packageandoleObjectrelationships underword/embeddings/, linked objects, other documents imported into the bodyimagedataare read. The preview picture of an object is kept. The embedded payload is not read. Pull request #4716 states the same.mspowerpoint_backend.py)msexcel_backend.py)opendocument_backend.py)draw:objectsub-documents andObjectReplacementspreviewsdraw:objectis opened only to read a chart title. Replacement previews are skipped when a chart is found..emland.msg(email_backend.py)message/rfc822list_attachments(defaultFalse) adds a list of names and content types. The content is never decoded. A.msgis projected to RFC 822 with empty payloads (email_message.add_attachment(b"", ...)).calendar/render.py)ATTACHwith a URI or inline binary datahtml_backend.py)<iframe>,<object>,<embed>, links to documents, MIME sub-partsscript,noscript,style, andtemplatetags are removed. MHTML sub-parts are collected as image resources only. No handling of frames or objects found.epub_backend.py)fetch_images). Not an attachment case.iwork/)Two points follow from the inventory:
4.2 Existing precedents that a contract should reuse
Folders and ZIP archives. A folder passed to the converter yields one result per file. A ZIP archive that is not itself a known Office format is expanded in
_DocumentConversionInput.docs(docling/datamodel/document.py). Each member becomes a separateInputDocumentnamed<archive>/<member>. Absolute paths,..segments, and macOS metadata members are skipped. Each member and the cumulative expanded size are checked againstlimits.max_file_size. Members are not expanded again: a nested archive has no known format and gets the dummy backend. This is the closest existing precedent: independent documents, linked only by name, with a size check.Archive limits. Several backends define
max_total_bytes,max_file_bytes, andmax_member_countindocling/datamodel/backend_options.py(EPUB, METS GBS, iWork, Excel, Word) and apply them throughdocling/backend/utils/archive_limits.py(ArchiveLimits,check_archive_limits). According to the discussion on #4716, the PowerPoint backend does not apply these limits yet. A contract for nested packages needs the same check for every package that is opened.Remote and local fetch.
BaseBackendOptionshasenable_remote_fetchandenable_local_fetch, bothFalseby default. This is the right place for the policy on linked resources.Global limits.
DocumentLimits(docling/datamodel/settings.py) hasmax_num_pages,max_file_size, andpage_range. They apply to one input and aresys.maxsizeby default. They do not know about attachments.Errors and status.
ConversionResulthasstatus,errors, andtimings. A backend that handles an attachment failure on its own cannot write toerrors. The author of #3978 notes this: a failed attachment leaves a note in the text while the status staysSUCCESS.Layering. Backends must not import
DocumentConverter. Pull request #3978 works around this with an injected factory. This is a sign that orchestration of nested conversions may belong in the converter and not in the backend.4.3 The data model has no place for a child document
DoclingDocumenthas these top-level fields:origin,body,furniture(deprecated),groups,texts,pictures,tables,key_value_items,form_items,field_regions,field_items, andpages.GroupLabelhas no attachment label.DoclingDocument.add_document(doc, parent)copies the body children of one document under a node of another.Consequences of merging a child document into the host with
add_document. The last column shows what pull request #4716 does about each one. It is a reading of the diff and its sample output, not a run of the code.originis single. The name, MIME type, andbinary_hashof the child have no place in the host.embedded: <name>). The MIME type, size, and hash are not kept. Its status is not kept.pagesis keyed by page number, and each item hasprovwith a page number. A paginated child (for example a PDF or a PPTX) would add pages to a host that has other page semantics.item.prov = []on every copied item. The child has no page or position information in the result. This is acceptable for a Word host and for the Excel and PowerPoint children in its tests, but it is a loss, and it would not work for a paginated host such as a PDF.add_documentdeep-copies the items and gives them newself_refvalues. It does not touchcaptions,footnotes,references,comments, orRichTableCell.ref, which still carry the old indexes._add_embedded_documentmaps old to new items and rewrites these references. This confirms that the merge needs repair code. Every backend that merges must repeat it, and a future new reference field in the model would silently break.DoclingDocument. The child content is placed under aGroupLabel.SECTIONgroup namedembedded: <file name>, and the group also holds the preview picture. A consumer can find the boundary by group name. The boundary is a naming convention. It has no dedicated label or field, so a real section of the host with that name looks the same. Markdown and text exports show no boundary: in the reference Markdown of the PR, the embedded workbook and presentation appear as plain tables and headings after a<!-- image -->placeholder. DocLang output is weaker still (section 4.4).Quarterly report). The title of the embedded presentation is exported as a top-level#heading in Markdown, next to the##heading of the host section. I did not check how this affects chunking.DoclingDocument.In short: #4716 keeps the boundary as a named group and avoids page problems by removing provenance. It does not lose the boundary completely in the
DoclingDocument. It still loses the origin details, the page information, and a machine-readable marker, and it needs backend code to repair references. The general statement "merging loses the boundary" is true for a plainadd_document, and only partly true for #4716.4.4 What DocLang and the
.dclxarchive allow todayThe specification has no attachment or child-document concept. The document model is one
<doclang>root per instance.4.5 PDF files can contain embedded files
The PDF specification (ISO 32000) defines several ways to carry other files inside a PDF.
EmbeddedFilesname tree. Viewers show them in an "Attachments" panel.AF)The same file can be a trap in two ways. The attachment may be a different document than the visible pages (the visible invoice says one amount and the embedded XML says another). And the attachment may be anything: another PDF, an Office file, or an executable.
Why this matters for the contract:
list_attachments.6. Design space
This section lists the decisions and the main options. It does not rank them, except where the trade-off is one-sided.
6.1 Which attachments
allowed_formatsof the converterPull request #3978 parses everything that Docling detects, and #4716 only parses OOXML. A contract should state a rule that applies to all formats. Option C looks natural, but the behavior for excluded types must be defined.
6.2 Default behavior
Opt-in (default off) keeps output stable for existing users and keeps the reference data unchanged. It is the pattern used by
render_chart_imagesandlist_attachments. Default-on is convenient but changes output, cost, and risk for all users, and it is hard to reverse.A related question is whether one switch should apply to all backends (for example a field on the converter or the pipeline options) or each backend keeps its own switch. A single switch gives a uniform contract and a single place for the service to enforce policy. Per-backend switches match the current pattern in
backend_options.pybut multiply names.6.3 Where orchestration lives
format_options,allowed_formats) apply naturally.The second option fits the layer rule better, and it is the only one where a global budget and honest status reporting are easy. It also means a backend can offer a method such as "list attachments" that returns names, MIME types, sizes, hashes, and an opener, without parsing anything.
6.4 Representation of the result
Option 1: merge into the host body. Both open pull requests do this. #4716 places the child in a named group (
embedded: <file name>) at the position of the object, so it is a "merge with a naming convention". #3978 places the content in a subsection of an "Attachments" heading.Option 2: separate documents, linked by metadata. The converter returns one result for the host and one for each attachment, as for a ZIP archive. Each result has its own
DoclingDocument, origin, and status. A parent link and a relationship type connect them.convert_all.ConversionResultor in the host as a text item). Todayconvert_allyields one result per input, and many callers rely on that; extra results can surprise them unless they are in a separate field. Serialization of a host with its children is not defined (DocLang file per document?).Option 3: first-class attachment elements in
DoclingDocument. The data model gets a node, for example an attachment item placed in the body at the anchor position. It holds name, MIME type, size, hash, relationship type (attached, embedded, linked), status (parsed, skipped, failed, not requested), and either a nestedDoclingDocumentor a reference to one stored beside it..dclxarchive can carry converted children as extra parts, so a single file holds the whole result.docling-coreand a new schema version. All serializers (Markdown, HTML, text, DocTags, DocLang), chunkers,iterate_items, and validators need rules for the new node. A nested document in a Pydantic model makes the JSON schema recursive. Default exports must decide whether to include children; if not, content seems lost again, and if yes, the boundary is lost again. The DocLang specification has no element for a child document today (section 4.4), so this option needs a specification change in the DocLang repository.Option 4: manifest in the document, converted children beside it. The host
DoclingDocumentcarries only light-weight references (name, MIME type, size, hash, relationship, anchor, status). The converted children travel in theConversionResultor in an archive, keyed by the reference id.docling-corechange and an archive layout decision.Option 5: extraction API only. Backends expose "list" and "open" for their attachments, and the user decides what to do. No parsing inside Docling.
Options 2, 4, and 5 can be combined into a staged path (see section 8). Option 3 is the most complete and also the largest commitment.
6.5 Depth, count, and size
A contract needs five numbers or rules, with clear meaning:
Related choices:
max_num_pagesanddocument_timeoutto the sum over all attachments or to each document. The sum is safer.6.6 Options for nested conversions
An attachment may use the converter's
format_optionsandallowed_formats, or default options.A third path: inherit the options and let the caller override per format for attachments.
6.7 Failure reporting
Choices are a note in the text, a warning in the log, an entry in
ConversionResult.errors, a new status such asPARTIAL_SUCCESS, or a status field on the attachment record. Notes in the text pollute RAG content. Log lines are invisible to API users. A structured record with a status and a reason is the only option that works for both humans and programs, and it can be mapped toErrorItemandFailureCategory.POLICYfor limits.PARTIAL_SUCCESSalready exists inConversionStatus.6.8 Identity and provenance
Each attachment should carry a name (sanitized, never used as a file path), a MIME type from content detection and not only from the declared type, size, and a content hash. The hash links duplicates and makes outputs reproducible. The path in the container (for example
word/embeddings/oleObject1.bin) helps debugging. The relationship type (attached, embedded, linked, associated file) helps consumers decide what to index. For an embedded document, the place in the reading order is also needed.6.9 Export and downstream use
Whatever the representation, the contract should say what each exporter does by default. For example, Markdown export could omit children by default and include them with a parameter that adds a heading and a clear boundary. Chunkers should be able to label chunks with the attachment they came from. The same question applies to
.dclx: whether converted children are stored as additional parts of the archive (see section 4.4 for what the archive format allows). The loader limits inload_from_doclang_archive(max_member_size512 MiB,max_total_size2 GiB) show that the format already treats archive size as a security topic.6.10 Test and reference data
Reference data is regenerated rarely and reviewed by two maintainers. Fixtures for nested packages must be produced by real applications, include bomb and fan-out cases, and avoid platform-dependent preview images (EMF conversion differs by platform, as noted in #4716).
7. Open questions for maintainers
docling-core, the DocLang specification, and.dclxevolve together, and what is the compatibility policy for the JSON schema version?PARTIAL_SUCCESS, structured records, or both?All reactions