Proposal to keep semantics (especially tables, lists and similar)

[BogdanCerovac]

First of all - thanks for this, tested it on a couple of large documents and it was really fast.

I suggest to keep the semantics provided by the PDF - as per now we don't keep the relationships of tables and lists (as example).
Keeping relationships (th vs. td as a minimum) would help both people and LLMs in my opinion. Parsing tables now, even when PDF is made correctly (semantically, tagged) produces linear text that may be misunderstood and the important content hierarchy is absent).

I understand that raw text is not an option here, but we could perhaps do it in the JSON or perhaps introduce markdown (simplified)?
I am aware of the complexity, especially when we get data from OCR, but for the explicitly tagged PDF semantics parsing should support it by default, to ensure better understanding for the benefits of people and AI.

I don't know if this is perhaps already in the pipeline or planned or already discussed somewhere.
Was not able to find by searching the repo or docs, but these are kind partially similar:

• [Feature] Extract table of contents (outline) from PDF documents #248 (getting table of contents)
• [Feature] Support extracting URLs from links #152 (gettings urls from links)