pdf citation

[Surajlambor]

I love using this library for PDF purposes. Now, I'm building a PDF summarization tool. For citations, we need the PDF name and the page number of the retrieved answer. Do you have any ideas on how I can accomplish this? Please help with this

[JorjMcKie]

These data can be achieved like so:

• PDF name = os.path.abspath(doc.name), full path name of the PDF file.
• There may be other useful info in the metadata, doc.metadata, e.g. title, creator, producer, creation date, change date.
• The page number simply is page.number (0-based)

[Surajlambor]

I've managed to extract text, file names, and page numbers. however, when I chunk the text, I lose the metadata associated with each chunk. so, I want each chunk to retain its corresponding file name and page number. then, I'll embed and store this information in the VectorDB. afterward, users can input queries, and my retrieval system will retrieve similar chunks along with their metadata

[Surajlambor]

ALL_TEXTS = []

def extract_text_from_pdf(pdf_file):
try:
text = ''
with fitz.open(pdf_file) as doc:
total_pages = doc.page_count
with concurrent.futures.ThreadPoolExecutor() as executor:
futures = [executor.submit(process_page, doc, page_num) for page_num in range(total_pages)]
for future in concurrent.futures.as_completed(futures):
page_num, page_text = future.result()
text += page_text
ALL_TEXTS.append({'text': page_text, 'file': pdf_file, 'page': page_num + 1})
return text
except Exception as e:
print(f"Error extracting text from {pdf_file}: {e}")
return None

[alonet23]

These data can be achieved like so:

• PDF name = os.path.abspath(doc.name), full path name of the PDF file.
• There may be other useful info in the metadata, doc.metadata, e.g. title, creator, producer, creation date, change date.
• The page number simply is page.number (0-based)

Can you assist to know if it is feasible to add page number in markdown that can be used for citation by LLM. if not how we can use page.number effectively without using page_chuck=true. thanks!

[JorjMcKie]

How about using package pymupdf4llm instead? It produces markdown text, can read multi-column pages and tables.
It also ha a built-in chunking method using page chunks. The page chunks contain the page's text plus a bunch of metadata:

import pymupdf4llm
data = pymupdf4llm.to_markdown("input.pdf", page_chunks=True)

Here, data is a list of dictionaries, one per page. for details please see the homepage.

[Surajlambor]

{'page_num': 1, 'text': "Challenges and consideration \n• Technical Complexity: Combining different modalities requires complex algorithms \nthat can process and analyze data from different sources. \n• User Interface Design: The interface should seamlessly handle multimodal inputs and \npresent information clearly through text, visuals, or audio depending on the user's \npreference. \n• Real-time Processing Power: Handling large volumes of real-time data requires robust \ninfrastructure. \n \n", 'pdf_path': 'E:\poc2\media\pibot\resources\Multimodal Conversational AI Chatbot.pdf'}.I achieve this using :ALL_TEXTS = []

def extract_text_from_pdf(pdf_file):
try:
text = ''
with fitz.open(pdf_file) as doc:
total_pages = doc.page_count
with concurrent.futures.ThreadPoolExecutor() as executor:
futures = [executor.submit(process_page, doc, page_num) for page_num in range(total_pages)]
for future in concurrent.futures.as_completed(futures):
page_num, page_text = future.result()
text += page_text
ALL_TEXTS.append({'text': page_text, 'file': pdf_file, 'page': page_num + 1})
return text
except Exception as e:
print(f"Error extracting text from {pdf_file}: {e}")
return None .here is the problem: We get metadata for a page's text, but we later chunk this text. how will metadata be associated with the chunks?

[Surajlambor]

for each chunk we want page number and pdf name

[JorjMcKie]

You don't seem to use the latest version of pymupdf4llm. 0.0.5. Example:

import pymupdf4llm
data = pymupdf4llm.to_markdown("test.pdf", page_chunks=True)
from pprint import pprint
pprint(data[0])
{'graphics': [],
 'images': [],
 'metadata': {'author': '',
              'creationDate': '',
              'creator': '',
              'encryption': None,
              'file_path': 'test.pdf',   <===
              'format': 'PDF 1.7',
              'keywords': '',
              'modDate': '',
              'page': 1,    <===
              'page_count': 1,
              'producer': '',
              'subject': '',
              'title': '',
              'trapped': ''},
 'tables': [],
 'text': 'Lorem ipsum dolor sit amet, consectetur\n'
         'adipisici elit, sed eiusmod tempor incidunt ut\n'
         'labore et dolore magna aliqua. Ut enim ad\n'
         'minim veniam, quis nostrud exercitation\n'
         '**ullamco** **_laboris_** nisi ut aliquid ex ea commodi\n'
         'consequat. Quis aute iure\n'
         'reprehenderit\n'
         'in **voluptate** velit esse cillum dolore eu fugiat\n'
         'nulla pariatur. Excepteur sint obcaecat cupiditat\n'
         'non proident, sunt in culpa qui [officia](https://www.artifex.com) '
         'deserunt\n'
         'mollit anim id est laborum.\n'
         '\n'
         '\n'
         '-----\n'
         '\n',
 'toc_items': []}

What is missing?