Error when extracting images with PyMuPDFLoader and PyPDFLoader

[BennisonDevadoss]

TRANSFERED BY @mdrxy
DOCS NOTE: SEE ATTACHED CLOSED PR THAT SHOULD BE APPLIED HERE

Checked other resources

• I added a very descriptive title to this issue.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

1. Use the following code to load a PDF with image extraction enabled with PyMuPDFLoader:

########################################
# PyMuPDFLoader
########################################
from langchain_community.document_loaders import PyMuPDFLoader

loader = PyMuPDFLoader("google-2024-environmental-report.pdf", extract_images=True)
pages = loader.load()

for page in pages:
    print(page.page_content)

2. Download the PDF located at: Google 2024 Environmental Report.
3. Additionally, I also tried using PyPDFLoader with the same PDF, and I encountered the same issue.

Error Message and Stack Trace (if applicable)

Traceback (most recent call last):
  File "/mnt/c/Users/BennisonJ/Yavar/projects/zypher-2.0/backend/apps/rag/main.py", line 811, in store_doc
    data = loader.load()
           ^^^^^^^^^^^^^
  File "/home/bennison/miniconda3/envs/open-webui/lib/python3.11/site-packages/langchain_community/document_loaders/pdf.py", line 387, in load
    return list(self._lazy_load(**kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bennison/miniconda3/envs/open-webui/lib/python3.11/site-packages/langchain_community/document_loaders/pdf.py", line 384, in _lazy_load
    yield from parser.lazy_parse(blob)
  File "/home/bennison/miniconda3/envs/open-webui/lib/python3.11/site-packages/langchain_community/document_loaders/parsers/pdf.py", line 244, in lazy_parse
    yield from [
               ^
  File "/home/bennison/miniconda3/envs/open-webui/lib/python3.11/site-packages/langchain_community/document_loaders/parsers/pdf.py", line 247, in <listcomp>
    + self._extract_images_from_page(doc, page),
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bennison/miniconda3/envs/open-webui/lib/python3.11/site-packages/langchain_community/document_loaders/parsers/pdf.py", line 283, in _extract_images_from_page
    return extract_from_images_with_rapidocr(imgs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bennison/miniconda3/envs/open-webui/lib/python3.11/site-packages/langchain_community/document_loaders/parsers/pdf.py", line 74, in extract_from_images_with_rapidocr
    result, _ = ocr(img)
                ^^^^^^^^
  File "/home/bennison/miniconda3/envs/open-webui/lib/python3.11/site-packages/rapidocr_onnxruntime/rapid_ocr_api.py", line 80, in __call__
    dt_boxes, det_elapse = self.text_detector(img)
                           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bennison/miniconda3/envs/open-webui/lib/python3.11/site-packages/rapidocr_onnxruntime/ch_ppocr_v3_det/text_detect.py", line 66, in __call__
    data = transform(data, self.preprocess_op)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bennison/miniconda3/envs/open-webui/lib/python3.11/site-packages/rapidocr_onnxruntime/ch_ppocr_v3_det/utils.py", line 220, in transform
    data = op(data)
           ^^^^^^^^
  File "/home/bennison/miniconda3/envs/open-webui/lib/python3.11/site-packages/rapidocr_onnxruntime/ch_ppocr_v3_det/utils.py", line 75, in __call__
    data['image'] = (img * self.scale - self.mean) / self.std
                     ~~~~~~~~~~~~~~~~~^~~~~~~~~~~
ValueError: operands could not be broadcast together with shapes (896,800) (1,1,3)

Description

I am encountering a ValueError when using both PyMuPDFLoader and PyPDFLoader to extract images from certain PDFs. The error message indicates that operands could not be broadcast together with shapes (896,800) (1,1,3). This occurs specifically when the extract_images parameter is set to True.

Expected Behavior
The code should successfully extract text and images from the PDF without errors.

Additional Information
This issue seems to occur with specific PDFs that may have unique formatting or image properties. I would appreciate any guidance on how to resolve this issue or if there are any workarounds available.

System Info

langchain==0.1.16
langchain-chroma==0.1.0
langchain-community==0.0.34
langchain-core==0.1.52
langchain-text-splitters==0.0.2

PyMuPDF Version: 1.24.10
PyPDF Version: 4.2.0
Operating System: Ubuntu 22 LTS

[venkatram-dev]

Looks like you might be using a old version of rapidocr_onnxruntime.
Please upgrade to latest or 1.3.24.

I did not get the error using this version.

pip show rapidocr_onnxruntime
Name: rapidocr-onnxruntime
Version: 1.3.24

https://github.com/RapidAI/RapidOCR/blob/50ce26671ab4a78955f8bc773348f82ad9fe0024/python/rapidocr_onnxruntime/ch_ppocr_v3_det/utils.py#L44](https://github.com/RapidAI/RapidOCR/commit/1de4db02497f832536ed1578e9bce5195850db8f)

[kartheekyakkala]

I think rapidocr-onnxruntime is a required package for extracting information from images. Can you please provide the versions of the rapidocr-onnxruntime?.

I have used all the latest versions of the packages. It worked fine.

[BennisonDevadoss]

@kartheekyakkala, Thank you for addressing the issue!

After upgrading to the latest versions of the packages, the solution now works correctly for most scanned documents, which is a significant improvement.

However, I encountered a new issue when testing with the following PDF:

Scan Sample PDF

Unfortunately, the updated solution does not work for this particular PDF. The problem persists in a similar manner, where the image extraction and processing fail.

Details

1. RapidOCR ONNXRuntime Version: 1.3.24
2. Python Version: 3.11

Here I have attached the screenshot of the error

[venkatram-dev]

@BennisonDevadoss ,
This new error is coming from langchain-community package.
Looks like you are using a older version of it.

Please upgrade to latest version of langchain-community. Using the latest version worked for above file without any errors.

You might also want to update all the other langchain packages also to latest.

[BennisonDevadoss]

@venkatram-dev, I updated the plugin versions to the latest, but I'm still encountering the same issue.

Below are my updated package versions:

langchain-community: 0.2.16
langchain-core:  0.2.38
langchain :0.2.16

[venkatram-dev]

I am not able to reproduce the error. If possible, please share a google colab.

[BennisonDevadoss]

@venkatram-dev , as I mentioned earlier, I'm still encountering the issue we discussed previously, even with the latest version of the LangChain modules. I've shared the reproducible code on Google Colab, which you can access via the link below: Colab Link

Additionally, here's the PDF link I used to read the text from: PDF link

Please let me know if you find any breakthroughs!

[jamesrooti]

Is there progress on this? Experiencing the same issue. Thanks

[BennisonDevadoss]

@jamesrooti, Yes I am having same issue.

[dosubot]

Hi, @BennisonDevadoss. I'm Dosu, and I'm helping the LangChain team manage their backlog. I'm marking this issue as stale.

Issue Summary:

• You reported difficulties extracting images from PDFs using PyMuPDFLoader and PyPDFLoader.
• The issue persists with specific PDFs, even after updating related packages.
• Venkatram-dev suggested updating to the latest versions, but the problem remains.
• You provided a Google Colab link for reproducibility.
• Jamesrooti also reported experiencing the same issue, indicating a broader problem.

Next Steps:

• Please confirm if this issue is still relevant with the latest version of LangChain.
• If the issue is still present, you can keep the discussion open by commenting here; otherwise, it will be automatically closed in 7 days.

Thank you for your understanding and contribution!

[BennisonDevadoss]

Yes, this issue still exists.

[dosubot]

@eyurtsev, the user @BennisonDevadoss has confirmed that the issue with extracting images from PDFs using PyMuPDFLoader and PyPDFLoader is still relevant. Could you please assist them with this problem?

[skyking363]

Me too