KeyError: 'style' #72

RaphSte · 2024-06-14T11:57:59Z

When trying to run a pdf file through it I get the KeyError: 'style', with the following stacktrace:

error uploading file, stacktrace: Traceback (most recent call last):
  File "/app/nlm_ingestor/ingestion_daemon/__main__.py", line 48, in parse_document
    return_dict, _ = ingestor_api.ingest_document(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/nlm_ingestor/ingestor/ingestor_api.py", line 37, in ingest_document
    ingestor = pdf_ingestor.PDFIngestor(doc_location, parse_options)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/nlm_ingestor/ingestor/pdf_ingestor.py", line 35, in __init__
    blocks, _block_texts, _sents, _file_data, result, page_dim, num_pages = parse_blocks(
                                                                            ^^^^^^^^^^^^^
  File "/app/nlm_ingestor/ingestor/pdf_ingestor.py", line 176, in parse_blocks
    parsed_doc = visual_ingestor.Doc(pages, ignore_blocks, render_format)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/nlm_ingestor/ingestor/visual_ingestor/visual_ingestor.py", line 117, in __init__
    self.parse(pages)
  File "/app/nlm_ingestor/ingestor/visual_ingestor/visual_ingestor.py", line 198, in parse
    p["style"], p.text, page_width
    ~^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/bs4/element.py", line 1573, in __getitem__
    return self.attrs[key]
           ~~~~~~~~~~^^^^^
KeyError: 'style'
Traceback (most recent call last):
  File "/app/nlm_ingestor/ingestion_daemon/__main__.py", line 48, in parse_document
    return_dict, _ = ingestor_api.ingest_document(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/nlm_ingestor/ingestor/ingestor_api.py", line 37, in ingest_document
    ingestor = pdf_ingestor.PDFIngestor(doc_location, parse_options)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/nlm_ingestor/ingestor/pdf_ingestor.py", line 35, in __init__
    blocks, _block_texts, _sents, _file_data, result, page_dim, num_pages = parse_blocks(
                                                                            ^^^^^^^^^^^^^
  File "/app/nlm_ingestor/ingestor/pdf_ingestor.py", line 176, in parse_blocks
    parsed_doc = visual_ingestor.Doc(pages, ignore_blocks, render_format)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/nlm_ingestor/ingestor/visual_ingestor/visual_ingestor.py", line 117, in __init__
    self.parse(pages)
  File "/app/nlm_ingestor/ingestor/visual_ingestor/visual_ingestor.py", line 198, in parse
    p["style"], p.text, page_width
    ~^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/bs4/element.py", line 1573, in __getitem__
    return self.attrs[key]
           ~~~~~~~~~~^^^^^
KeyError: 'style'

Steps to reproduce:

(tested on linux server)

docker pull ghcr.io/nlmatics/nlm-ingestor:latest
docker run -p 5010:5001 ghcr.io/nlmatics/nlm-ingestor
After that, from a client:

from llmsherpa.readers import LayoutPDFReader
llmsherpa_api_url = "https://my-url/api/parseDocument?renderFormat=all"

#both mehtods, local and online will produce the same error
pdf_url = "https://arxiv.org/pdf/1910.13461.pdf"
pdf_url = "./arxiv.org/pdf/1910.13461.pdf"


pdf_reader = LayoutPDFReader(llmsherpa_api_url)
doc = pdf_reader.read_pdf(pdf_url)

The text was updated successfully, but these errors were encountered:

RaphSte · 2024-06-14T12:10:57Z

this seems to be the same error as in #24
One comment suggests, that the tika server is not running. How can I verify that?

eburnette · 2024-06-16T19:38:37Z

v0.1.8 and v0.1.7 have this problem for me. v0.1.6 works fine.

shshnk158 · 2024-06-17T07:27:37Z

Hey @RaphSte did the latest version work for you. If so, can you update me how?

aleksvercau · 2024-06-17T07:51:38Z

Same issue here, latest version has an issue.

RaphSte · 2024-06-17T07:59:09Z

Hey @RaphSte did the latest version work for you. If so, can you update me how?

hey, @shshnk158, no, it diddn't work for me. I'll use v0.1.6 for now.

shshnk158 · 2024-06-17T11:34:08Z

Yes v0.1.6 is working fine, but it comes with tika-server-standard-nlm-modified-2.4.1_v6.jar, I wanted to try out with the latest jar file [tika-server-standard-nlm-modified-2.9.2_v1.jar](https://github.com/nlmatics/nlm-ingestor/blob/main/jars/tika-server-standard-nlm-modified-2.9.2_v1.jar) any suggestions @ansukla

thenicekat · 2024-06-20T06:50:00Z

yes, facing this issue on v0.1.7 and v0.1.8

vitorhirota · 2024-06-27T13:13:41Z

The issue is because paragraphs are missing metadata. PR #70 solves this issue.

While not merged, you can use it locally with git fetch origin pull/70/head:PR70 and git switch PR70

rednag · 2024-06-27T14:09:51Z

I'm facing the same issue, trying now to build the container with PR70.

rednag · 2024-06-27T14:23:39Z

Container build fails with "Failed to build pandas"

vitorhirota · 2024-06-28T14:28:19Z

@rednag PR #73 is related, but in my case I just updated requirements.txt to have pandas >= 1.24

nlm-ingestor/requirements.txt

Line 13 in 465e6a1

pandas==1.2.4

heyalistair · 2024-07-03T09:23:14Z

@vitorhirota Changing that to >= fixes the pandas error.

Now I'm hitting a problem with python -m nltk.downloader punkt.

➜  nlm-ingestor git:(PR70) ✗ docker build --platform=linux/x86_64 -t ohalo-nlm-ingestor .
[+] Building 2.6s (22/24)                                                                                                                                           docker:desktop-linux
 => [internal] load .dockerignore                                                                                                                                                   0.0s
 => => transferring context: 2B                                                                                                                                                     0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                0.0s
 => => transferring dockerfile: 1.54kB                                                                                                                                              0.0s
 => resolve image config for docker.io/docker/dockerfile:experimental                                                                                                               0.4s
 => CACHED docker-image://docker.io/docker/dockerfile:experimental@sha256:600e5c62eedff338b3f7a0850beb7c05866e0ef27b2d2e8c02aa468e78496ff5                                          0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                0.0s
 => [internal] load .dockerignore                                                                                                                                                   0.0s
 => [internal] load metadata for docker.io/library/python:3.11-bookworm                                                                                                             0.4s
 => [ 1/16] FROM docker.io/library/python:3.11-bookworm@sha256:4eee56938c2f48480acb90db616162cfa361f5987dd43e1371e5288ed3e5e95e                                                     0.0s
 => => resolve docker.io/library/python:3.11-bookworm@sha256:4eee56938c2f48480acb90db616162cfa361f5987dd43e1371e5288ed3e5e95e                                                       0.0s
 => [internal] load build context                                                                                                                                                   0.1s
 => => transferring context: 186.95kB                                                                                                                                               0.1s
 => CACHED [ 2/16] RUN apt-get update && apt-get -y --no-install-recommends install libgomp1                                                                                        0.0s
 => CACHED [ 3/16] RUN mkdir -p /usr/share/man/man1 &&   apt-get update -y &&   apt-get install -y openjdk-17-jre-headless                                                          0.0s
 => CACHED [ 4/16] RUN apt-get install -y   libxml2-dev libxslt-dev   build-essential libmagic-dev                                                                                  0.0s
 => CACHED [ 5/16] RUN apt-get install -y   tesseract-ocr   lsb-release   && echo "deb https://notesalexp.org/tesseract-ocr5/$(lsb_release -cs)/ $(lsb_release -cs) main" | tee /e  0.0s
 => CACHED [ 6/16] RUN apt-get install unzip -y &&   apt-get install git -y &&   apt-get autoremove -y                                                                              0.0s
 => CACHED [ 7/16] WORKDIR /app                                                                                                                                                     0.0s
 => CACHED [ 8/16] COPY . ./                                                                                                                                                        0.0s
 => CACHED [ 9/16] RUN pip install --upgrade pip setuptools                                                                                                                         0.0s
 => CACHED [10/16] RUN apt-get install -y libmagic1                                                                                                                                 0.0s
 => CACHED [11/16] RUN mkdir -p -m 0600 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts                                                                                      0.0s
 => CACHED [12/16] RUN pip install -r requirements.txt                                                                                                                              0.0s
 => CACHED [13/16] RUN python -m nltk.downloader stopwords                                                                                                                          0.0s
 => ERROR [14/16] RUN python -m nltk.downloader punkt                                                                                                                               1.6s
------                                                                                                                                                                                   
 > [14/16] RUN python -m nltk.downloader punkt:                                                                                                                                          
0.505 <frozen runpy>:128: RuntimeWarning: 'nltk.downloader' found in sys.modules after import of package 'nltk', but prior to execution of 'nltk.downloader'; this may result in unpredictable behaviour                                                                                                                                                                          
0.874 [nltk_data] Downloading package punkt to /root/nltk_data...
1.525 [nltk_data]   Unzipping tokenizers/punkt.zip.
1.526 [nltk_data] Error with downloaded zip file
1.526 Error installing package. Retry? [n/y/e]
1.529 Traceback (most recent call last):
1.529   File "<frozen runpy>", line 198, in _run_module_as_main
1.530   File "<frozen runpy>", line 88, in _run_code
1.530   File "/usr/local/lib/python3.11/site-packages/nltk/downloader.py", line 2537, in <module>
1.532     rv = downloader.download(
1.532          ^^^^^^^^^^^^^^^^^^^^
1.532   File "/usr/local/lib/python3.11/site-packages/nltk/downloader.py", line 790, in download
1.533     choice = input().strip()
1.533              ^^^^^^^
1.534 EOFError: EOF when reading a line
------
Dockerfile:34
--------------------
  32 |     RUN pip install -r requirements.txt
  33 |     RUN python -m nltk.downloader stopwords
  34 | >>> RUN python -m nltk.downloader punkt
  35 |     RUN python -c "import tiktoken; tiktoken.get_encoding(\"cl100k_base\")"
  36 |     RUN chmod +x run.sh
--------------------
ERROR: failed to solve: process "/bin/sh -c python -m nltk.downloader punkt" did not complete successfully: exit code: 1

EDIT: Nevermind, turning on my VPN resolved this issue. I really need to switch ISPs... :)

heyalistair · 2024-07-03T13:27:15Z

I can confirm I can building the docker image PR #70, with pandas>=1.2.4 works and the container does not show the KeyError. Thanks!

jamesvillarrubia · 2024-07-09T17:01:36Z

Sorry, this one's on me. First PR was massive code refresh on top of the latest Tika and I missed some key elements that my tests didn't cover. Second PR with jar v2 should resolve it, but waiting on @ansukla or someone to merge here.

Here's the docker image I'm using with everything baked in:
jamesmtc/nlm-ingestor

ddose-inferyx · 2024-07-17T09:10:39Z

v0.1.8 and v0.1.7

are you talking about nlm-ingestor version. I cant see v0.1.6 there

RaphSte · 2024-07-17T11:32:01Z

v0.1.8 and v0.1.7

are you talking about nlm-ingestor version. I cant see v0.1.6 there

@ddose-inferyx yes, this is about the nlm ingestor version. You can either pull the image directly (see ) or build it yourself selecting the tag
I diddn't try building it myself though. I just pulled the image directly and it worked for me.

ddose-inferyx · 2024-07-17T15:06:12Z

v0.1.8 and v0.1.7 have this problem for me. v0.1.6 works fine.

Thanks! @RaphSte , It started working for me as I used -http://localhost:5010/api/parseDocument?renderFormat=all&applyOcr=yes&useNewIndentParser=yes. Using "NewIndentParser=yes." will also work with the latest.

irash03 · 2024-07-24T09:14:25Z

Here's the docker image I'm using with everything baked in: jamesmtc/nlm-ingestor

docker pull ghcr.io/jamesmtc/nlm-ingestor

Error response from daemon: Head "https://ghcr.io/v2/jamesmtc/nlm-ingestor/manifests/latest": denied

ddose-inferyx · 2024-07-25T09:26:56Z

Thank you so much. Everything is working fine with this.

…

-Dipak

On Wed, 24 Jul 2024 at 14:44, irash03 ***@***.***> wrote: Here's the docker image I'm using with everything baked in: jamesmtc/nlm-ingestor docker pull ghcr.io/jamesmtc/nlm-ingestor Error response from daemon: Head " https://ghcr.io/v2/jamesmtc/nlm-ingestor/manifests/latest": denied — Reply to this email directly, view it on GitHub <#72 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BIY7W33X2P7H6U32BKRKKGLZN5WATAVCNFSM6AAAAABJKGY5SKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBXGMZDOMBVGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

AeRabelais · 2024-07-25T11:37:17Z

Here's the docker image I'm using with everything baked in: jamesmtc/nlm-ingestor

docker pull ghcr.io/jamesmtc/nlm-ingestor

Error response from daemon: Head "https://ghcr.io/v2/jamesmtc/nlm-ingestor/manifests/latest": denied

I was getting the same error, and I just needed to reset my authentication info for ghcr. I removed any preset ghcr configs, then followed the setup instructions here. After that, running docker pull jamesmtc/nlm-ingestor:latest worked fine

ansukla · 2024-07-26T14:18:11Z

Merging changes from @jamesvillarrubia. Apologies for the delay. Thanks James for putting together the fix. Feel free to send me a note on LinkedIn if something needs attention.

Saii626 mentioned this issue Jul 14, 2024

KeyError: 'style' while running on local using docker image nlmatics/llmsherpa#93

Open

This was referenced Jul 16, 2024

JSONDecodeError nlmatics/llmsherpa#64

Open

read_pdf fails on specific pdf locally, not through hosted api nlmatics/llmsherpa#63

Open

JSv4 mentioned this issue Jul 19, 2024

[BUG] - NLM Ingestor Error - height is missing JSv4/OpenContracts#148

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KeyError: 'style' #72

KeyError: 'style' #72

RaphSte commented Jun 14, 2024 •

edited

Loading

RaphSte commented Jun 14, 2024

eburnette commented Jun 16, 2024

shshnk158 commented Jun 17, 2024 •

edited

Loading

aleksvercau commented Jun 17, 2024

RaphSte commented Jun 17, 2024

shshnk158 commented Jun 17, 2024

thenicekat commented Jun 20, 2024

vitorhirota commented Jun 27, 2024

rednag commented Jun 27, 2024

rednag commented Jun 27, 2024

vitorhirota commented Jun 28, 2024

heyalistair commented Jul 3, 2024 •

edited

Loading

heyalistair commented Jul 3, 2024

jamesvillarrubia commented Jul 9, 2024 •

edited

Loading

ddose-inferyx commented Jul 17, 2024

RaphSte commented Jul 17, 2024

ddose-inferyx commented Jul 17, 2024

irash03 commented Jul 24, 2024

ddose-inferyx commented Jul 25, 2024 via email

AeRabelais commented Jul 25, 2024

ansukla commented Jul 26, 2024

KeyError: 'style' #72

KeyError: 'style' #72

Comments

RaphSte commented Jun 14, 2024 • edited Loading

Steps to reproduce:

RaphSte commented Jun 14, 2024

eburnette commented Jun 16, 2024

shshnk158 commented Jun 17, 2024 • edited Loading

aleksvercau commented Jun 17, 2024

RaphSte commented Jun 17, 2024

shshnk158 commented Jun 17, 2024

thenicekat commented Jun 20, 2024

vitorhirota commented Jun 27, 2024

rednag commented Jun 27, 2024

rednag commented Jun 27, 2024

vitorhirota commented Jun 28, 2024

heyalistair commented Jul 3, 2024 • edited Loading

heyalistair commented Jul 3, 2024

jamesvillarrubia commented Jul 9, 2024 • edited Loading

ddose-inferyx commented Jul 17, 2024

RaphSte commented Jul 17, 2024

ddose-inferyx commented Jul 17, 2024

irash03 commented Jul 24, 2024

ddose-inferyx commented Jul 25, 2024 via email

AeRabelais commented Jul 25, 2024

ansukla commented Jul 26, 2024

RaphSte commented Jun 14, 2024 •

edited

Loading

shshnk158 commented Jun 17, 2024 •

edited

Loading

heyalistair commented Jul 3, 2024 •

edited

Loading

jamesvillarrubia commented Jul 9, 2024 •

edited

Loading