IndexError: list index out of range

[OliverOffing]

I'm trying to add 250 documents but I'm hitting this error. It seems that this error shows up when trying to add >100 documents. Is that a hard limit I'm hitting or could there be something else, like perhaps one of the files if faulty?

Traceback (most recent call last):
  File "/app/main.py", line 24, in <module>
    docs.add(d)
  File "/usr/local/lib/python3.10/site-packages/paperqa/docs.py", line 111, in add
    citation = self.cite_chain.run(texts[0])
IndexError: list index out of range

[OliverOffing]

Actually, the problem was due to a document being too small.

Contents of the file that caused the problem:

# FAQs

Source:

paper-qa/paperqa/docs.py

Line 109 in b94a06e

texts, _ = read_doc(path, "", "", chunk_chars=chunk_chars)

We should do either one of these:

• Improve the error message so that when we can't process a file, the file name specific is presented to the user, or
• outright ignore the file and issue a warning message telling that that file is being ignored

[amittos]

Actually, I have the same issue only my document is definitely not too small. I believe that it's too large.

Is there an intuitive way to handle this error?

[OliverOffing]

I'm doing this until we find a fix:

    for d in my_docs:
        try:
            docs.add(d)
        except Exception as e:
            print('Error adding %s: %s' % (d, e))

[whitead]

@OliverOffing's solution is the preferred solution. You add them, get an exception if it fails, and you decide how to deal with the exception. You can skip and continue or try to figure out what is wrong yourself. The best we have is checks to see if it looks like a document.

The specific error above, about the index error for very short documents, has been fixed