HTTP Error 403 and 418 on add_url in paper-qa

[janki-vaghasiya]

When I try to add url in Doc, it gives me 403 and 418 for some URLs even though they are publicly available.

My code

docs = Docs(llm=llm_client)
for url in urls:
    try:
        docs.add_url(url)
    except Exception as e:
        print(f'Exception in adding URL {url}: {e}')

Examples for failed cases:
403 Forbidden Error:

• https://dl.acm.org/doi/pdf/10.1145/3581783.3612034
• https://bsj.uobaghdad.edu.iq/index.php/BSJ/article/download/7168/4295

418 Unknown Error:

• https://ieeexplore.ieee.org/ielx7/6287639/10005208/10024290.pdf
• https://ieeexplore.ieee.org/ielx7/6287639/9668973/09712284.pdf

Can someone kindly explain why is it happening for a few URLs but for others it works fine?

[whitead]

Many websites, especially publishers, will block requests that look like they arise from non-users (e.g., Python). You're seeing that here. You can try to research scraping to get the URLs in an automated way or you can just download them manually first and then use paths