-
Notifications
You must be signed in to change notification settings - Fork 29
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
UnexpectedEmptyPageError and associated errorscre #31
Comments
Hi @robisen1, The root of this problem lies in the arxiv API which is used from the |
Closing this here, but please comment below if you expect anything to happen from our side. |
Sorry I am on a plane. I will try to comment on the next two days. Thank you
…On Mon, Jul 10, 2023, 3:34 PM Jannis Born ***@***.***> wrote:
Closing this here, but please comment below if you expect anything to
happen from our side.
—
Reply to this email directly, view it on GitHub
<#31 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABSF2RKTXKY5RKTI76OKBTDXPRRNDANCNFSM6AAAAAAZYBBW7I>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Hi, I have the same issue:
Can the code have somewhere if this is returned it just skips and continues and goes to next fetching of data? |
Hi, Since this is an issue from the |
Please excuse me if I do this incorrectly. I a noob. I am using python 3.11 on Windows 11 and Ubuntu 22.04.2. on I have run into an error like this on arxiv as well as medarxiv:
arxiv.arxiv.UnexpectedEmptyPageError: Page of results was unexpectedly empty (http://export.arxiv.org/api/query?search_query=%28all%3Apyschological+flow+state%29&id_list=&sortBy=relevance&sortOrder=descending&start=29500&max_results=100)
this seems to be an issue in the original code and was patched here lukasschwab/arxiv.py#43
I did not see that and I took a similar path. My code can checks to see if a URL is malformed or is empty. It handles it and logs it. If it runs into a URL that is not responding or hangs it waits some user-defined amount of time and moves on. You can also make it create smaller jsonl for various reasons. I was also going to implement querying by date. Right now it's all hardcoded variables but I was thinking I should make it so that you can call the options from the command line or a config file. I am also thinking about multi-threaded and being able to throttle your calls to service and or a back-off algorithm. I don't know what I am supposed to do. Do I provide my fixes, if needed, and how or do I go to the arxiv team? I also think these issues lurk in other libraries but I have not made anything like extensive testing. Thank you I appreciate your time and paper scraper.
The text was updated successfully, but these errors were encountered: