long id_list not allowed by the API

[lab156]

when making a query with the length of id_list of 642 article names I get:

File "arxiv.py/arxiv/arxiv.py", line 34, in query
raise Exception("HTTP Error " + str(results.get('status', 'no status')) + " in query")
Exception: HTTP Error 414 in query

This is probably due to some limit in the API. Does anybody know more about this?
Should the library deal with this issue or is it more appropriate to leave it to the user?

[lukasschwab]

Here are docs on the 414 HTTP response status.

From the arXiv API documentation:

As mentioned above, the API can be called with an HTTP request of type GET or POST. For our purposes, the main difference is that the parameters are included in the url for a GET request, but not for the POST request. Thus if the parameters list is unusually long, a POST request might be preferred.

That's potentially a work-around here. Seems like an improvement that could be within the scope of this project, though it'd require a complication of the feedparser logic.

I don't think such a switch is imminently likely; I'd recommend breaking this up into a number of requests for fewer IDs if your application allows the additional network time. Alternatively, you could make the POST request directly.

Hope this is helpful––I'll leave the issue open as a future enhancement.

[lukasschwab]

I did some drafting on the use-post-requests branch last year––the actual code changes required for POST requests should be pretty minimal––but couldn't get it to work.

I suspect there's a mismatch between the API documentation and the API's actual behavior; I emailed the arXiv API mailing list reporting the difference: https://groups.google.com/g/arxiv-api/c/EkD5aC9LUIk

I expect this won't be resolved before the release of the JSON API.