what's the normal time for downloading a paper?

[shizhediao]

Hi,
Thanks for your great work!
I was wondering what's the normal time for downloading a paper?
I would like to download as much as possible papers to do some research. Maybe the size is 10 K ~ 100 K.
But for now, it costs me 10 seconds for each paper downloading, so is it possible to speed up?
Thanks very much!

[lukasschwab]

Hmm, this depends on how you're finding the papers to download.

• Does the 10-second operation include a query, or is it just the call to arxiv.download? It may be possible to improve the query performance.
• arxiv.download uses urlretrieve; I don't know if this is the quickest solution for downloads in bulk. You might be interested in building your own bulk-download function.

Probably the most useful: if you just want as many papers as possible, arXiv offers bulk access to tarfiles of PDFs and source files via S3: https://arxiv.org/help/bulk_data_s3

I'll close this issue for the time being; feel free to reopen it if this doesn't answer your question!

[shizhediao]

Thanks for your reply!
In my experiment, it costs 10-second only for the call to arxiv.download.
Thanks for pointing out the bulk access, I'll take a look.
Thanks very much!