New Extractor Idea: `scihub-dl` to auto-detect inline DOI numbers and download academic paper PDFs

[pirate]

New extractor idea: SCIHUB

e.g. take this academic paper for example: https://www.cell.com/current-biology/fulltext/S0960-9822(19)31469-1

If a full paper PDF is available on scihub, e.g.: https://sci-hub.se/https://www.cell.com/current-biology/fulltext/S0960-9822(19)31469-1 it could be downloaded to a ./archive/<timestmap>/scihub/ output folder.

# try downloading via verbatim URL first
$ scihub.py -d https://www.cell.com/current-biology/fulltext/S0960-9822(19)31469-1'
DEBUG:Sci-Hub:Successfully downloaded file with identifier https://www.cell.com/current-biology/fulltext/S0960-9822(19)31469-1

We could also look for a DOI number in the page URL or page html contents e.g.: 10.1016/j.cub.2019.11.030 using a regex and try downloading that.

# otherwise try downloading via any regex-extracted bare DOI numbers on the page or in the URL
$ scihub.py -d '10.1016/j.cub.2019.11.030'
DEBUG:Sci-Hub:Successfully downloaded file with identifier 10.1016/j.cub.2019.11.030

$ ls
c28dc1242df6f931c29b9cd445a55597-.cub.2019.11.030.pdf

New Dependencies:

• https://github.com/zaytoun/scihub.py

New Extractors:

• extractors/scihub.py

New Config Options:

• SAVE_SCIHUB=True

[danisztls]

Captchas would be a problem.

[pirate]

In my testing so far it's not a problem as long as you're not doing dozens of papers at a time.

[jezcope]

Two more avenues to explore:

1. Zotero has a big collection of translators to get full text from many different publishers and repositories https://github.com/zotero/translators
2. Unpaywall has a huge database of open access copies of articles https://unpaywall.org/

Would be great to capture full text for academic publications where its available.

[pirate]

Parsing DOIs out of PDF/HTML content:

• https://github.com/MicheleCotrufo/pdf2doi
• https://www.findingyourway.io/blog/2019/03/13/2019-03-13_extracting-doi-from-text/
• https://stackoverflow.com/questions/27910/finding-a-doi-in-a-document-or-page

Found good candidate extractor dependencies to try for the major free scientific paper DBs:

• https://github.com/Tishacy/SciDownl
• • https://pypi.org/project/arxiv-dl/
• https://github.com/viown/libgen-dl
• https://github.com/unpywall/unpywall
• https://openaccessbutton.org/?from=integration&id=
• https://github.com/bibcure/scihub2pdf
• https://github.com/dheison0/annas-archive-api

and more via...

• https://github.com/mrossinek/cobib ⭐️ (awesome bibliography manager that supports DOI search)
• https://gist.github.com/taskylizard/5ba73bf97dccf159316edcf4c6520856#-academic-papers
• https://github.com/cipher387/osint_stuff_tool_collection
• https://www.reddit.com/r/FREEMEDIAHECKYEAH/wiki/tools-index

something like this might be interesting for linking together citations too https://inciteful.xyz/

[benmuth]

I have a rough script for this working (just using scihub.py as a module and downloading a pdf with the url->doi fallback). I had to slightly modify scihub.py to get it to work, but I was thinking we can just add the modified version to the vendor directory or something. It looks like scihub.py isn't available on PyPI anyway.

Should I just wait for the plugin system? Or I can create a repo with the script and modified scihub.py so it can be looked at and improved in the meantime. Or I can try to minimally integrate it with ArchiveBox (just get it working as a oneshot?) and open a PR for feedback and testing, whatever makes sense.

[pirate]

If you can post it in a gist and share the link that would be helpful @benmuth!

[benmuth]

Here's what I have so far https://gist.github.com/benmuth/b2c12cbb40ca4d8183c6f17f819e2f2d @pirate

Usage:

python scihub-extractor.py -d <DOI|URL> -o <output directory>

or

python scihub-extractor.py -f <text file with newline separated DOIs|URLs> -o <output directory>

It should either

• download the paper directly identified by the DOI or URL you gave it
  or
• if you gave it a URL but it can't directly find a paper through it, then it will parse the page's HTML for DOIs and attempt to download all DOIs that are found.

In the second case, we can probably be smarter about only downloading the DOIs intended (based on title or something?), but it's pretty dumb right now.

[pirate]

@benmuth it might take me a month or more till I'm able to merge this, as I'm working on some paid ArchiveBox projects right now for a client that are taking up most of my time. Don't worry, I'm keen to merge it though, I've been wanting this feature personally for a long time.

[benmuth]

@benmuth it might take me a month or more till I'm able to merge this, as I'm working on some paid ArchiveBox projects right now for a client that are taking up most of my time. Don't worry, I'm keen to merge it though, I've been wanting this feature personally for a long time.

No worries! In the meantime maybe I can keep adding to it. I've kept it simple for demonstration purposes, but it seems straightforward to add some of the ideas you linked here, like the PDF parsing.