text mine via Crossref's TDM
Ruby
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
bin
extra
lib
test
.gitignore
.travis.yml
CHANGELOG.md
Gemfile
Gemfile.lock
README.md
Rakefile
textminer.gemspec

README.md

textminer

gem version Build Status codecov.io

textminer helps you text mine through Crossref's TDM (Text & Data Mining) services:

Changes

For changes see the CHANGELOG

gem API

  • Textiner.search - search by DOI, query string, filters, etc. to get Crossref metadata, which you can use downstream to get full text links. This method essentially wraps Serrano.works(), but only a subset of params - this interface may change depending on feedback.
  • Textiner.fetch - Fetch full text given a url, supports Crossref's Text and Data Mining service
  • Textiner.extract - Extract text from a pdf

Install

Release version

gem install textminer

Development version

git clone git@github.com:sckott/textminer.git
cd textminer
rake install

Examples

Within Ruby

Search

Search by DOI

require 'textminer'
# link to full text available
Textminer.search(doi: '10.7554/elife.06430')
# no link to full text available
Textminer.search(doi: "10.1371/journal.pone.0000308")

Many DOIs at once

require 'serrano'
dois = Serrano.random_dois(sample: 6)
Textminer.search(doi: dois)

Search with filters

Textminer.search(filter: {has_full_text: true})

Get full text links

The object returned form Textminer.search is a class, which has methods for pulling out all links, xml only, pdf only, or plain text only

x = Textminer.search(filter: {has_full_text: true})
x.links_xml
x.links_pdf
x.links_plain

Fetch full text

Textminer.fetch() gets full text based on URL input. We determine how to pull down and parse the content based on content type.

# get some metadata
res = Textminer.search(member: 2258, filter: {has_full_text: true});
# get links
links = res.links_xml(true);
# Get full text for an article
res = Textminer.fetch(url: links[0]);
# url
res.url
# file path
res.path
# content type
res.type
# parse content
res.parse

Extract text from PDF

Textminer.extract() extracts text from a pdf, given a path for a pdf

res = Textminer.search(member: 2258, filter: {has_full_text: true});
links = res.links_pdf(true);
res = Textminer.fetch(url: links[0]);
Textminer.extract(res.path)

On the CLI

Coming soon...

To do

  • CLI executable
  • better test suite
  • better documentation