Skip to content

Manual

Jump to bottom Edit New page
louismullie edited this page Aug 4, 2012 · 35 revisions

##Configuration

All values shown on this page are the default values for the relevant configuration options.

###Encoding

Ruby 1.9 does not parse files with non-ASCII characters unless you specify the encoding. You can do so by adding a Ruby comment at the very top of the file, with the appropriate encoding, e.g.: # encoding: utf-8

###Verbosity

# A boolean value indicating whether to silence 
# the output of external libraries (e.g. Stanford
# tools, Enju, LDA) when they are used. 
Treat.core.verbosity.silence = true

# A boolean value indicating whether to output 
# information useful for debugging purposes.
Treat.core.verbosity.debug = false

###Languages

# A boolean value indicating whether Treat should 
# try to detect the language of newly input text.
Treat.core.language.detect = false

# The language to default to when detection is off.
Treat.core.language.default = :english

# A symbol representing the finest level at which 
# language detection should be performed if 
# language detection is turned on.
Treat.core.language.detect_at = :sentence

###Paths

# A directory in which to create temporary files.
Treat.core.paths.tmp = '/X/tmp/'

# A directory in which to store downloaded files.
Treat.core.paths.files = '/X/files/'

# The directory containing executables and JAR files.
Treat.core.paths.bin = '/X/bin/'

# The directory containing training models.
Treat.core.paths.models = '/X/models/'

Where /X/ is the Treat gem folder.

Databases

Currently, Treat only supports MongoDB, although support for more DB formats are on the way. You can configure MongoDB as such:

Treat.databases.mongo.db = 'your_database'
Treat.databases.mongo.host = 'localhost'
Treat.databases.mongo.port = '27017'

Entities

Textual entities can be created by using the special "builder" methods available in the global namespace. These methods are aliases for the corresponding Treat::Entities::X.build(...) methods, so that Word('hello') is equivalent to Treat::Entities::Word.build('hello').

###Creating Textual Entities From Strings

# Create a word
word = Word('run')
	
# Create a phrase
phrase = Phrase('am running')

# Create a sentence
sentence = Sentence('Welcome to Treat!')

# Create a section
section = Section("A small text\nA factitious paragraph.")

###Creating Documents From Files or URLs

# If a filename is supplied, the file format and 
# the appropriate parser to use will be determined
# based on the file extension:
d = Document('text.extension')
# N.B. Supports .txt, .doc, .htm(l), .abw, .odt;
# .pdf can be parsed with poppler-utils installed
# .doc can be parsed with antiword installed
# .jpg, .gif or .png with ocropus installed
# Tip: `port install ocropus poppler antiword`

# If a URL is specified, the file will be downloaded
# and then parsed as a regular file:
d = Document('http://www.example.com/XX/XX')
# N.B. By default, files will be downloaded into the 
# '/files/' folder of the gem's directory. This can 
# be changed by modifying 'Treat.core.paths.files'
# N.B. Format will assumed to be HTML if the web page
# does not have any file extension. Otherwise,
# will be determined based on the file extension.

# If a hash is provided, that hash will be used as
# a selector to retrieve a document from the DB.
# This can be done based on the ID of the document:
d = Document({id: 103757301323})
# Or through any given feature of a certain document:
d = Document({'features.file' => 'somefile.txt'})
# N.B. Currently, MongoDB is the only supported DB.
# You need to configure the database adapter before
# using this particular way of creating collections.

###Creating Collections from Folders

A collection is a set of documents that are grouped together.

# If an existing folder is passed to the builder,
# that folder is recursively searched for files 
# with any of the supported formats, and these 
# files are loaded into the collection object:
c = Collection('existing_folder')

# If a non-existing folder is passed to the builder,
# that folder is created and collection is opened:
c = Collection('new_folder')

# If a collection has been created with a folder,
# documents added to the collection are copied 
# to the newly created folder:
c = Collection('some_folder')
d = Document 'http://www.someurl.com'
c << d

# If a hash is passed to the builder, that hash
# will be used as a selector to retrieve documents
# from the DB, and a collection containing these 
# documents will be loaded. An empty hash loads 
# all documents in the DB:
c = Collection({})
# You can also use any arbitrary feature that 
# you have defined on the documents to create 
# a collection on-the-fly:
c = Collection({'features.topic' => 'news'})
# N.B. Currently, MongoDB is the only supported DB.
# You need to configure the database adapter before
# using this particular way of creating collections.
# N.B. When collections are created this way,
# copying a new document into the collection has no
# persistent effect (unlike if the collection was
# created with a specific folder name in mind).

Transparent String-to-Entity Casting

If a method defined by Treat is called on a plain old string object, that method will be caught through method_missing(), and Treat will attempt to cast the string object to the proper type of textual entity using String#to_entity. The method will then be called on the result of that casting. For example, let's say we do something like 'glass'.synonyms. The string 'glass' will be casted to an object of type Treat::Entities::Word, which happens to have a hook to respond to synonyms. Thus, the synonyms of 'glass' will be returned. In other words, "glass".synonyms is shorthand for "glass".to_entity.synonyms, which itself is equivalent to Word("glass").synonyms.

Consider the following examples to further illustrate how casting works:

# Plain string that is provided                         # => Casted type
"A syntactical phrase".to_entity                        # => Phrase
"A super little sentence.".to_entity                    # => Sentence
"Sentence number one. Sentence number two.".to_entity   # => Paragraph
"A title\nA short amount of text.".to_entity            # => Section

##Text Processing

About Text Processors

The first step once a textual entity has been created is usually to split it into smaller bits and pieces that are more useful to work with. Treat allows to successively split a text into logical zones, sentences, syntactical phrases, and, finally, tokens (which include words, numbers, punctuation, etc.) All text processors work destructively on the receiving object, returning the modified object. They add the results of their operations in the @children hash of the receiving object. Note that each of these methods are only available on specific types of entities, which are bolded in the text below.

Workers for Text Processors

Also note that when called without any options, each of the text processing tasks will be done using the default worker. The default worker will be determined based on the format of the supplied file (for chunkers) or the language of the text (for segmenters, tokenizers and parsers). You can specify a non-default worker by passing it as a symbol to the method, e.g. paragraph.segment :punkt, sentence.parse :enju, sentence.tokenize :stanford, etc.

###Chunkers

Chunkers split a document into its logical sections (text subdivisions) and zones (titles and paragraphs). A section contains at least one zone, and usually contains a title along with multiple paragraphs.

d = Document('doc.html').chunk

###Segmenters

Segmenters split a zone of text (a title or a paragraph) into sentences.

p = Paragraph('A walk in the park. A trip on a boat.').segment

###Tokenizers

Tokenizers split groups of words (sentences, phrases and fragments) into tokens.

s = Sentence('An uninteresting sentence, yes it is.').tokenize

###Parsers

Parsers parse a groups of words (sentences, phrases and fragments) into their syntactical tree.

s = Sentence('The prospect of an Asian arms race is genuinely frightening.').parse

###Chains

You can chain any number of processors using do. This allows rapid splitting of any textual entity down to the desired granularity. The tree of the receiver will be recursively searched for entities to which each of the supplied processors can be applied.

sect = Section  "A walk in the park\n"+
'Obama and Sarkozy met this friday to investigate ' +
'the possibility of a new rescue plan. The French ' +
'president Sarkozy is to meet Merkel next Tuesday.'

sect.do(:chunk, :segment, :parse)

Non-Default Text Processors

As when calling the text processors directly, you can specify a non-default text processor to use when chaining them. The name of the text processor becomes a hash key, pointing to a value which represents the worker to use:

sect.do(:chunk => :txt, :segment => :punkt, :parse => :stanford)

Note that you should not mix and match the "default worker" syntax with the "non-default worker" syntax, e.g.:

sect.do(:chunk => :txt, :segment, :tokenize)  # INCORRECT!

Instead, to keep the default workers for a text processing task, use the :default worker:

sect.do(:chunk => :txt, :segment => :default, :tokenize => :default)

##Annotations

Annotations are a type of metadata which can be grafted to a textual entity to help in further classification tasks. All annotators work destructively on the receiving object, and store their result in the @features hash of that object. Each annotator is available only on specific types of entities (it makes no sense to get the synonyms of a sentence, or the topics of a word). This section is split by entity type, and lists the annotations that are available on each particular type of entity.

Note that in many of the following examples, transparent string-to-entity casting is used. Refer to the relevant section above for more information.

###Manual Annotations

You can set your own arbitrary annotations on any entity by using set, check if an annotation is defined on an entity by using has?, and retrieve an annotation by using get:

w = Word('hello')
w.set  :topic, "conversation"
w.has? :topic  # => true
w.get  :topic  # => "conversation"

###Non-Specific Annotations

Language

The "language" annotation is available on all types of entities. Note that Treat.core.language.detect must be set to true for language detection to be performed when the language method is called. Otherwise, Treat.core.language.default will be returned regardless of the actual content of the entity.

    Treat.core.language.detect = true

    a = "I want to know God's thoughts; the rest are details. - Albert Einstein"
    b = "El mundo de hoy no tiene sentido, así que ¿por qué debería pintar cuadros que lo tuvieran? - Pablo Picasso"
    c = "Un bon Allemand ne peut souffrir les Français, mais il boit volontiers les vins de France. - Goethe"
    d = "Wir haben die Kunst, damit wir nicht an der Wahrheit zugrunde gehen. - Friedrich Nietzsche"

    puts a.language    # => :english
    puts b.language    # => :spanish
    puts c.language    # => :french
    puts d.language    # => :german

###Annotations Available on Words

Lexical Category and Tag

'running'.tag			# => "VBG"
'running'.category              # => "noun"
'inflection'.tag		# => "NN"
'inflection'.category		# => "noun"

Synonyms, Antonyms, Hypernyms and Hyponyms

'ripe'.synonyms
	# => ["mature", "ripe(p)", "good", "right", "advanced"]
'ripe'.antonyms
	# => ["green", "unripe", "unripened", "immature"]
'coffee'.hypernyms
	# => ["beverage", "drink",  [...], "drinkable", "potable"]
'juice'.hyponyms
	# => ["lemon_juice", "lime_juice", [...], "digestive_fluid"]

Word Stemming

'running'.stem			# => "run"
'inflection'.stem		# => "inflect"

Noun and Adjective Declensions

'inflection'.plural		# => "inflections"
'inflections'.singular		# => "inflection"

Verb Inflections

'running'.infinitive		# => "run"
'run'.present_participle	# => "running"
'runs'.plural_verb		# => "run"

Ordinals & Cardinals

20.ordinal			# => "twentieth"
20.cardinal			# => "twenty"

Named Entity Tags

The annotation :name_tag allows to retrieve person, location and time expressions in texts.

    p = Paragraph "Obama and Sarkozy met on January 1st to investigate the possibility of a new rescue plan." +
        "President Sarkozy is to meet Merkel next Tuesday in Berlin."

    p.do(:chunk, :segment, :tokenize, :name_tag)

Date and Time

The annotation :time allows to retrieve natural language expressions describing events in time.

    s = Section  "A bad day for leaders\n2011-12-23 - Obama and Sarkozy announced that they will start meeting every Tuesday."
   
    s.do(
      :chunk, :segment, :parse, :time,
      :visualize => [:dot, {:file => 'test-time-extraction.dot'}]
    )

Documents, Sections and Zones

Keyword Extraction

The annotation :keywords allows to retrieve the keywords of a document, section or zone. Uses a naive TF*IDF approach.

    Treat.sweeten!
    c = Collection 'economist'

    c.do(
      :chunk, :segment,
      :tokenize, :keywords,
      :visualize => [
        :dot, { 
          :file => 'test-tfidf.dot', 
          :remove_types => [:section] 
        }
      ]
    )

 Notice that we filter out every section (and their children) when calling the DOT visualizer. Otherwise, the graph would be very cluttered.

General Topic

The annotation :topic allows to retrieve the general topic of a document, section or zone. Uses a model trained on a large set of Reuters articles.

    Treat.sweeten!

    s = Paragraph 'Michigan, Ohio (Reuters) - Unfortunately, the RadioShack is closing.'

    s.do(
      :segment, :tokenize, :topics,
      :visualize => [:dot, {:file => 'test-topics-reuters.dot'}]
    )

Collections

Topic Words

The annotation :topic_words allows you to retrieve clusters of topics within documents. Uses Latent Dirichlet Allocation (LDA).

    require 'treat'
    Treat.sweeten!
    c = Collection 'economist'

    c.do(:chunk, :segment, :tokenize)

    puts c.topic_words(
      :lda,
      :num_topics => 10,
      :words_per_topic => 5,
      :iterations => 20
    ).inspect

Computers

Collection

Indexing

This builds a searchable index for a collection.

c = Collection 'folder'
c.index

Searching

This allows to retrieve documents inside a collection by searching through the index.

c.search(:q => 'some query').each do |doc|
  # Process the document of interest
end

Other Languages

Current support for other languages is as follows, in theory (most of these models are untested):

  • Parsers: English, French, German, Arabic, Chinese.
  • Segmenters: Dutch, English, French, German, Greek, Italian, Polish, Portuguese, Russian, Spanish and Swedish.
  • Taggers: English, French, German, Arabic, Chinese.
  • Tag sets:
    • Penn Treebank for English
    • Stuttgart-Tübingen Tag Set for German
    • Paris7 Tag Set for French

Refer to configuration options for how to set language defaults or automatic detection appropriately.

Example: POS Tagging with Language Detection

Treat.detect_language = true

s1 = "I want to know God's thoughts; the rest are details - AE. "
s2 = "Bienvenue au château! Mettez-vous tous bien à l'aise - SI."
par = Paragraph (s1 + s2)

par.do :segment, :tag, :category

##Extending Treat

Extending Treat within your program is extremely simple. You can dynamically define your own pluggable workers by adding them in the right group. Once this is done, the algorithm will be available on the right types of entities. For example, if you add a stemmer, it will be callable on words or strings representing words. If you add a tokenizer, it will be callable on any phrase/sentence or a string representing one.

Stemmer Example

Here is a dummy stemmer that removes the last letter of a word:

Treat::Workers::Inflectors::Stemmers.add(:dummy) do |word, options={}| 
  word.to_s[0..-2]
end

'dummy'.stem(:dummy)     # => dumm

Tokenizer Example

Here is a tokenizer that naively splits on space characters:

Treat::Workers::Processors::Tokenizers.add(:dummy) do |sentence, options={}| 
  sentence.to_s.split(' ').each do |token|
    sentence << Treat::Entities::Token.from_string(token)
  end
end

s = Sentence 'A sentence to tokenize.'
s.tokenize(:dummy)
Add a custom footer
Add a custom sidebar
Clone this wiki locally