Skip to content
This repository

Named entity recognition tool based on linear-chain CRFs

branch: master

Fetching latest commit…

Octocat-spinner-32-eaf2f5

Cannot retrieve the latest commit at this time

Octocat-spinner-32 src
Octocat-spinner-32 tools
Octocat-spinner-32 LICENSE
Octocat-spinner-32 README.md
Octocat-spinner-32 Setup.lhs
Octocat-spinner-32 nerf.cabal
README.md

Nerf

Nerf is a statistical named entity recognition (NER) tool based on linear-chain conditional random fields (CRFs). It has been adapted to recognize tree-like structures of NEs (i.e., with recursively embedded NEs) by using the joined label tagging method which -- for a particular sentence -- works as follows:

  • CRF model is used to determine the most probable sequence of labels,
  • Extended IOB method is used to decode the sequence into a forerst of NEs.

The extended IOB method also provides the inverse encoding function which is needed during the model training.

Building Nerf

You will need the Glasgow Haskell Compiler and the Cabal tool to build Nerf. The easiest way to get both GHC and Cabal is to install the latest Haskell Platform.

To install Nerf from the official Hackage repository just run:

cabal install nerf

If you want to update Nerf to a newer version you should update the package list first:

cabal update
cabal install nerf

To install the latest development version from github just run

cabal install

from the nerf top level directory.

Data formats

The only data encoding supported by Nerf is UTF-8.

Training data

The current version of Nerf works with a simple data format in which:

  • Each sentence is kept in a separate line,
  • Named entities are represented with embedded beginning and ending tags,
  • Contents of individual tags represent named entity types.

For example:

<organization>Church of the <deity>Flying Spaghetti Monster</deity></organization>

Text and label values should be escaped by prepending the \ character before special >, <, \ and (space) characters.

NER input data

Below is a list of data formats supported within the NER mode.

Raw text

Nerf can be used to annotate raw text with named entites. The annotated data will be presented in the format which is also used for training and has already been described above. Each sentence should be supplied in a separate line -- currently, Nerf doesn't perform any sentence-level segmentation.

XCES format

It is also possible to annotate data stored in the XCES format.

Training

Once you have an annotated data file train.nes (and, optionally, an evaluation material eval.nes) conformant with the format described above you can train the Nerf model using the following command:

nerf train train.nes -e eval.nes -o model.bin

Run nerf train --help to learn more about the program arguments and possible training options.

The nerf tool can be also supplied with additional runtime system options. For example, to train the model using four threads, use:

nerf train train.nes -e eval.nes -o model.bin +RTS -N4

Dictionaries

Nerf supports a list of NE-related dictionaries:

To use the particular dictionary during NER you have to supply it as a command line argument during the training process, for example:

nerf train train.nes --polimorf PoliMorf-0.6.1.tab

Named entity recognition

To annotate the input.txt data file using the trained model.bin model, run:

nerf ner model.bin < input.txt

Annotated data will be printed to stdout. Data formats currently supported within the NER mode has been described above. Run nerf ner --help to learn more about the additional NER arguments.

Server

Nerf provides also a client/server mode. It is handy when, for example, you need to annotate a large collection of small files. Loading Nerf model from a disk takes considerable amount of time which makes the tagging method described above very slow in such a setting.

To start the Nerf server, run:

nerf server model.bin

You can supply a custom port number using a --port option. For example, to run the server on the 10101 port, use the following command:

nerf server model.bin --port 10101

To use the server in a multi-threaded environment, you need to specify the -N RTS option. A set of options which usually yield good server performance is presented in the following example:

nerf server model.bin +RTS -N -A4M -qg1 -I0

Run nerf server --help to learn more about possible server-mode options.

The client mode works just like the tagging mode. The only difference is that, instead of supplying your client with a model, you need to specify the port number (in case you used a custom one when starting the server; otherwise, the default port number will be used).

nerf client --port 10101 < input.txt > output.nes

Run nerf client --help to learn more about the possible client-mode options.

Something went wrong with that request. Please try again.