Colibri core is an NLP tool as well as a C++ and Python library for working with basic linguistic constructions such as n-grams and skipgrams (i.e patterns with one or more gaps, either of fixed or dynamic size) in a quick and memory-efficient way. At the core is the tool ``colibri-patternmodeller`` whi ch allows you to build, view, manipulate a…
C++ Python Jupyter Notebook Shell M4 Makefile TeX
Clone or download
Latest commit 25a9847 Apr 19, 2018
Permalink
Failed to load latest commit information.
exp fixed broken test Aug 19, 2016
include use numeric_limits for max 32-bit value, rather than literal (solves … Feb 21, 2017
m4 fix Oct 22, 2013
scripts fixes and aliases for options Dec 3, 2015
src fix: colibri-classencode -t behaviour was wrong (was interpreted as +1) Sep 7, 2017
.travis.yml [travis] removing python 3.2 from build matrix Mar 1, 2018
AUTHORS intial add from colibri refactor branch Sep 21, 2013
ChangeLog major colibri refactoring, new pattern backend Sep 27, 2013
Doxyfile configuration for C++ API documentation using Doxygen Sep 16, 2015
MANIFEST.in added May 20, 2015
Makefile.am added codemeta.json Apr 18, 2018
NEWS intial add from colibri refactor branch Sep 21, 2013
README.rst added publication information Aug 2, 2016
benchmarks.py benchmark fix Jun 7, 2016
benchmarks.sh fix Jun 7, 2016
bootstrap added a test script Feb 16, 2017
brainstorm2 update Oct 14, 2015
codemeta.json added codemeta.json Apr 18, 2018
codereview_input bugfix in unigram token count Aug 28, 2014
colibri-core.pc.in Added ArrayValueHandler Oct 31, 2013
colibricore-python-tutorial.ipynb Python tutorial update, now with Python 2.7 compatibility Jun 10, 2016
colibricore_alignmodel.pxi forgot to add/commit Feb 16, 2017
colibricore_classes.in.pxd cleanup part two: removed trailing whitespaces in cython code, no fun… Feb 16, 2017
colibricore_indexedpatternmodel.pxi typo May 29, 2017
colibricore_patterndict.pxi cleanup part two: removed trailing whitespaces in cython code, no fun… Feb 16, 2017
colibricore_patternmodel.pxi cleanup part two: removed trailing whitespaces in cython code, no fun… Feb 16, 2017
colibricore_patternset.pxi cleanup part two: removed trailing whitespaces in cython code, no fun… Feb 16, 2017
colibricore_relationmap.clang.pxi Compiler detection and some compiler-conditional includes to compensa… Sep 18, 2015
colibricore_relationmap.gcc.pxi Compiler detection and some compiler-conditional includes to compensa… Sep 18, 2015
colibricore_unindexedpatternmodel.pxi cleanup part two: removed trailing whitespaces in cython code, no fun… Feb 16, 2017
colibricore_wrapper.in.pyx Implemented Python binding for BasicPatternAlignmentModel Feb 11, 2017
configure.ac Mac OS X compilation fix (explicitly use libc++) Mar 3, 2018
setup.py Mac OS X compilation fix (explicitly use libc++) Mar 3, 2018
test.py Fixed issue with cython and IndexCorpus iterator Nov 27, 2015
unordered_map.pxd update Oct 21, 2013
unordered_set.pxd updated cython code with unordered_set for getreverseindex Dec 1, 2015

README.rst

Colibri Core

https://travis-ci.org/proycon/colibri-core.svg?branch=master http://applejack.science.ru.nl/lamabadge.php/colibri-core

by Maarten van Gompel, proycon@anaproy.nl, Radboud University Nijmegen

Licensed under GPLv3 (See http://www.gnu.org/licenses/gpl-3.0.html)

Colibri Core is software to quickly and efficiently count and extract patterns from large corpus data, to extract various statistics on the extracted patterns, and to compute relations between the extracted patterns. The employed notion of pattern or construction encompasses the following categories:

  • n-gram -- n consecutive words
  • skipgram -- An abstract pattern of predetermined length with one or multiple gaps (of specific size).
  • flexgram -- An abstract pattern with one or more gaps of variable-size.

N-gram extraction may seem fairly trivial at first, with a few lines in your favourite scripting language, you can move a simple sliding window of size n over your corpus and store the results in some kind of hashmap. This trivial approach however makes an unnecessarily high demand on memory resources, this often becomes prohibitive if unleashed on large corpora. Colibri Core tries to minimise these space requirements in several ways:

  • Compressed binary representation -- Each word type is assigned a numeric class, which is encoded in a compact binary format in which highly frequent classes take less space than less frequent classes. Colibri core always uses this representation rather than a full string representation, both on disk and in memory.
  • Informed iterative counting -- Counting is performed more intelligently by iteratively processing the corpus in several passes and quickly discarding patterns that won't reach the desired occurrence threshold.

Skipgram and flexgram extraction are computationally more demanding but have been implemented with similar optimisations. Skipgrams are computed by abstracting over n-grams, and flexgrams in turn are computed either by abstracting over skipgrams, or directly from n-grams on the basis of co-occurrence information (mutual pointwise information).

At the heart of the sofware is the notion of pattern models. The core tool, to be used from the command-line, is colibri-patternmodeller which enables you to build pattern models, generate statistical reports, query for specific patterns and relations, and manipulate models.

A pattern model is simply a collection of extracted patterns (any of the three categories) and their counts from a specific corpus. Pattern models come in two varieties:

  • Unindexed Pattern Model -- The simplest form, which simply stores the patterns and their count.
  • Indexed Pattern Model -- The more informed form, which retains all indices to the original corpus, at the cost of more memory/diskspace.

The Indexed Pattern Model is much more powerful, and allows more statistics and relations to be inferred.

The generation of pattern models is optionally parametrised by a minimum occurrence threshold, a maximum pattern length, and a lower-boundary on the different types that may instantiate a skipgram (i.e. possible fillings of the gaps).

Technical Details

Colibri Core is available as a collection of standalone command-line tools, as a C++ library, and as a Python library.

Please consult the full documentation at https://proycon.github.io/colibri-core

Installation instructions are here: https://proycon.github.io/colibri-core/doc/#installation

Publication

This software is extensively described in the following peer-reviewed publication:

van Gompel, M and van den Bosch, A (2016) Efficient n-gram, Skipgram and Flexgram Modelling with Colibri Core. Journal of Open Research Software 4: e30, DOI: http://dx.doi.org/10.5334/jors.105

Click the link to access the publication and please cite it if you make use of Colibri Core in your work.