Python bindings to the Compact Language Detector
Switch branches/tags
Clone or download
Pull request Compare This branch is 36 commits ahead of aboSamoor:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
bindings
ci
cld2
cld2full
msinttypes
tests
.editorconfig
.gitignore
.propspector.yml
.travis.yml
AUTHORS.rst
BUG_REPORTS.rst
CONTRIBUTING.rst
LICENSE
MANIFEST.in
README.rst
appveyor.yml
requirements.txt
setup.py
tox.ini

README.rst

CLD2-CFFI - Python (CFFI) Bindings for Compact Language Detector 2

CFFI bindings for CLD2


Latest version released on PyPi Build status Windows Build Status:: Coverage Code Health


This package contains the CLD (Compact Language Detection) library as maintained by Dick Sites (https://code.google.com/p/cld2/). The first fork was done at revision r161. It also contains python bindings that were originally created by Mike McCandless. The bindings have gone through several hands, with the latest changes being made to rework the bindings for CFFI.

These bindings are identical in API to the original cld2 bindings, and as a result can be used as a drop in replacement.

The LICENSE is the same as Chromium's LICENSE and is included in the LICENSE file for reference.

Installing

Should be as simple as

$ pip install cld2-cffi

Development Version

The latest development version can be installed directly from GitHub:

$ pip install --upgrade 'git+https://github.com/GregBowyer/cld2-cffi.git'

Usage

import cld2

isReliable, textBytesFound, details = cld2.detect("This is my sample text")
print('  reliable: %s' % (isReliable != 0))
print('  textBytes: %s' % textBytesFound)
print('  details: %s' % str(details))

# The output looks like so:
#  reliable: True
#  textBytes: 24
#  details: (('ENGLISH', 'en', 95, 1736.0), ('Unknown', 'un', 0, 0.0), ('Unknown', 'un', 0, 0.0))

Documentation

First, you must get your content (plain text or HTML) encoded into UTF8 bytes. Then, detect like this:

isReliable, textBytesFound, details = cld2.detect(bytes)
isReliable
is True if the top language is much better than 2nd best language.
textBytesFound
tells you how many actual bytes CLD analyzed (after removing HTML tags, collapsing areas of too-many-spaces, etc.).
details
has an entry per top 3 languages that matched, that includes the percent confidence of the match as well as a separate normalized score.

The module exports these global constants:

cld2.ENCODINGS
list of the encoding names CLD recognizes (if you provide hintEncoding, it must be one of these names).
cld2.LANGUAGES
list of languages and their codes (if you provide hintLanguageCode, it must be one of the codes from these codes).
cld2.EXTERNAL_LANGUAGES
list of external languages and their codes. Note that external languages cannot be hinted, but may be matched if you pass includeExtendedLanguages=True (the default).
cld2.DETECTED_LANGUAGES
list of all detectable languages, as best I can determine (this was reverse engineered from a unit test, ie it contains a language X if that language was tested and passes for at least one example text).

Authors

Please see AUTHORS.

Reporting bugs

Please see BUG_REPORTS.

Contribute

Please see CONTRIBUTING.

Licence

Please see LICENSE.