Skip to content
A C# port of Mozilla Universal Charset Detector.
Branch: master
Clone or download
Pull request Compare This branch is 57 commits ahead of errepi:master.
Latest commit b4c8cb6 Jan 18, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
license
m4
src add info Jan 18, 2019
.gitattributes create .gitattributes Mar 17, 2017
.gitignore create .gitattributes Mar 17, 2017
AUTHORS
COPYING initial import fixed Mar 29, 2009
ChangeLog initial import fixed Mar 29, 2009
INSTALL initial import fixed Mar 29, 2009
Makefile.am
Makefile.include initial import fixed Mar 29, 2009
NEWS initial import fixed Mar 29, 2009
README initial import fixed Mar 29, 2009
README.md update readme Mar 18, 2017
Ude.mds initial import fixed Mar 29, 2009
Ude.nuspec
Ude.sln Reconstruct project file Jan 18, 2019
aclocal.m4 initial import fixed Mar 29, 2009
appveyor.yml remove build steps Mar 18, 2017
autogen.sh initial import fixed Mar 29, 2009
configure initial import fixed Mar 29, 2009
configure.ac
install-sh initial import fixed Mar 29, 2009
missing initial import fixed Mar 29, 2009

README.md

NuGet

Build status

Ude is a C# port of Mozilla Universal Charset Detector.

The article "A composite approach to language/encoding detection" describes the charsets detection algorithms implemented by the library.

Ude can recognize the following charsets:

  • UTF-8
  • UTF-16 (BE and LE)
  • UTF-32 (BE and LE)
  • windows-1252 (mostly equivalent to iso8859-1)
  • windows-1251 and ISO-8859-5 (cyrillic)
  • windows-1253 and ISO-8859-7 (greek)
  • windows-1255 (logical hebrew. Includes ISO-8859-8-I and most of x-mac-hebrew)
  • ISO-8859-8 (visual hebrew)
  • Big5
  • gb18030 (superset of gb2312)
  • HZ-GB-2312
  • Shift-JIS
  • EUC-KR, EUC-JP, EUC-TW
  • ISO-2022-JP, ISO-2022-KR, ISO-2022-CN
  • KOI8-R
  • x-mac-cyrillic
  • IBM855 and IBM866
  • X-ISO-10646-UCS-4-3412 and X-ISO-10646-UCS-4-2413 (unusual BOM)
  • ASCII

Platform

  • Windows and Linux (Mono)
  • .NET Core
  • Universal Windows Platform
  • Windows Phone
  • Xamarin

Install

The release consists in the main library (Ude.dll),(Ude.NetStandard.dll) and a command-line client (udetect.exe) that can be used for one-shot tests.

On Windows, compile the Visual Studio 2017 solution ude.sln. On Linux you can build the library, the example and the nunit tests with monodelop and its solution ude.mds, or using make. To compile the sources tarball:

$ ./configure.sh --prefix=/usr/local --enable-tests=yes
$ make

To compile from svn:

$ ./autogen.sh --prefix=/usr/local --enable-tests=yes
$ make

You can pick the library (Ude.dll) from the toplevel build directory (./bin) or you can install it to $prefix/lib/ude by typing:

$ make install

This will installs a command-line example program ($prefix/bin/udetect) to test the library on a given file as:

$ udetect filename 

To run the nunit tests type:

$ make test

Usage

Example

public static void Main(String[] args)
{
    string filename = args[0];
    using (var fs = File.OpenRead(filename)) {
        Ude.CharsetDetector cdet = new Ude.CharsetDetector();
        cdet.Feed(fs);
        cdet.DataEnd();
        if (cdet.Charset != null) {
            Console.WriteLine("Charset: {0}, confidence: {1}", 
                 cdet.Charset, cdet.Confidence);
        } else {
            Console.WriteLine("Detection failed.");
        }
    }
}    

Other portings

The original Mozilla Universal Charset Detector has been ported to a variety of languages. Among these, a Java port:

from which I copied a few data structures, and a Python port:

License

The library is subject to the Mozilla Public License Version 1.1 (the "License"). Alternatively, it may be used under the terms of either the GNU General Public License Version 2 or later (the "GPL"), or the GNU Lesser General Public License Version 2.1 or later (the "LGPL").

Test data has been extracted from Wikipedia and The Project Gutenberg books and is subject to their licenses.

You can’t perform that action at this time.