Unicode to ASCII transliteration
Table of Contents
- Description Examples
- Background Stats Unidecode Sources
Converts Unicode characters to their best ASCII representation
AnyAscii provides ASCII-only replacement strings for practically all Unicode characters. Text is converted character-by-character without considering the context. The mappings for each script are based on popular existing romanization systems. Symbolic characters are converted based on their meaning or appearance. All ASCII characters in the input are left unchanged, every other character is replaced with printable ASCII characters. Unknown characters and some known characters are replaced with an empty string and removed.
Representative examples for different languages comparing the AnyAscii output to the conventional romanization:
|French (Latin)||René François Lacôte||Rene Francois Lacote||Rene Francois Lacote|
|Vietnamese (Latin)||Trần Hưng Đạo||Tran Hung Dao||Tran Hung Dao|
|Ancient Greek (Greek)||Φειδιππίδης||Feidippidis||Pheidippides|
|Modern Greek (Greek)||Δημήτρης Φωτόπουλος||Dimitris Fotopoylos||Dimitris Fotopoulos|
|Russian (Cyrillic)||Борис Николаевич Ельцин||Boris Nikolaevich El'tsin||Boris Nikolayevich Yeltsin|
|Ukrainian (Cyrillic)||Володимир Горбулін||Volodimir Gorbulin||Volodymyr Horbulin|
|Mandarin Chinese (Han)||深圳||ShenZhen||Shenzhen|
|Cantonese Chinese (Han)||深水埗||ShenShuiBu||Sham Shui Po|
|Amharic (Ethiopic)||ደብረ ዘይት||debre zeyt||Debre Zeyit|
|Hebrew||אברהם הלוי פרנקל||'vrhm hlvy frnkl||Abraham Halevi Fraenkel|
|Unified English Braille (Braille)||⠠⠎⠁⠽⠀⠭⠀⠁⠛||+say x ag||Say it again|
|Misc.||☆ ♯ ♰ ⚄ ⛌||* # + 5 X|
|Letterlike||№ ℳ ⅋ ⅍||No M & A/S|
AnyAscii is implemented across multiple programming languages with the same behavior and versioning
iex> AnyAscii.transliterate("άνθρωποι") |> IO.iodata_to_binary() "anthropoi"
import "github.com/anyascii/go" s := anyascii.Transliterate("άνθρωποι") // anthropoi
Go 1.10+ compatible
String s = AnyAscii.transliterate("άνθρωποι"); assert s.equals("anthropoi");
Java 6+ compatible
<dependency> <groupId>com.anyascii</groupId> <artifactId>anyascii</artifactId> <version>LATEST</version> </dependency>
import anyAscii from 'any-ascii'; const s = anyAscii('άνθρωποι'); // anthropoi
npm install any-ascii
julia> using AnyAscii julia> anyascii("άνθρωποι") "anthropoi"
Julia 1.0+ compatible
pkg> add AnyAscii
$s = AnyAscii::transliterate('άνθρωποι'); // anthropoi
PHP 5.3+ compatible
composer require anyascii/anyascii
from anyascii import anyascii s = anyascii('άνθρωποι') assert s == 'anthropoi'
Python 3.3+ compatible
pip install anyascii
require 'any_ascii' s = AnyAscii.transliterate('άνθρωποι') # anthropoi
Ruby 2.0+ compatible
gem install any_ascii
use any_ascii::any_ascii; let s = any_ascii("άνθρωποι"); // anthropoi
Rust 1.42+ compatible
# Cargo.toml [dependencies] any_ascii = "*"
cargo install any_ascii
$ anyascii άνθρωποι anthropoi $ echo άνθρωποι | anyascii anthropoi
$ anyascii άνθρωποι anthropoi $ echo άνθρωποι | anyascii anthropoi
// C# using AnyAscii; string s = "άνθρωποι".Transliterate(); // anthropoi
.NET Core 3.0+ and .NET 5.0+ compatible
Unicode is the universal character encoding. This encoding standard provides the basis for processing, storage and interchange of text data in any language in all modern software and information technology protocols. [Unicode's scope] covers all the characters for all the writing systems of the world, modern and ancient. It also includes technical symbols, punctuations, and many other characters used in writing text. *
Unicode provides a unique numeric value for each character and uses UTF-8 to encode sequences of characters into bytes. UTF-8 uses a variable number of bytes for each character and is backwards compatible with ASCII. UTF-16 and UTF-32 are also specified but not common. There is a name and various properties for each character along with algorithms for casing, collation, equivalence, line breaking, segmentation, text direction, and more.
ASCII is the lowest common denominator character encoding, established in 1967 and using 7 bits for 128 characters. The printable characters are English letters, digits, and punctuation, with the remaining being control characters. The characters found on a standard US keyboard are from ASCII. Most legacy 8-bit encodings were backwards compatible with ASCII.
... expressed only in the original non-control ASCII range so as to be as widely compatible with as many existing tools, languages, and serialization formats as possible and avoid display issues in text editors and source control *
A language is written using characters from a script. Some languages use multiple scripts and some scripts are used by multiple languages. English uses the Latin script which is based on the alphabet the Romans used for writing Latin. Other languages using the Latin script may require additional letters and diacritics.
The Unicode Standard encodes scripts rather than languages. When writing systems for more than one language share sets of graphical symbols that have historically related derivations, the union of all of those graphical symbols ... is identified as a single script. *
When converting text between languages there are multiple properties that can be preserved:
|Original||Transliteration (Spelling)||Transcription (Sound)||Translation (Meaning)|
Romanization is the conversion into the Latin script using transliteration and transcription, it is most commonly used when representing the names of people and places. Some nations have an official romanization standard for their language. Several organizations publish romanization standards for multiple languages.
Geographical names are Romanized to help foreigners find the place they intend to go to and help them remember cities, villages and mountains they visited and climbed. But it is Koreans who make up the Roman transcription of their proper names to print on their business cards and draw up maps for international tourists. Sometimes, they write the lyrics of a Korean song in Roman letters to help foreigners join in a singing session or write part of a public address (in Korean) in Roman letters for a visiting foreign VIP. In this sense, it is for both foreigners and the local public. The Romanization system must not be a code only for the native English-speaking community here but an important tool for international communication between Korean society, foreign residents in the country and the entire external world. *
Supports Unicode 14.0 (2021). Covers 99k of the 144k total Unicode characters, missing 43k very rare CJK characters and 2k other rare characters.
Bundled data files total 200-500 KB depending on the implementation
AnyAscii is an alternative to (and inspired by) Unidecode and its many ports. Unidecode only supports a subset of the basic mulitlingual plane. AnyAscii gives better results, supports more than twice as many characters, and often has a smaller file size. To compare the mappings see