Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Remove app/unidecode and reinstall it in app/vendors using pip command.
- Loading branch information
Showing
199 changed files
with
1,182 additions
and
100 deletions.
There are no files selected for viewing
185 changes: 185 additions & 0 deletions
185
app/vendors/Unidecode-0.04.20.dist-info/DESCRIPTION.rst
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,185 @@ | ||
Unidecode, lossy ASCII transliterations of Unicode text | ||
======================================================= | ||
|
||
It often happens that you have text data in Unicode, but you need to | ||
represent it in ASCII. For example when integrating with legacy code that | ||
doesn't support Unicode, or for ease of entry of non-Roman names on a US | ||
keyboard, or when constructing ASCII machine identifiers from | ||
human-readable Unicode strings that should still be somewhat intelligible | ||
(a popular example of this is when making an URL slug from an article | ||
title). | ||
|
||
In most of these examples you could represent Unicode characters as | ||
`???` or `\\15BA\\15A0\\1610`, to mention two extreme cases. But that's | ||
nearly useless to someone who actually wants to read what the text says. | ||
|
||
What Unidecode provides is a middle road: function `unidecode()` takes | ||
Unicode data and tries to represent it in ASCII characters (i.e., the | ||
universally displayable characters between 0x00 and 0x7F), where the | ||
compromises taken when mapping between two character sets are chosen to be | ||
near what a human with a US keyboard would choose. | ||
|
||
The quality of resulting ASCII representation varies. For languages of | ||
western origin it should be between perfect and good. On the other hand | ||
transliteration (i.e., conveying, in Roman letters, the pronunciation | ||
expressed by the text in some other writing system) of languages like | ||
Chinese, Japanese or Korean is a very complex issue and this library does | ||
not even attempt to address it. It draws the line at context-free | ||
character-by-character mapping. So a good rule of thumb is that the further | ||
the script you are transliterating is from Latin alphabet, the worse the | ||
transliteration will be. | ||
|
||
Note that this module generally produces better results than simply | ||
stripping accents from characters (which can be done in Python with | ||
built-in functions). It is based on hand-tuned character mappings that for | ||
example also contain ASCII approximations for symbols and non-Latin | ||
alphabets. | ||
|
||
This is a Python port of `Text::Unidecode` Perl module by | ||
Sean M. Burke <sburke@cpan.org>. | ||
|
||
|
||
Module content | ||
-------------- | ||
|
||
The module exports a function that takes an Unicode object (Python 2.x) or | ||
string (Python 3.x) and returns a string (that can be encoded to ASCII bytes in | ||
Python 3.x):: | ||
|
||
>>> from unidecode import unidecode | ||
>>> unidecode(u'ko\u017eu\u0161\u010dek') | ||
'kozuscek' | ||
>>> unidecode(u'30 \U0001d5c4\U0001d5c6/\U0001d5c1') | ||
'30 km/h' | ||
>>> unidecode(u"\u5317\u4EB0") | ||
'Bei Jing ' | ||
|
||
A utility is also included that allows you to transliterate text from the | ||
command line in several ways. Reading from standard input:: | ||
|
||
$ echo hello | unidecode | ||
hello | ||
|
||
from a command line argument:: | ||
|
||
$ unidecode -c hello | ||
hello | ||
|
||
or from a file:: | ||
|
||
$ unidecode hello.txt | ||
hello | ||
|
||
The default encoding used by the utility depends on your system locale. You can specify another encoding with the `-e` argument. See `unidecode --help` for a full list of available options. | ||
|
||
Requirements | ||
------------ | ||
|
||
Nothing except Python itself. | ||
|
||
You need a Python build with "wide" Unicode characters (also called "UCS-4 | ||
build") in order for unidecode to work correctly with characters outside of | ||
Basic Multilingual Plane (BMP). Common characters outside BMP are bold, italic, | ||
script, etc. variants of the Latin alphabet intended for mathematical notation. | ||
Surrogate pair encoding of "narrow" builds is not supported in unidecode. | ||
|
||
If your Python build supports "wide" Unicode the following expression will | ||
return True:: | ||
|
||
>>> import sys | ||
>>> sys.maxunicode > 0xffff | ||
True | ||
|
||
See PEP 261 for details regarding support for "wide" Unicode characters in | ||
Python. | ||
|
||
|
||
Installation | ||
------------ | ||
|
||
To install the latest version of Unidecode from the Python package index, use | ||
these commands:: | ||
|
||
$ pip install unidecode | ||
|
||
To install Unidecode from the source distribution and run unit tests, use:: | ||
|
||
$ python setup.py install | ||
$ python setup.py test | ||
|
||
|
||
Performance notes | ||
----------------- | ||
|
||
By default, `unidecode` optimizes for the use case where most of the strings | ||
passed to it are already ASCII-only and no transliteration is necessary (this | ||
default might change in future versions). | ||
|
||
For performance critical applications, two additional functions are exposed: | ||
|
||
`unidecode_expect_ascii` is optimized for ASCII-only inputs (approximately 5 | ||
times faster than `unidecode_expect_nonascii` on 10 character strings, more on | ||
longer strings), but slightly slower for non-ASCII inputs. | ||
|
||
`unidecode_expect_nonascii` takes approximately the same amount of time on | ||
ASCII and non-ASCII inputs, but is slightly faster for non-ASCII inputs than | ||
`unidecode_expect_ascii`. | ||
|
||
Apart from differences in run time, both functions produce identical results. | ||
For most users of Unidecode, the difference in performance should be | ||
negligible. | ||
|
||
|
||
Source | ||
------ | ||
|
||
You can get the latest development version of Unidecode with:: | ||
|
||
$ git clone https://www.tablix.org/~avian/git/unidecode.git | ||
|
||
There is also an official mirror of this repository on GitHub at | ||
https://github.com/avian2/unidecode | ||
|
||
|
||
Contact | ||
------- | ||
|
||
Please send bug reports, patches and suggestions for Unidecode to | ||
tomaz.solc@tablix.org. | ||
|
||
Alternatively, you can also open a ticket or pull request at | ||
https://github.com/avian2/unidecode | ||
|
||
|
||
Copyright | ||
--------- | ||
|
||
Original character transliteration tables: | ||
|
||
Copyright 2001, Sean M. Burke <sburke@cpan.org>, all rights reserved. | ||
|
||
Python code and later additions: | ||
|
||
Copyright 2017, Tomaz Solc <tomaz.solc@tablix.org> | ||
|
||
This program is free software; you can redistribute it and/or modify it | ||
under the terms of the GNU General Public License as published by the Free | ||
Software Foundation; either version 2 of the License, or (at your option) | ||
any later version. | ||
|
||
This program is distributed in the hope that it will be useful, but WITHOUT | ||
ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or | ||
FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for | ||
more details. | ||
|
||
You should have received a copy of the GNU General Public License along | ||
with this program; if not, write to the Free Software Foundation, Inc., 51 | ||
Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. The programs and | ||
documentation in this dist are distributed in the hope that they will be | ||
useful, but without any warranty; without even the implied warranty of | ||
merchantability or fitness for a particular purpose. | ||
|
||
.. | ||
vim: set filetype=rst: | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,201 @@ | ||
Metadata-Version: 2.0 | ||
Name: Unidecode | ||
Version: 0.04.20 | ||
Summary: ASCII transliterations of Unicode text | ||
Home-page: UNKNOWN | ||
Author: Tomaz Solc | ||
Author-email: tomaz.solc@tablix.org | ||
License: GPL | ||
Platform: UNKNOWN | ||
Classifier: License :: OSI Approved :: GNU General Public License v2 or later (GPLv2+) | ||
Classifier: Programming Language :: Python | ||
Classifier: Programming Language :: Python :: 2 | ||
Classifier: Programming Language :: Python :: 3 | ||
Classifier: Topic :: Text Processing | ||
Classifier: Topic :: Text Processing :: Filters | ||
|
||
Unidecode, lossy ASCII transliterations of Unicode text | ||
======================================================= | ||
|
||
It often happens that you have text data in Unicode, but you need to | ||
represent it in ASCII. For example when integrating with legacy code that | ||
doesn't support Unicode, or for ease of entry of non-Roman names on a US | ||
keyboard, or when constructing ASCII machine identifiers from | ||
human-readable Unicode strings that should still be somewhat intelligible | ||
(a popular example of this is when making an URL slug from an article | ||
title). | ||
|
||
In most of these examples you could represent Unicode characters as | ||
`???` or `\\15BA\\15A0\\1610`, to mention two extreme cases. But that's | ||
nearly useless to someone who actually wants to read what the text says. | ||
|
||
What Unidecode provides is a middle road: function `unidecode()` takes | ||
Unicode data and tries to represent it in ASCII characters (i.e., the | ||
universally displayable characters between 0x00 and 0x7F), where the | ||
compromises taken when mapping between two character sets are chosen to be | ||
near what a human with a US keyboard would choose. | ||
|
||
The quality of resulting ASCII representation varies. For languages of | ||
western origin it should be between perfect and good. On the other hand | ||
transliteration (i.e., conveying, in Roman letters, the pronunciation | ||
expressed by the text in some other writing system) of languages like | ||
Chinese, Japanese or Korean is a very complex issue and this library does | ||
not even attempt to address it. It draws the line at context-free | ||
character-by-character mapping. So a good rule of thumb is that the further | ||
the script you are transliterating is from Latin alphabet, the worse the | ||
transliteration will be. | ||
|
||
Note that this module generally produces better results than simply | ||
stripping accents from characters (which can be done in Python with | ||
built-in functions). It is based on hand-tuned character mappings that for | ||
example also contain ASCII approximations for symbols and non-Latin | ||
alphabets. | ||
|
||
This is a Python port of `Text::Unidecode` Perl module by | ||
Sean M. Burke <sburke@cpan.org>. | ||
|
||
|
||
Module content | ||
-------------- | ||
|
||
The module exports a function that takes an Unicode object (Python 2.x) or | ||
string (Python 3.x) and returns a string (that can be encoded to ASCII bytes in | ||
Python 3.x):: | ||
|
||
>>> from unidecode import unidecode | ||
>>> unidecode(u'ko\u017eu\u0161\u010dek') | ||
'kozuscek' | ||
>>> unidecode(u'30 \U0001d5c4\U0001d5c6/\U0001d5c1') | ||
'30 km/h' | ||
>>> unidecode(u"\u5317\u4EB0") | ||
'Bei Jing ' | ||
|
||
A utility is also included that allows you to transliterate text from the | ||
command line in several ways. Reading from standard input:: | ||
|
||
$ echo hello | unidecode | ||
hello | ||
|
||
from a command line argument:: | ||
|
||
$ unidecode -c hello | ||
hello | ||
|
||
or from a file:: | ||
|
||
$ unidecode hello.txt | ||
hello | ||
|
||
The default encoding used by the utility depends on your system locale. You can specify another encoding with the `-e` argument. See `unidecode --help` for a full list of available options. | ||
|
||
Requirements | ||
------------ | ||
|
||
Nothing except Python itself. | ||
|
||
You need a Python build with "wide" Unicode characters (also called "UCS-4 | ||
build") in order for unidecode to work correctly with characters outside of | ||
Basic Multilingual Plane (BMP). Common characters outside BMP are bold, italic, | ||
script, etc. variants of the Latin alphabet intended for mathematical notation. | ||
Surrogate pair encoding of "narrow" builds is not supported in unidecode. | ||
|
||
If your Python build supports "wide" Unicode the following expression will | ||
return True:: | ||
|
||
>>> import sys | ||
>>> sys.maxunicode > 0xffff | ||
True | ||
|
||
See PEP 261 for details regarding support for "wide" Unicode characters in | ||
Python. | ||
|
||
|
||
Installation | ||
------------ | ||
|
||
To install the latest version of Unidecode from the Python package index, use | ||
these commands:: | ||
|
||
$ pip install unidecode | ||
|
||
To install Unidecode from the source distribution and run unit tests, use:: | ||
|
||
$ python setup.py install | ||
$ python setup.py test | ||
|
||
|
||
Performance notes | ||
----------------- | ||
|
||
By default, `unidecode` optimizes for the use case where most of the strings | ||
passed to it are already ASCII-only and no transliteration is necessary (this | ||
default might change in future versions). | ||
|
||
For performance critical applications, two additional functions are exposed: | ||
|
||
`unidecode_expect_ascii` is optimized for ASCII-only inputs (approximately 5 | ||
times faster than `unidecode_expect_nonascii` on 10 character strings, more on | ||
longer strings), but slightly slower for non-ASCII inputs. | ||
|
||
`unidecode_expect_nonascii` takes approximately the same amount of time on | ||
ASCII and non-ASCII inputs, but is slightly faster for non-ASCII inputs than | ||
`unidecode_expect_ascii`. | ||
|
||
Apart from differences in run time, both functions produce identical results. | ||
For most users of Unidecode, the difference in performance should be | ||
negligible. | ||
|
||
|
||
Source | ||
------ | ||
|
||
You can get the latest development version of Unidecode with:: | ||
|
||
$ git clone https://www.tablix.org/~avian/git/unidecode.git | ||
|
||
There is also an official mirror of this repository on GitHub at | ||
https://github.com/avian2/unidecode | ||
|
||
|
||
Contact | ||
------- | ||
|
||
Please send bug reports, patches and suggestions for Unidecode to | ||
tomaz.solc@tablix.org. | ||
|
||
Alternatively, you can also open a ticket or pull request at | ||
https://github.com/avian2/unidecode | ||
|
||
|
||
Copyright | ||
--------- | ||
|
||
Original character transliteration tables: | ||
|
||
Copyright 2001, Sean M. Burke <sburke@cpan.org>, all rights reserved. | ||
|
||
Python code and later additions: | ||
|
||
Copyright 2017, Tomaz Solc <tomaz.solc@tablix.org> | ||
|
||
This program is free software; you can redistribute it and/or modify it | ||
under the terms of the GNU General Public License as published by the Free | ||
Software Foundation; either version 2 of the License, or (at your option) | ||
any later version. | ||
|
||
This program is distributed in the hope that it will be useful, but WITHOUT | ||
ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or | ||
FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for | ||
more details. | ||
|
||
You should have received a copy of the GNU General Public License along | ||
with this program; if not, write to the Free Software Foundation, Inc., 51 | ||
Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. The programs and | ||
documentation in this dist are distributed in the hope that they will be | ||
useful, but without any warranty; without even the implied warranty of | ||
merchantability or fitness for a particular purpose. | ||
|
||
.. | ||
vim: set filetype=rst: | ||
|
Oops, something went wrong.