AB-Natural Chinese Languange Processing, The movement of C makes it ab-natural.
Python Shell
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
develop
user
README.org

README.org

abcNLP Readme

Introduce

abcNLP

AB-Natural Chinese Languange Processing, The movement of C makes it ab-natural.

Thanks to censorship of internet, people oftern meet trouble to express themselves freely. Usually the SYSTEM will filter contents by (keywords) pattern matching. If we can make the language hard understood to SYSTEM, but not to human, we make the expression “Immune 2 Censorship” (at least, to some extent)!

Requirements

  • develop: CJKLIB, sqlite3
  • use: sqlite3

Usage

  • for example :
      abc = abcChineseChar()    
      for word in abc.getHxWords(u'茉莉花',  maxnum=5):
          print word    
        
  • output:
    苿峲芘
    苿峲芼
    苿萂芘
    炗峲芘
    苿峲芭
        

    (~ use tempory db, need to refine ~)

  • for example :
  • output :
    • ouput of 锦
      abc = abcChineseChar()    
      for char in abc.getHxCharacters(u'锦'):
          print char    		
        
      jǐn
      钅帛
      仅
      尽
      錦
      馑
        
    • ouput of 陈水扁:
      abc = abcChineseChar()    
      for word in abc.getHxWords(u'陈水扁',  maxnum=15):
          print word  
        
      <chén><shuǐ><biān>
      <阝东><shuǐ><biān>
      <chén><shuǐ>边
      <阝东><shuǐ>边
       尘<shuǐ><biān>
       臣<shuǐ><biān>
      <chén><shuǐ>便
      <chén><shuǐ>碥
       <chén><shuǐ>艑
      <chén>氵<biān>
       <阝东><shuǐ>便
       <阝东><shuǐ>碥
       <阝东><shuǐ>艑
       <阝东>氵<biān>
       尘<shuǐ>边
        

(~ use db version 1 ~)

Develope / Design

Methods to Immune 2 Censorship

Semantical Substitutes

Metaphor related methods. This method is not used in this project.

Use variants of the Character

The variant of a character is alao a valid character. It can be seen as another shape (form) of original one. But some of forms are rare used nowdays.

㒲 --> 財 
㒲 --> 才 
㒲 --> 财 
㒲 --> 纔 
㒷 --> 興 
㒷 --> 兴 
㓁 --> 网 
㓁 --> 網 
㓁 --> 罔

Use Character with same reading(Pinyin)

In the following list, value in brackets is score for the replacement. More small, more better.

㐲 --> dài (0) * Pinyin is a special type
㐲 --> 大 (3)
㐲 --> 代 (5)
㐲 --> 黱 (22)
㐳 --> wù (0)
㐳 --> 兀 (3)
㐳 --> 乌 (4)
㐳 --> 鼿 (17)

Split Character to two or three parts

After split character, each of its part can be further replaced with its similar character, which has ending mark of “1” in the following list.

川 --> <丿丨丨> 0
巧 --> <工丂> 0
垛 --> <土朶> 1
垜 --> <土朵> 1
⽻ --> <習習> 1
⾽ --> <镸三> 1
䜌 --> <⺯讠⺯> 1
丬 --> <氷丨> 1
乢 --> <山隠> 1
乣 --> <庅乚> 1
乨 --> <枱乚> 1
乩 --> <佔乚> 1
⽻ --> <习习> 0
⾽ --> <镸彡> 0
丬 --> <冫丨> 0
乢 --> <山乚> 0
乣 --> <幺乚> 0
乨 --> <台乚> 0
乩 --> <占乚> 0
亿 --> <亻乙> 0
什 --> <亻十> 0
仁 --> <亻二> 0
亿 --> <人乙> 1
什 --> <人十> 1
仁 --> <人二> 1
仂 --> <人力> 1
仃 --> <人丁> 1
仅 --> <人又> 1
仆 --> <人卜> 1
仇 --> <人九> 1

Choose a character looks like the origion. ( AI ?)

The score smaller is the better.

⺡ --> ⺍ (1)
⺡ --> 乊 (3)
⺡ --> 丬 (3)
⺡ --> 习 (4)
⺡ --> 乥 (8)
⺆ --> ⼌ (2)
⺆ --> ⼓ (3)
⺆ --> ⼏ (4)
⺆ --> 九 (4)
丨 --> ⼁ (0)
丨 --> ⼃ (2)
丨 --> 丿 (2)
丨 --> ⼅ (2)

This is the most important part of the project. After the module is refined, more examples will be added.

release

  1. Sun Mar 25 17:54:46 CST 2012 ,initial version
  2. Thu Mar 29 01:30:37 CST 2012, re-establish Repository
  3. Sun Apr 1 12:04:11 CST 2012, two db with different size

by wuliang [www.wuliang.cn at gmail.com]