by Omer Shapira
ACCEPTING CONTRIBUTORS! Read the current tasks in the 'Issues' list and join in. If there are any questions, feel free to contact me at info∞omershapiraºcom.
Syntactic is a program that reads huge texts and divides common words in the text to categories. Here are some of its automatic categorizations in the Simple English Wikipedia:
Cluster 17 ---------- with, including, without, involving, containing, featuring, requiring, reaching, covering Cluster 52 ---------- city, town, district, province, university, river, county, site, community, village, moon, field, state, series
It does this by examining contexts in 3-grams. For example, if the sentences
"the cat sat on the mat"
"the dog sat on the porch"
appear in the text, then the words "cat" and "dog" are likely to appear in the same category.
When jar'd to
java -jar Syntactic.jar [name] [input folder] [output folder] [clusters] [threshold] [epsilon]
[name]is the corpus name. Only alphanumeric characters and underscores(_).
[input folder]is the folder where the corpus is. By default, only
.txtfiles are read.
[output folder]is a folder in which Syntactic will create the output root folder, with a timestamp. If it is set to
Output/, then Syntactic will place everything in
Output/CorpusName dd.MM.yy HH.mm.ss/
[clusters]the amount of resulting groups. Good results appear above
75. Speed decreases polynomially with the number of clusters. Default is
[threshold]the minimum frequency a word has to have in order to be clustered. Default is
[epsilon]clusters who are not mutually separated by this distance are merged. Values vary significantly. Typical values are between
The program has a replacable class for parsing texts, so it can be modified to remove XML tags (or just read them), or any other modification in regular expressions.
The program outputs JSON in a very chatty form (lots of info), which can be reduced quickly.
Syntactic was written by Omer Shapira, based on an algorithm described by Alexander Clark.
Syntactic \_syntaxLearner.java | |__LearnerMain.java |__Learner.java |__Recorder.java |__Cluster.java |__ClusterContext.java | \_Corpus . | . |__Context.java . |__VocabularyContext.java . |__Word.java . |__Corpus.java . |__Vocabulary.java . \_source . .| . .|__CorpusSource.java . .|__PlainTextFile.java . .|__WikiDump.java \_UI .| .|__Console.java .|__Report.java ..
Eventually, this project is planned to output navigable data about language, with data which can be used with NLP applications such as semantic web results, entity extraction, and automatic dictionary builders. We are gradually adding algorithms and testing their stability.
- I'm currently relying on Clark's description of the learner. Here are his notes:
- Implement the EM algorithm in order to deal with ambiguity (Chapter 5.5 in the second paper). This requires intrusive surgery, so I suggest talking to me before the cutting begins.
- Implement a method to deal with rare words (Chapter 5.6 in the second paper).