GitHub - grantspink/LanguageIdentifier: Coded in 2014 - Java program that is able to detect what language a piece of text is written in

grantspink / LanguageIdentifier Public

Notifications You must be signed in to change notification settings
Fork 0
Star 1

Coded in 2014 - Java program that is able to detect what language a piece of text is written in

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 2 Commits
nbproject		nbproject
src		src
Languages.txt		Languages.txt
README.txt		README.txt
Testcase		Testcase
ace.txt		ace.txt
ace_graphs.txt		ace_graphs.txt
af.txt		af.txt
af_graphs.txt		af_graphs.txt
als.txt		als.txt
als_graphs.txt		als_graphs.txt
am.txt		am.txt
am_graphs.txt		am_graphs.txt
an.txt		an.txt
an_graphs.txt		an_graphs.txt
ang.txt		ang.txt
ang_graphs.txt		ang_graphs.txt
ar.txt		ar.txt
ar_graphs.txt		ar_graphs.txt
arc.txt		arc.txt
arc_graphs.txt		arc_graphs.txt
arz.txt		arz.txt
arz_graphs.txt		arz_graphs.txt
as.txt		as.txt
as_graphs.txt		as_graphs.txt
ast.txt		ast.txt
ast_graphs.txt		ast_graphs.txt
av.txt		av.txt
av_graphs.txt		av_graphs.txt
ay.txt		ay.txt
ay_graphs.txt		ay_graphs.txt
az.txt		az.txt
az_graphs.txt		az_graphs.txt
ba.txt		ba.txt
ba_graphs.txt		ba_graphs.txt
bar.txt		bar.txt
bar_graphs.txt		bar_graphs.txt
bat-smg.txt		bat-smg.txt
bat-smg_graphs.txt		bat-smg_graphs.txt
bcl.txt		bcl.txt
bcl_graphs.txt		bcl_graphs.txt
be.txt		be.txt
be_graphs.txt		be_graphs.txt
bg.txt		bg.txt
bg_graphs.txt		bg_graphs.txt
bh.txt		bh.txt
bh_graphs.txt		bh_graphs.txt
bjn.txt		bjn.txt
bjn_graphs.txt		bjn_graphs.txt
bn.txt		bn.txt
bn_graphs.txt		bn_graphs.txt
bo.txt		bo.txt
bo_graphs.txt		bo_graphs.txt
bpy.txt		bpy.txt
bpy_graphs.txt		bpy_graphs.txt
br.txt		br.txt
br_graphs.txt		br_graphs.txt
bs.txt		bs.txt
bs_graphs.txt		bs_graphs.txt
bug.txt		bug.txt
bug_graphs.txt		bug_graphs.txt
build.xml		build.xml
ca.txt		ca.txt
ca_graphs.txt		ca_graphs.txt
cbk-zam.txt		cbk-zam.txt
cbk-zam_graphs.txt		cbk-zam_graphs.txt
ce.txt		ce.txt
ce_graphs.txt		ce_graphs.txt
ceb.txt		ceb.txt
ceb_graphs.txt		ceb_graphs.txt
ckb.txt		ckb.txt
ckb_graphs.txt		ckb_graphs.txt
co.txt		co.txt
co_graphs.txt		co_graphs.txt
crh.txt		crh.txt
crh_graphs.txt		crh_graphs.txt
cs.txt		cs.txt
cs_graphs.txt		cs_graphs.txt
csb.txt		csb.txt
csb_graphs.txt		csb_graphs.txt
cv.txt		cv.txt
cv_graphs.txt		cv_graphs.txt
cy.txt		cy.txt
cy_graphs.txt		cy_graphs.txt
da.txt		da.txt
da_graphs.txt		da_graphs.txt
de.txt		de.txt
de_graphs.txt		de_graphs.txt
diq.txt		diq.txt
diq_graphs.txt		diq_graphs.txt
dsb.txt		dsb.txt
dsb_graphs.txt		dsb_graphs.txt
dv.txt		dv.txt
dv_graphs.txt		dv_graphs.txt
el.txt		el.txt
el_graphs.txt		el_graphs.txt
eml.txt		eml.txt
eml_graphs.txt		eml_graphs.txt
en.txt		en.txt
en_graphs.txt		en_graphs.txt

Repository files navigation

-------------------------------
--Language Identifier README--
-------------------------------

--------General Overview-------
Detects what language a piece of text is written in.
Can detect the languages of multiple text files at once.

----------How it Works---------
1. Constructs a database of language samples from Wikipedia articles.
2. Accepts text file(s) for detection.
3. Returns the most probable language for each text file given the language samples used for comparison.

Language analysis utilizes the monographs, digraphs, and trigraphs that appear in the text files and language samples.

-------Modes of Operation-------
1. Comparing one text file to multiple language samples.
The language sample which most closely matches the text file will be returned.

2. Comparing multiple text files to a single language sample.
The text file which most closely matches the language sample will be returned.

----Understanding the Output----
The numbers printed out represent the relative probabilities of a language match, calculated using either a monograph, digraph, or trigraph analysis.
The final language match probability is then calculated using a 10-20-70 weighting of these probabilities.

- A monograph is a single letter that may appear in a word.
- A digraph is a two-letter consecutive string inside a word.
- A trigraph is a three-letter consecutive string inside a word.
Ex: the word "probability has:
Monographs: [p,r,o,b,a,b,i,l,i,t,y]
Digraphs: [pr,ro,ob,ba,ab,bi,il,li,it,ty]
Trigraphs: [pro,rob,oba,bab,abi,bil,ili,lit,ity]

The analysis consists of finding the frequency of the graphemes in the language(s), and the frequency of the graphemes in the input text(s).
Using these, we calculate the probability of a random assortment of graphemes from the language giving exactly the assortment of graphemes in the input text.
(Think monkeys on a typewriter writing Hamlet).
Finally, using these absolute probabilities, we calculate the relative probabilities of the users' selections by dividing by an appropriate factor.
Most of the time, the relative probabilities work out such that one language is 100% likely to be a match, as compared to any other language.