Skip to content

Latest commit

 

History

History
396 lines (396 loc) · 44.5 KB

File metadata and controls

396 lines (396 loc) · 44.5 KB
alpha3 language sentences dataset_percentage mean_len
1 eng English 1479733 15.42% 39.3277
2 rus Russian 849653 8.86% 33.4655
3 ita Italian 787053 8.20% 33.4897
4 tur Turkish 709573 7.40% 34.7355
5 epo Esperanto 659632 6.87% 40.3975
6 deu German 553727 5.77% 47.4774
7 kab Kabyle 553550 5.77% 30.1797
8 fra French 466192 4.86% 41.3866
9 por Portuguese 385737 4.02% 38.2929
10 ber Berber languages 385227 4.01% 32.4545
11 spa Spanish 338781 3.53% 38.8894
12 hun Hungarian 323048 3.37% 34.0299
13 jpn Japanese 208761 2.18% 18.2659
14 heb Hebrew 197226 2.06% 25.5678
15 ukr Ukrainian 171674 1.79% 27.8153
16 nld Dutch 144340 1.50% 34.7853
17 fin Finnish 128011 1.33% 35.7946
18 pol Polish 109662 1.14% 33.2333
19 mkd Macedonian 77938 0.81% 27.3793
20 cmn Mandarin Chinese 68325 0.71% 11.4165
21 mar Marathi 64126 0.67% 27.587
22 lit Lithuanian 59659 0.62% 30.1439
23 ces Czech 57030 0.59% 28.3683
24 dan Danish 49399 0.51% 33.7159
25 srp Serbian 45176 0.47% 30.4306
26 swe Swedish 41677 0.43% 30.1428
27 lat Latin 39718 0.41% 36.1448
28 ara Arabic 35991 0.38% 26.7817
29 ell Greek 34071 0.36% 30.3915
30 ina Interlingua 26680 0.28% 47.0018
31 ron Romanian 24943 0.26% 34.4097
32 bul Bulgarian 24503 0.26% 31.7201
33 pes Iranian Persian 23866 0.25% 34.7018
34 vie Vietnamese 19234 0.20% 38.7891
35 tlh Klingon 18764 0.20% 22.5339
36 nds Low German 17921 0.19% 33.1626
37 lfn Lingua Franca Nova 17223 0.18% 37.2197
38 fil Filipino 16649 0.17% 36.8098
39 jbo Lojban 15925 0.17% 30.7496
40 slk Slovak 14660 0.15% 25.7422
41 ind Indonesian 14542 0.15% 37.4785
42 hin Hindi 14230 0.15% 27.6058
43 nob Norwegian Bokmål 14223 0.15% 37.4732
44 tat Tatar 13674 0.14% 44.7585
45 bel Belarusian 12633 0.13% 37.3887
46 isl Icelandic 11091 0.12% 35.3886
47 kmr Northern Kurdish 8615 0.09% 25.6143
48 cat Catalan 7971 0.08% 37.334
49 uig Uyghur 7792 0.08% 31.1084
50 kor Korean 7570 0.08% 16.8085
51 ido Ido 7373 0.08% 29.0663
52 ile Interlingue 7352 0.08% 26.2331
53 bre Breton 7195 0.07% 27.4111
54 yid Yiddish 6895 0.07% 29.8983
55 tuk Turkmen 6729 0.07% 33.9633
56 dtp Central Dusun 6460 0.07% 37.3447
57 eus Basque 6166 0.06% 32.999
58 yue Cantonese 6134 0.06% 11.739
59 kat Georgian 5732 0.06% 26.0173
60 oci Occitan 5693 0.06% 32.3931
61 gos Gronings 5618 0.06% 24.0573
62 aze Azerbaijani 5348 0.06% 26.2691
63 run Rundi 5278 0.06% 23.107
64 hrv Croatian 5204 0.05% 30.058
65 ben Bangla 4714 0.05% 23.7809
66 glg Galician 4613 0.05% 35.8186
67 wuu Wu Chinese 4549 0.05% 12.0015
68 zsm Standard Malay 4336 0.05% 43.9765
69 mhr Eastern Mari 4300 0.04% 33.9802
70 vol Volapük 4132 0.04% 23.3129
71 avk Kotava 4101 0.04% 26.9268
72 afr Afrikaans 4031 0.04% 29.676
73 cor Cornish 3925 0.04% 21.5801
74 kaz Kazakh 3685 0.04% 49.9183
75 est Estonian 3637 0.04% 27.6646
76 tha Thai 3528 0.04% 20.5697
77 grn Guarani 2961 0.03% 28.4691
78 asm Assamese 2912 0.03% 24.1006
79 frr Northern Frisian 2855 0.03% 28.1916
80 mon Mongolian 2757 0.03% 30.9387
81 cbk Chavacano 2621 0.03% 30.0397
82 sqi Albanian 2526 0.03% 32.2743
83 ilo Iloko 2455 0.03% 34.4794
84 gle Irish 2389 0.02% 27.7765
85 arq Algerian Arabic 2306 0.02% 23.7493
86 hye Armenian 2248 0.02% 22.5934
87 gcf Guadeloupean Creole French 2085 0.02% 18.7789
88 war Waray 2025 0.02% 34.3274
89 urd Urdu 2008 0.02% 30.7495
90 lvs Standard Latvian 1961 0.02% 29.3544
91 swg Swabian 1937 0.02% 45.1187
92 ota Ottoman Turkish 1881 0.02% 55.4588
93 lzh Literary Chinese 1822 0.02% 16.2651
94 zza Zaza 1668 0.02% 21.8921
95 nno Norwegian Nynorsk 1576 0.02% 30.8249
96 chv Chuvash 1566 0.02% 25.6124
97 khm Khmer 1511 0.02% 24.2766
98 pam Pampanga 1482 0.02% 29.168
99 ceb Cebuano 1478 0.02% 32.1069
100 hsb Upper Sorbian 1423 0.01% 23.6339
101 kha Khasi 1372 0.01% 28.0758
102 pcd Picard 1346 0.01% 23.318
103 cym Welsh 1344 0.01% 29.3058
104 zgh Standard Moroccan Tamazight 1310 0.01% 28.2832
105 orv Old Russian 1307 0.01% 23.3841
106 grc Ancient Greek 1133 0.01% 32.8535
107 lad Ladino 1114 0.01% 22.8339
108 slv Slovenian 1093 0.01% 28.4282
109 ckb Central Kurdish 1089 0.01% 22.5794
110 gla Scottish Gaelic 1033 0.01% 33.5334
111 dsb Lower Sorbian 991 0.01% 26.7508
112 sah Sakha 943 0.01% 28.315
113 csb Kashubian 937 0.01% 30.3095
114 sdh Southern Kurdish 923 0.01% 20.428
115 bua Buriat 905 0.01% 21.8807
116 xal Kalmyk 870 0.01% 29.3621
117 hrx Hunsrik 857 0.01% 26.2754
118 uzb Uzbek 855 0.01% 25.9813
119 prg Prussian 838 0.01% 38.8043
120 mal Malayalam 827 0.01% 36.8222
121 pms Piedmontese 823 0.01% 34.9721
122 ltz Luxembourgish 805 0.01% 25.4099
123 arz Egyptian Arabic 798 0.01% 20.2331
124 rom Romany 794 0.01% 29.597
125 nst Tase Naga 793 0.01% 33.9281
126 lij Ligurian 653 0.01% 24.6508
127 hoc Ho 651 0.01% 19.7512
128 jav Javanese 615 0.01% 28.7333
129 swh Swahili (individual language) 589 0.01% 23.1613
130 thv Tahaggart Tamahaq 585 0.01% 20.7692
131 bos Bosnian 567 0.01% 23.3492
132 tpw Tupí 561 0.01% 22.2602
133 zlm Malay (individual language) 543 0.01% 37.0737
134 ckt Chukot 531 0.01% 70.4708
135 mya Burmese 433 0.00% 33.5219
136 max North Moluccan Malay 427 0.00% 30.63
137 que Quechua 422 0.00% 25.6588
138 nov Novial 410 0.00% 23.678
139 fkv Kven Finnish 408 0.00% 31.951
140 fao Faroese 402 0.00% 28.6119
141 mri Maori 388 0.00% 39.1186
142 gsw Swiss German 381 0.00% 21.5459
143 ori Odia 374 0.00% 12.7567
144 fry Western Frisian 355 0.00% 28.569
145 cha Chamorro 343 0.00% 29.1312
146 tam Tamil 334 0.00% 35.2784
147 tzl Talossan 330 0.00% 18.8576
148 ang Old English 328 0.00% 32.9634
149 crh Crimean Turkish 308 0.00% 35.7792
150 mus Muscogee 292 0.00% 30.3664
151 ast Asturian 280 0.00% 27.4429
152 got Gothic 269 0.00% 21.2825
153 kir Kyrgyz 254 0.00% 23.8504
154 tel Telugu 254 0.00% 28.0157
155 awa Awadhi 253 0.00% 17.5099
156 xho Xhosa 252 0.00% 28.369
157 shy Tachawit 247 0.00% 29.7166
158 bzt Brithenig 243 0.00% 19.8313
159 oss Ossetic 236 0.00% 24.5297
160 rif Riffian 226 0.00% 21.4735
161 lao Lao 219 0.00% 27.5388
162 bak Bashkir 215 0.00% 38.6512
163 nah Nahuatl languages 212 0.00% 40.1274
164 amh Amharic 211 0.00% 12.7867
165 mlt Maltese 208 0.00% 25.9952
166 bar Bavarian 200 0.00% 35.065
167 pan Punjabi 196 0.00% 32.8622
168 vec Venetian 190 0.00% 31.5947
169 ain Ainu 187 0.00% 22.0481
170 krl Karelian 185 0.00% 24.2541
171 sme Northern Sami 181 0.00% 20.0884
172 npi Nepali (individual language) 176 0.00% 18.108
173 kan Kannada 176 0.00% 35.3636
174 guj Gujarati 168 0.00% 24.244
175 sux Sumerian 160 0.00% 11.6438
176 san Sanskrit 156 0.00% 24.0962
177 haw Hawaiian 155 0.00% 23.5226
178 krc Karachay-Balkar 148 0.00% 30.1824
179 egl Emilian 144 0.00% 19.3125
180 nog Nogai 144 0.00% 23.3194
181 afb Gulf Arabic 137 0.00% 19.7591
182 qya Quenya 136 0.00% 21.3529
183 ldn Láadan 135 0.00% 24.4148
184 enm Middle English 130 0.00% 39.4692
185 min Minangkabau 121 0.00% 23.0826
186 kum Kumyk 116 0.00% 34.4655
187 evn Evenki 116 0.00% 24.0259
188 rue Rusyn 116 0.00% 16.9052
189 wol Wolof 114 0.00% 20.193
190 bal Baluchi 105 0.00% 25.419
191 tir Tigrinya 103 0.00% 19
192 arg Aragonese 103 0.00% 15.9515
193 cho Choctaw 98 0.00% 23.2551
194 stq Saterland Frisian 96 0.00% 23.7708
195 shs Shuswap 95 0.00% 23.0526
196 fro Old French 93 0.00% 45.8602
197 tpi Tok Pisin 89 0.00% 22.5169
198 pap Papiamento 89 0.00% 18.1461
199 apc North Levantine Arabic 88 0.00% 16.0114
200 sjn Sindarin 86 0.00% 19.593
201 nav Navajo 85 0.00% 27.0706
202 lld Ladin 85 0.00% 38.9882
203 cmo Central Mnong 85 0.00% 10.2235
204 lin Lingala 85 0.00% 29.1176
205 mrj Western Mari 83 0.00% 35.5181
206 sco Scots 82 0.00% 37.2561
207 ksh Colognian 82 0.00% 54.6707
208 mww Hmong Daw 82 0.00% 25.7073
209 som Somali 80 0.00% 30.025
210 afh Afrihili 79 0.00% 16.5443
211 zul Zulu 77 0.00% 30.4156
212 smo Samoan 76 0.00% 21.4342
213 bvy Baybayanon 76 0.00% 28.5263
214 kal Kalaallisut 76 0.00% 16.3947
215 moh Mohawk 74 0.00% 31.2027
216 nus Nuer 68 0.00% 15.1912
217 udm Udmurt 68 0.00% 27.3676
218 ext Extremaduran 66 0.00% 17.7424
219 tly Talysh 66 0.00% 15.5455
220 hat Haitian Creole 64 0.00% 22.4688
221 tmr Jewish Babylonian Aramaic (ca. 200-1200 CE) 64 0.00% 28.5312
222 tgk Tajik 63 0.00% 31.4603
223 mgm Mambae 60 0.00% 32.5667
224 hau Hausa 60 0.00% 31.4833
225 sma Southern Sami 59 0.00% 19.9492
226 ppl Pipil 59 0.00% 30.7627
227 lut Lushootseed 59 0.00% 19.8983
228 mlg Malagasy 59 0.00% 21.0339
229 tet Tetum 59 0.00% 36.3051
230 xmf Mingrelian 58 0.00% 24.2069
231 iii Sichuan Yi 58 0.00% 7.08621
232 hbo Ancient Hebrew 57 0.00% 86.5965
233 bfz Mahasu Pahari 57 0.00% 12.8772
234 pdc Pennsylvania German 57 0.00% 30.0702
235 dws Dutton World Speedwords 57 0.00% 10.1579
236 tts Northeastern Thai 55 0.00% 11.6364
237 pag Pangasinan 55 0.00% 27.2545
238 bho Bhojpuri 55 0.00% 27.0727
239 hak Hakka Chinese 53 0.00% 9.28302
240 chn Chinook Jargon 53 0.00% 31.0189
241 wln Walloon 53 0.00% 28.2642
242 ary Moroccan Arabic 52 0.00% 13.9808
243 ike Eastern Canadian Inuktitut 50 0.00% 15.56
244 kas Kashmiri 49 0.00% 14.7347
245 cjy Jinyu Chinese 49 0.00% 11.3673
246 acm Mesopotamian Arabic 49 0.00% 23.9796
247 sgs Samogitian 47 0.00% 11.6383
248 iba Iban 46 0.00% 47.913
249 sin Sinhala 45 0.00% 22.0222
250 kpv Komi-Zyrian 45 0.00% 13.5333
251 hil Hiligaynon 44 0.00% 30.5455
252 pus Pashto 44 0.00% 35.9318
253 koi Komi-Permyak 43 0.00% 15.1395
254 ngu Guerrero Nahuatl 43 0.00% 32.4884
255 sna Shona 42 0.00% 20.5952
256 tsn Tswana 41 0.00% 35.0732
257 bod Tibetan 41 0.00% 33.1951
258 nch Central Huasteca Nahuatl 39 0.00% 23
259 kjh Khakas 38 0.00% 23.5
260 fij Fijian 38 0.00% 13.0263
261 yor Yoruba 37 0.00% 23.2973
262 gbm Garhwali 36 0.00% 15
263 fuv Nigerian Fulfulde 36 0.00% 16.1389
264 drt Drents 35 0.00% 23.9714
265 scn Sicilian 35 0.00% 35.3429
266 pnb Western Panjabi 35 0.00% 24.6857
267 hif Fiji Hindi 34 0.00% 28.0588
268 pau Palauan 34 0.00% 15.9706
269 glv Manx 33 0.00% 19.5455
270 lim Limburgish 33 0.00% 19.303
271 rap Rapanui 33 0.00% 15.6364
272 liv Livonian 33 0.00% 20.0303
273 niu Niuean 32 0.00% 17.8438
274 ibo Igbo 32 0.00% 20.0938
275 frm Middle French 32 0.00% 58.3125
276 dng Dungan 31 0.00% 14.6129
277 fur Friulian 31 0.00% 23.2258
278 ady Adyghe 31 0.00% 18.5806
279 sun Sundanese 31 0.00% 25.7742
280 umb Umbundu 30 0.00% 17.9667
281 tah Tahitian 30 0.00% 14.5
282 bjn Banjar 29 0.00% 22.4483
283 lmo Lombard 29 0.00% 20.9655
284 che Chechen 28 0.00% 26.6429
285 akl Aklanon 28 0.00% 13.9643
286 aln Gheg Albanian 28 0.00% 18.1786
287 chr Cherokee 28 0.00% 11.9643
288 kin Kinyarwanda 28 0.00% 23.3571
289 mvv Tagal Murut 28 0.00% 31.4286
290 bis Bislama 28 0.00% 17.3929
291 abk Abkhazian 27 0.00% 19.2222
292 mah Marshallese 27 0.00% 23.4815
293 ewe Ewe 26 0.00% 28.6538
294 div Divehi 26 0.00% 45.7692
295 myv Erzya 26 0.00% 19
296 jam Jamaican Creole English 26 0.00% 13.1538
297 ood Tohono O'odham 26 0.00% 22
298 kek Kekchí 24 0.00% 14.875
299 cos Corsican 24 0.00% 20.1667
300 aoz Uab Meto 24 0.00% 10.2917
301 nya Nyanja 24 0.00% 23.375
302 ltg Latgalian 23 0.00% 19.5652
303 mwl Mirandese 22 0.00% 28.9091
304 osp Old Spanish 22 0.00% 30.7727
305 lkt Lakota 22 0.00% 19.1364
306 izh Ingrian 21 0.00% 22.3333
307 oar Old Aramaic (up to 700 BCE) 21 0.00% 69.8571
308 klj Khalaj 20 0.00% 19.85
309 ton Tongan 20 0.00% 24.65
310 roh Romansh 20 0.00% 56.55
311 cay Cayuga 19 0.00% 13.3158
312 ngt Kriang 19 0.00% 20.3684
313 new Newari 17 0.00% 16.7647
314 tyv Tuvinian 17 0.00% 20
315 guc Wayuu 16 0.00% 12.125
316 vro Võro 15 0.00% 35.7333
317 phn Phoenician 15 0.00% 17.7333
318 toi Tonga (Zambia) 15 0.00% 19.7333
319 srd Sardinian 15 0.00% 31.0667
320 tvl Tuvalu 14 0.00% 17.2857
321 qxq Qashqa'i 14 0.00% 23.5
322 non Old Norse 14 0.00% 56.4286
323 gil Gilbertese 14 0.00% 22
324 gag Gagauz 14 0.00% 19.4286
325 nlv Orizaba Nahuatl 14 0.00% 55.4286
326 vep Veps 13 0.00% 33.9231
327 bam Bambara 13 0.00% 9.61538
328 ofs Old Frisian 13 0.00% 42.2308
329 emx Erromintxela 12 0.00% 24.6667
330 tkl Tokelau 12 0.00% 19.1667
331 kaa Kara-Kalpak 11 0.00% 20.4545
332 brx Bodo 11 0.00% 18.5455
333 nau Nauru 11 0.00% 9.81818
334 lzz Laz 11 0.00% 18.7273
335 zea Zeelandic 10 0.00% 22.2
336 kam Kamba 10 0.00% 22.9
337 yua Yucateco 10 0.00% 14.1
338 laa Southern Subanen 10 0.00% 76.1
339 xqa Karakhanid 10 0.00% 45.1
340 jdt Judeo-Tat 9 0.00% 18.1111
341 mdf Moksha 9 0.00% 28.3333
342 mnw Mon 9 0.00% 30.8889
343 sag Sango 8 0.00% 14.5
344 mai Maithili 8 0.00% 9.75
345 mic Mi'kmaq 8 0.00% 10.5
346 chg Chagatai 8 0.00% 47.625
347 otk Old Turkish 7 0.00% 14.1429
348 quc Kʼicheʼ 7 0.00% 12.8571
349 fuc Pulaar 7 0.00% 13.5714
350 mad Madurese 7 0.00% 22
351 cpi Chinese Pidgin English 6 0.00% 31.5
352 kiu Kirmanjki 6 0.00% 16
353 diq Dimli (individual language) 6 0.00% 15.5
354 aii Assyrian Neo-Aramaic 6 0.00% 25
355 mfe Morisyen 6 0.00% 16.1667
356 snd Sindhi 6 0.00% 22.3333
357 jpa Jewish Palestinian Aramaic 6 0.00% 48.1667
358 ajp South Levantine Arabic 5 0.00% 9.2
359 bcl Central Bikol 5 0.00% 21.8
360 pal Pahlavi 5 0.00% 44.8
361 tmw Temuan 5 0.00% 14.8
362 hsn Xiang Chinese 4 0.00% 6.5
363 crk Plains Cree 4 0.00% 6.25
364 hnj Hmong Njua 4 0.00% 22.25
365 gom Goan Konkani 4 0.00% 37.5
366 kxi Keningau Murut 3 0.00% 11.3333
367 syc Classical Syriac 3 0.00% 40.3333
368 hdn Northern Haida 3 0.00% 26.3333
369 ban Balinese 3 0.00% 18.6667
370 gan Gan Chinese 3 0.00% 8.66667
371 ayl Libyan Arabic 3 0.00% 8.33333
372 pfl Palatine German 3 0.00% 39.3333
373 gaa Ga 3 0.00% 20
374 alt Southern Altai 3 0.00% 15.3333
375 tig Tigre 3 0.00% 9
376 mni Manipuri 3 0.00% 13.6667
377 tso Tsonga 3 0.00% 17
378 aym Aymara 2 0.00% 33
379 sot Southern Sotho 2 0.00% 66.5
380 srn Sranan Tongo 2 0.00% 10.5
381 mnc Manchu 2 0.00% 28
382 pli Pali 2 0.00% 86.5
383 osx Old Saxon 2 0.00% 55.5
384 lug Ganda 2 0.00% 23.5
385 crs Seselwa Creole French 2 0.00% 121
386 ssw Swati 1 0.00% 55
387 hax Southern Haida 1 0.00% 23
388 ryu Central Okinawan 1 0.00% 17
389 lou Louisiana Creole 1 0.00% 24
390 cyo Cuyonon 1 0.00% 35
391 urh Urhobo 1 0.00% 15
392 rel Rendille 1 0.00% 13
393 oji Ojibwa 1 0.00% 21
394 nys Nyungar 1 0.00% 5