Accuracy comparision between lingua and fasttext

[vanga]

I just wanted to share these considering the comparisions mentioned in the readme.

• I see that fasttext seems to be better overall for my usecase, though fasttext may just be biased towards english and hence may seem better given that my data is biased towards english.
• Lingua seems to be better at correctly recognizing non-english texts I think while it classifies even english phrases excessively as some other language.
• Lingua is ran with the default high accuracy model, with all languages loaded.

+--------------------------------------------------------------------------------+----------+--------------------+
|position                                                                        |lingua_ln |fasttext_ln         |
+--------------------------------------------------------------------------------+----------+--------------------+
|Gestor comercial                                                                |PORTUGUESE|Catalan             |
|Endodontist                                                                     |LATIN     |English             |
|مراجع داخلى                                                                     |ARABIC    |NULL                |
|Account Manager                                                                 |ENGLISH   |English             |
|Bank Teller                                                                     |BOKMAL    |English             |
|Associate                                                                       |ENGLISH   |English             |
|كاشير                                                                           |ARABIC    |Arabic              |
|إداري                                                                           |ARABIC    |Arabic              |
|Assistant                                                                       |FRENCH    |English             |
|مهندس مدني                                                                      |ARABIC    |Arabic              |
|安全监督员                                                                      |CHINESE   |Chinese             |
|teacher                                                                         |ENGLISH   |English             |
|ceo                                                                             |IRISH     |German              |
|Assistant                                                                       |FRENCH    |English             |
|group produt Manager                                                            |ENGLISH   |Portuguese          |
|AC TECHINICAN                                                                   |LATIN     |German              |
|Vice President                                                                  |ENGLISH   |English             |
|Chairperson                                                                     |ENGLISH   |English             |
|Finance Officer                                                                 |ENGLISH   |English             |
|Kitchen Manager                                                                 |ENGLISH   |English             |
|IT Engineer                                                                     |ENGLISH   |English             |
|Senior Account Manager                                                          |ENGLISH   |English             |
|Registry Officer                                                                |ENGLISH   |English             |
|Sales Business Development                                                      |ENGLISH   |English             |
|Sales Representative                                                            |ENGLISH   |English             |
|Sales Representative                                                            |ENGLISH   |English             |
|Store keeper & warehouse incharge                                               |ENGLISH   |English             |
|Nondestructive Testing Technician                                               |ENGLISH   |English             |
|Oracle Application Specialist                                                   |ENGLISH   |English             |
|Sales Coordinator                                                               |LATIN     |English             |
|Store Keeper                                                                    |BOKMAL    |English             |
|Senior Business Analyst                                                         |ENGLISH   |English             |
|Facilities Services                                                             |ENGLISH   |English             |
|Regional Manager- MENA                                                          |ENGLISH   |English             |
|Local purcahser                                                                 |ENGLISH   |English             |
|Sourcing Specialist                                                             |ENGLISH   |English             |
|مهندس معماري                                                                    |ARABIC    |Arabic              |
|Senior Corporate Travel Consultant                                              |ENGLISH   |English             |
|Accounts and Admin                                                              |ENGLISH   |English             |
|Accountant                                                                      |ENGLISH   |English             |
|Senior Accountant                                                               |ENGLISH   |English             |
|مشرف عمليات اول                                                                 |ARABIC    |Arabic              |
|manajer                                                                         |INDONESIAN|Indonesian          |
|Commie                                                                          |DUTCH     |English             |
|Sales Manager                                                                   |ENGLISH   |English             |
|Pharmacist                                                                      |ESPERANTO |English             |
|mohamd                                                                          |MALAY     |French              |
|Customer Care Executive                                                         |ENGLISH   |English             |
|Technical Engineer                                                              |ENGLISH   |English             |
|Sales Representative                                                            |ENGLISH   |English             |
|Chief Accountant                                                                |ENGLISH   |English             |
|Channel Sales Manager                                                           |ENGLISH   |English             |
|Contracts Manager                                                               |ENGLISH   |English             |
|Accountant                                                                      |ENGLISH   |English             |
|Project Manager                                                                 |ENGLISH   |English             |
|Material Planner                                                                |ENGLISH   |English             |
|Senior Immigration Officer ||  Canadian permanent Residency Visa Specialist 🇨🇦|ENGLISH   |English             |
|Accountant                                                                      |ENGLISH   |English             |
|Administrative                                                                  |ALBANIAN  |English             |
|printer                                                                         |ROMANIAN  |English             |
|Business Development Manager                                                    |ENGLISH   |English             |
|Head of Sales                                                                   |ENGLISH   |English             |
|Officer – Advertising & Marketing                                               |ENGLISH   |English             |
|Inventory Control Coordinator                                                   |ENGLISH   |English             |
|*******                                                                         |NULL      |Modern Greek (1453-)|
|Product Manager (Digital Platform)                                              |ENGLISH   |English             |
|عامل                                                                            |ARABIC    |Arabic              |
|Store Manager                                                                   |BOKMAL    |English             |
|Administration Assistant                                                        |FRENCH    |English             |
|Management                                                                      |ENGLISH   |English             |
|plant ,Transport and Machinary Manager                                          |ENGLISH   |English             |
|iO Relations Coordinator                                                        |ENGLISH   |English             |
|Team Leader                                                                     |ENGLISH   |English             |
|Supervisor                                                                      |ENGLISH   |English             |
|Sales Operations Specialist                                                     |ENGLISH   |English             |
|Senior Arabic Editor, Copywriter, Translator                                    |ENGLISH   |English             |
|Commercial Diver                                                                |FRENCH    |English             |
|Project Coordinator                                                             |ENGLISH   |English             |
|Managing Partner                                                                |ENGLISH   |English             |
|Customs                                                                         |ENGLISH   |English             |
|Yardi Functional Consultant                                                     |LATIN     |English             |
|Salesperson                                                                     |ENGLISH   |English             |
|Head Waiter                                                                     |ENGLISH   |English             |
|MEP Engineer                                                                    |ENGLISH   |English             |
|Lead Manager - Global Brand Management                                          |ENGLISH   |English             |
|Medical Coder Course                                                            |ENGLISH   |English             |
|Store Supervisor                                                                |ENGLISH   |English             |
|HEAD  OF LEGAL DEP.                                                             |ENGLISH   |English             |
|planner                                                                         |ENGLISH   |English             |
|Intern                                                                          |GERMAN    |English             |
|Supervisor                                                                      |ENGLISH   |English             |
|Data Scientist                                                                  |ESPERANTO |English             |
|Lead Generation                                                                 |ENGLISH   |English             |
|Forklift Operator                                                               |ENGLISH   |English             |
|Public Relations Officer                                                        |ENGLISH   |English             |
|Production Line Incharge                                                        |FRENCH    |English             |
|Interior Decorator                                                              |LATIN     |English             |
|key accounts manager                                                            |ENGLISH   |English             |
|IT Project Manager                                                              |ENGLISH   |English             |
|Anesthesia Technician                                                           |LATIN     |English             |
+--------------------------------------------------------------------------------+----------+--------------------+

[x86Gr]

Interesting. Can you run that again with LanguageDetectorBuilder.from_all_spoken_languages() ?

[vanga]

tried for a few, it is giving the same response as from_all_languages()

[x86Gr]

For me it is improving the accuracy a bit, but i'm using short sentences.