-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
North sámi testdata in SMJ #103
Comments
ok, so the words are:
this is the distribution in lexc files: $ for w in $(cat tools/spellcheckers/test/fstbased/desktop/hfst/rejected_lemmas.txt | sed -e 's/^"//' -e 's/" is NOT.*//') ; do egrep "^$w\+" src/fst/morphology/stems/*; done
src/fst/morphology/stems/nouns.lexc:áhpeguollebivdár+N+Err/Der+CmpN/SgN+CmpN/SgG+CmpN/PlG+Sem/Hum:áhpe#guolle#bivdár GAHPER ;
src/fst/morphology/stems/nouns.lexc:álggididdje+N+Err/Der+CmpN/SgN+CmpN/SgG+CmpN/PlG+CmpN/SgNomLeft+CmpN/SgGenLeft+CmpN/PlGenLeft+Sem/Hum:álggididdje ACTOR ; !No verb "álggidit", so álggididdje isn't possible
src/fst/morphology/stems/adjectives.lexc:almasjlasj+A+Err/Der+CmpN/SgN+CmpN/PlG:almasjl DÁRBULASJ ;
src/fst/morphology/stems/smj-propernouns.lexc:Áváhårsa+Use/-Spell:Ává^hårsa MARJA-plc ; !
src/fst/morphology/stems/smj-abbreviations.lexc:avtl.+N:avtalåhko ab-dot-noun-itrab ;
src/fst/morphology/stems/smj-abbreviations.lexc:bba+N:bårråmbassti ab-dot-noun-itrab ;!bårråmbassti
src/fst/morphology/stems/adjectives.lexc:boajto+A:boajto VINJO- ;
src/fst/morphology/stems/smj-abbreviations.lexc:buojk+Adv:buojk ab-dot-adv-trab ; ! buojkulvis/vissan
src/fst/morphology/stems/smj-propernouns.lexc:Četčenia+Use/-Spell+OLang/SME:Četčenia ACCRA-plc ;
src/fst/morphology/stems/nouns.lexc:dárrolasj+N+Err/Der+CmpN/SgN+CmpN/SgG+CmpN/PlG+Sem/Hum:dárrol BERULASJ ; !should be dárulasj? It's more ok with dárrolasj than dárrulasj
src/fst/morphology/stems/smj-abbreviations.lexc:do+N:do ab-dot-adv-itrab ; !hæ?
src/fst/morphology/stems/smj-abbreviations.lexc:dub+Adv:dub ab-dot-adv-itrab ; !hæ?
src/fst/morphology/stems/smj-abbreviations.lexc:dus+Adv:dus ab-dot-adv-itrab ; !hæ?
src/fst/morphology/stems/adjectives.lexc:ednamlasj+A+Err/Der+CmpN/SgN+CmpN/PlG:ednamladtj ÅLLAGASJ ;
src/fst/morphology/stems/adjectives.lexc:ednamlasj+A+Err/Der:ednamladtj ÅLLAGASJ ; !ulikestavleses subtsantiv får ikke -lasj derivasjon
src/fst/morphology/stems/nouns.lexc:fárrolasj+N+Err/Der+CmpN/SgN+CmpN/SgG+CmpN/PlG+Sem/Hum:fárrol BERULASJ ; !should be fárulasj, but more ok with fárrolasj than fárrulasj
src/fst/morphology/stems/adjectives.lexc:fylkkasuohkanlasj+A+Err/Der:fylkka#suohkanl METÅVDÅLASJ;
src/fst/morphology/stems/pronouns.lexc:færtguhti+Pron+Indef:færtge%> guhtikobl ;
src/fst/morphology/stems/pronouns.lexc:færtguhti+Pron+Indef+Sg+Nom+Foc/Pos-k:fært#guhti%>k # ;
src/fst/morphology/stems/pronouns.lexc:færtguhti+Pron+Indef+Sg+Nom+Foc/Neg-k:fært#guhti%>k # ;
src/fst/morphology/stems/pronouns.lexc:færtguhti+Pron+Indef+Sg+Ine+Foc/Pos-k:fært#gænºna%>k # ;
src/fst/morphology/stems/pronouns.lexc:færtguhti+Pron+Indef+Sg+Ine+Foc/Neg-k:fært#gænºna%>k # ;
src/fst/morphology/stems/pronouns.lexc:færtguhti+Pron+Indef+Sg+Ine+Foc/Pos-k+Use/NG:fært#gænºna%>nik # ; !
src/fst/morphology/stems/pronouns.lexc:færtguhti+Pron+Indef+Sg+Ine+Foc/Neg-k+Use/NG:fært#gænºna%>nik # ; !
src/fst/morphology/stems/pronouns.lexc:færtguhti+Pron+Indef+Sg+Ela+Foc/Pos-k:fært#gæssta%>k # ;
src/fst/morphology/stems/pronouns.lexc:færtguhti+Pron+Indef+Sg+Ela+Foc/Neg-k:fært#gæssta%>k # ;
src/fst/morphology/stems/pronouns.lexc:færtguhti+Pron+Indef+Sg+Ela+Foc/Pos-k+Use/NG:fært#gæssta%>stik # ; !
src/fst/morphology/stems/pronouns.lexc:færtguhti+Pron+Indef+Sg+Ela+Foc/Neg-k+Use/NG:fært#gæssta%>stik # ; !
src/fst/morphology/stems/pronouns.lexc:færtguhti+Pron+Indef+Pl+Nom+Foc/Pos-k:fært#gudi%>k # ;
src/fst/morphology/stems/pronouns.lexc:færtguhti+Pron+Indef+Pl+Nom+Foc/Neg-k:fært#gudi%>k # ;
src/fst/morphology/stems/pronouns.lexc:gájkkasasj+Pron+Indef+Err/Orth:gájkkasa juohkkahasjcase ; !
src/fst/morphology/stems/numerals.lexc:gáktse+Err/Orth+Use/-Spell+Use/Marg+Use/NG:gáktse# NLX ; !Err/Orth?
src/fst/morphology/stems/adjectives.lexc:gávo+A:gávo VINJO- ;
src/fst/morphology/stems/nouns.lexc:guojmme+N+CmpN/SgN+CmpN/SgG+CmpN/PlG+Sem/Hum:guojmme MUORRA ; !
src/fst/morphology/stems/nouns.lexc:guojmme+N+CmpN/SgN+CmpN/SgG+CmpN/PlG+Sem/Hum:guojmme MUORRA ; !
src/fst/morphology/stems/numerals.lexc:guoktajuodevidálågåk+Num:guok#tjuode#vidá#lågåg9 VUOSTASJ ;
src/fst/morphology/stems/numerals.lexc:guoktajuohtevidálågåk+Num:guok#tjuohte#vidá#lågåg9 VUOSTASJ ;
src/fst/morphology/stems/nouns.lexc:guollebivdár+N+Err/Der+CmpN/SgN+CmpN/SgG+CmpN/PlG+Sem/Hum:guolle#bivdár GAHPER ; ! used derivation for contraced verb, when verb is even, bivddet-bivdde
src/fst/morphology/stems/adjectives.lexc:gånågislasj+A+Err/Der:gånågisladtj ÅLLAGASJ ;
src/fst/morphology/stems/adjectives.lexc:háldaduslasj+A+Err/Der:háldadusladtj ÅLLAGASJ ; !feil
src/fst/morphology/stems/smj-propernouns.lexc:Helmuk+Use/-Spell:Helmug9 LONDON-plc ; !
src/fst/morphology/stems/smj-abbreviations.lexc:huom+Adv:huom ab-dot-adv-trab ; !huomaha
src/fst/morphology/stems/pronouns.lexc:iesjguhti+Pron+Indef:iesj#ge%> guhtikobl ;
src/fst/morphology/stems/pronouns.lexc:iesjguhti+Pron+Indef+Pl+Nom+Foc/Neg-k:iesj#gudi%>k # ;
src/fst/morphology/stems/pronouns.lexc:iesjguhti+Pron+Indef+Pl+Nom+Foc/Pos-k:iesj#gudi%>k # ;
src/fst/morphology/stems/pronouns.lexc:iesjguhti+Pron+Indef+Sg+Ela+Foc/Neg-k+Use/NG:iesj#gæstá%>stik # ; !
src/fst/morphology/stems/pronouns.lexc:iesjguhti+Pron+Indef+Sg+Ela+Foc/Pos-k+Use/NG:iesj#gæstá%>stik # ; !
src/fst/morphology/stems/pronouns.lexc:iesjguhti+Pron+Indef+Sg+Ela+Foc/Neg-k:iesj#gæssta%>k # ;
src/fst/morphology/stems/pronouns.lexc:iesjguhti+Pron+Indef+Sg+Ela+Foc/Pos-k:iesj#gæssta%>k # ;
src/fst/morphology/stems/pronouns.lexc:iesjguhti+Pron+Indef+Sg+Ine+Foc/Neg-k+Use/NG:iesj#gænºna%>nik # ; !
src/fst/morphology/stems/pronouns.lexc:iesjguhti+Pron+Indef+Sg+Ine+Foc/Pos-k+Use/NG:iesj#gænºna%>nik # ; !
src/fst/morphology/stems/pronouns.lexc:iesjguhti+Pron+Indef+Sg+Ine+Foc/Neg-k:iesj#gænºna%>k # ;
src/fst/morphology/stems/pronouns.lexc:iesjguhti+Pron+Indef+Sg+Ine+Foc/Pos-k:iesj#gænºna%>k # ;
src/fst/morphology/stems/pronouns.lexc:iesjguhti+Pron+Indef+Sg+Nom+Foc/Neg-k:iesj#guhti%>k # ;
src/fst/morphology/stems/pronouns.lexc:iesjguhti+Pron+Indef+Sg+Nom+Foc/Pos-k:iesj#guhti%>k # ;
src/fst/morphology/stems/pronouns.lexc:iesjguhti+Pron+Indef+Attr+Foc/Neg-k:iesj#guhti%>k # ; !OBS
src/fst/morphology/stems/pronouns.lexc:iesjguhti+Pron+Indef+Attr+Foc/Pos-k:iesj#guhti%>k # ; !OBS
src/fst/morphology/stems/pronouns.lexc:iesjguhti+Pron+Indef+Attr:iesj#guhti%>k # ; ! double, harmonised with sme
src/fst/morphology/stems/smj-abbreviations.lexc:j.d+Adv:j.d ab-dot-adv-itrab ; !ja% dakkára
src/fst/morphology/stems/smj-abbreviations.lexc:jdd+Adv:jdd ab-dot-adv-itrab ; !hæ?
src/fst/morphology/stems/smj-abbreviations.lexc:jed+Adv:jed ab-dot-adv-itrab ; !hæ?
src/fst/morphology/stems/smj-abbreviations.lexc:jd+Adv:jd ab-dot-adv-itrab ; !hæ?
src/fst/morphology/stems/smj-abbreviations.lexc:jdd+Adv:jdd ab-dot-adv-itrab ; !hæ?
src/fst/morphology/stems/nouns.lexc:judos+N+CmpN/SgN+CmpN/SgG+CmpN/PlG+Sem/Dummytag:juhtos ÅRES ;
src/fst/morphology/stems/nouns.lexc:jådos+N+Sem/Act:jåhtos ÅRES ;
src/fst/morphology/stems/nouns.lexc:jådås+N+Sem/Dummytag:jåhtås GÁMAS ;
src/fst/morphology/stems/smj-abbreviations.lexc:j.d.s+Adv:j.d.s ab-dot-adv-itrab ; !hæ?
src/fst/morphology/stems/smj-abbreviations.lexc:j.e+Adv:j.e ab-dot-adv-itrab ; !hæ
src/fst/morphology/stems/smj-abbreviations.lexc:je+Adv:je ab-dot-adv-itrab ; !hæ?
src/fst/morphology/stems/smj-abbreviations.lexc:jed+Adv:jed ab-dot-adv-itrab ; !hæ?
src/fst/morphology/stems/smj-abbreviations.lexc:j.i+Adv:j.i ab-dot-adv-itrab ; !ja ienep
src/fst/morphology/stems/smj-abbreviations.lexc:j.n.v+Adv:j.n.v ab-dot-adv-itrab ; !ja nav vijdábun !
src/fst/morphology/stems/smj-abbreviations.lexc:j.s+Adv:j.s ab-dot-adv-itrab ; !hæ?
src/fst/morphology/stems/smj-abbreviations.lexc:jsg.+N:julevsámegiella ab-dot-noun-itrab ;
src/fst/morphology/stems/smj-abbreviations.lexc:Kr.m+Adv+Sem/Time:Kr.m ab-dot-adv-itrab ;
src/fst/morphology/stems/smj-abbreviations.lexc:Kr.å+Adv+Sem/Time:Kr.å ab-dot-adv-itrab ;
src/fst/morphology/stems/nouns.lexc:labun+N+Sem/Dummytag+Err/Der:labun GAHPER ; ! bad der?
src/fst/morphology/stems/nouns.lexc:lájbbár+N+Err/Der+CmpN/SgN+CmpN/SgG+CmpN/PlG+Sem/Hum:lájbbár GUOLLÁR ; ! !Feil, baker er "lájbbo", her har ordboksforfatterne gjort feil når de har laget en avledning
src/fst/morphology/stems/nouns.lexc:låbdun+N+Err/Der+Sem/Ctain:låbdun GAHPER ; ! contraced stems don't make NomInstr, no verb låbddot
src/fst/morphology/stems/numerals.lexc:lågenanvuostas+v1+A+Ord+Err/Orth:lågenan#vuostas VUOSTASJ ;
src/fst/morphology/stems/nouns.lexc:låptun+N+Err/Der+CmpN/SgN+CmpN/SgG+CmpN/PlG+Sem/Obj:bevkun GAHPER ; !låpptit can't make this derivation
src/fst/morphology/stems/nouns.lexc:låptun+N+Err/Der+Sem/Ctain:låptun GAHPER ; ! contraced stems don't make NomInstr, no verb låpptot
src/fst/morphology/stems/smj-propernouns.lexc:Mančuria+Use/-Spell+OLang/SME:Mančuria ACCRA-plc ;
src/fst/morphology/stems/adjectives.lexc:materiáladahtes+A+Err/Der:materi^álad DIEHTEMAHTES ;
src/fst/morphology/stems/smj-abbreviations.lexc:miljo+N:miljo ab-dot-num ; !millijåvnnå
src/fst/morphology/stems/smj-acronyms.lexc:MuVá+N+Prop+Sem/Org+ACR+Err/Orth:MuVá ACRO_cons ; ! - propername according to čállinrávvagat
src/fst/morphology/stems/adjectives.lexc:måjo+A+CmpN/SgN+CmpN/PlG:måjo VINJO- ;
src/fst/morphology/stems/adjectives.lexc:måtso+A:måtso VINJO- ;
src/fst/morphology/stems/numerals.lexc:niellje+Err/Orth+Use/-Spell+Use/Marg+Use/NG:niellje# NLX ; !Err/Orth?
src/fst/morphology/stems/numerals.lexc:nubbe+A+Ord:nupp nubbecase ;
src/fst/morphology/stems/numerals.lexc:nubbe+A+Ord+Sg+Nom:nubbe%> K ;
src/fst/morphology/stems/numerals.lexc:nubbe+A+Ord+Ess:nubbe%>n K ;
src/fst/morphology/stems/numerals.lexc:nubbe+A+Ord+Sg+Ill:nubbá%>j K ;
src/fst/morphology/stems/numerals.lexc:nubbe+A+Ord+Cmp/SgGen:nuppe%> NUMERALCOMPOUNDS ;
src/fst/morphology/stems/pronouns.lexc:nubbe+Pron+Recipr+Ess:nubbe%>n K-CONS ;
src/fst/morphology/stems/pronouns.lexc:nubbe+Pron+Recipr+Par:nuppe%>t # ;
src/fst/morphology/stems/pronouns.lexc:nubbe+Pron+Recipr+Sg+Ill:nubbá%>j K-CONS ;
src/fst/morphology/stems/pronouns.lexc:nubbe+Pron+Recipr+Sg+Nom:nubbe%> K-VOW ;
src/fst/morphology/stems/pronouns.lexc:nubbe+Pron+Recipr+Attr:nubbe%> K-VOW ;
src/fst/morphology/stems/pronouns.lexc:nubbe+Pron+Recipr:nupp nubbecase ;
src/fst/morphology/stems/pronouns.lexc:nubbe+Pron+Indef+Ess:nubbe%>n K-CONS ;
src/fst/morphology/stems/pronouns.lexc:nubbe+Pron+Indef+Par:nuppe%>t # ;
src/fst/morphology/stems/pronouns.lexc:nubbe+Pron+Indef+Sg+Ill:nubbá%>j K-CONS ;
src/fst/morphology/stems/pronouns.lexc:nubbe+Pron+Indef+Sg+Nom:nubbe%> K-VOW ;
src/fst/morphology/stems/pronouns.lexc:nubbe+Pron+Indef+Attr:nuppe K-VOW ;
src/fst/morphology/stems/pronouns.lexc:nubbe+Pron+Indef:nupp nubbecase ;
src/fst/morphology/stems/numerals.lexc:nubbe+A+Ord:nupp nubbecase ;
src/fst/morphology/stems/numerals.lexc:nubbe+A+Ord+Sg+Nom:nubbe%> K ;
src/fst/morphology/stems/numerals.lexc:nubbe+A+Ord+Ess:nubbe%>n K ;
src/fst/morphology/stems/numerals.lexc:nubbe+A+Ord+Sg+Ill:nubbá%>j K ;
src/fst/morphology/stems/numerals.lexc:nubbe+A+Ord+Cmp/SgGen:nuppe%> NUMERALCOMPOUNDS ;
src/fst/morphology/stems/pronouns.lexc:nubbe+Pron+Recipr+Ess:nubbe%>n K-CONS ;
src/fst/morphology/stems/pronouns.lexc:nubbe+Pron+Recipr+Par:nuppe%>t # ;
src/fst/morphology/stems/pronouns.lexc:nubbe+Pron+Recipr+Sg+Ill:nubbá%>j K-CONS ;
src/fst/morphology/stems/pronouns.lexc:nubbe+Pron+Recipr+Sg+Nom:nubbe%> K-VOW ;
src/fst/morphology/stems/pronouns.lexc:nubbe+Pron+Recipr+Attr:nubbe%> K-VOW ;
src/fst/morphology/stems/pronouns.lexc:nubbe+Pron+Recipr:nupp nubbecase ;
src/fst/morphology/stems/pronouns.lexc:nubbe+Pron+Indef+Ess:nubbe%>n K-CONS ;
src/fst/morphology/stems/pronouns.lexc:nubbe+Pron+Indef+Par:nuppe%>t # ;
src/fst/morphology/stems/pronouns.lexc:nubbe+Pron+Indef+Sg+Ill:nubbá%>j K-CONS ;
src/fst/morphology/stems/pronouns.lexc:nubbe+Pron+Indef+Sg+Nom:nubbe%> K-VOW ;
src/fst/morphology/stems/pronouns.lexc:nubbe+Pron+Indef+Attr:nuppe K-VOW ;
src/fst/morphology/stems/pronouns.lexc:nubbe+Pron+Indef:nupp nubbecase ;
src/fst/morphology/stems/smj-abbreviations.lexc:sadj+A:sadj ab-dot-adj-trab ; !sadjásasj
src/fst/morphology/stems/adjectives.lexc:sahtemus+A+Err/Der:sahte TJAVGGÁMUS ; !must be sademus (generated by sadep)
src/fst/morphology/stems/adjectives.lexc:sebrudaklasj+A+Err/Der:sebrudahkaladtj ÅLLAGASJ ;
src/fst/morphology/stems/adjectives.lexc:sebrudaklasj+A+Err/Der:sebrudakladtj ÅLLAGASJ ;
src/fst/morphology/stems/smj-abbreviations.lexc:su+Adv:su ab-dot-adv-numnoab ; ! La stå ! hæ?
src/fst/morphology/stems/adjectives.lexc:suohkanlasj+A+Err/Der:suohkanl METÅVDÅLASJ;
src/fst/morphology/stems/adjectives.lexc:såbadimahtes+A+Err/Der+CmpN/SgN+CmpN/PlG:såbadim DIEHTEMAHTES ;
src/fst/morphology/stems/smj-propernouns.lexc:Tearbmasymposia+Use/-Spell+OLang/SME:Tearbmasymposia ACCRA-obj ;
src/fst/morphology/stems/adjectives.lexc:tjábbámus+A+Err/Der:tjábbá TJAVGGÁMUS ; !must be tjáppámus (generated by tjáppep)
src/fst/morphology/stems/adjectives.lexc:ulmusjlasj+A+Err/Der+CmpN/SgN+CmpN/PlG:ulmusjl DÁRBULASJ ;
src/fst/morphology/stems/smj-propernouns.lexc:Vuottnánáhpe+Use/-Spell:Vuottnán^áhpe MARJA-plc ; !
src/fst/morphology/stems/adjectives.lexc:ålleslasj+A+Err/Der:ållesl DÁRBULASJ ;
src/fst/morphology/stems/smj-abbreviations.lexc:åss+N:åss ab-dot-noun-itrab ; !åssudahka I'm thinking we can exclude +Err/Der and ab-dot and VINJO- from testing? Then what is left is:
|
By default everything containing And it makes sense to also exclude |
"færtguhti" is NOT in the lexicon:> not a word, maybe "færtguhtik" I don't know what to do with MWE: |
these might be typoes in lexc files? I.e. https://github.com/giellalt/lang-smj/blob/main/src/fst/morphology/stems/pronouns.lexc#L354-L366 https://github.com/giellalt/lang-smj/blob/main/src/fst/morphology/stems/pronouns.lexc#L433-L449 and https://github.com/giellalt/lang-smj/blob/main/src/fst/morphology/stems/numerals.lexc#L643-L650
mm, for now we can manually filter them from testing, I'll use |
That might be a good option, the current template only has: |
I see now that the relevant line in the extract lemma script is not as thorough as it should be regarding noise: This should be fixed. |
Done in giellalt/giella-core@3b766fb |
SMJ make check is failing:
It seams like it north sámi test data in SMJ:
The text was updated successfully, but these errors were encountered: