-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Affixes propernouns test vil ha små bokstaver ( #56
Comments
Comment 9508Date: 2014-06-24 11:52:16 +0200 LEXC test 3: analyser-gt-norm.xfst + affixes/propernouns.lexc - 514/578/1092 FAIL [1/3][PASS] Tjuorri+N+Prop+Ani+Sg+Nom => Tjuorri Vanlig yaml test fungerer som den skal med tanke på store bokstaver for propernouns. Testen i affix-fila vil også ha propernouns med liten forbokstav. |
Comment 9509Date: 2014-06-24 11:53:24 +0200 Det er Sjur som har jobbet med disse testene så jeg endrer "Assigned to" til Sjur. |
Comment 9515Date: 2014-06-30 10:09:56 +0200 Testane viser at fst-en genererer ordformer som ikkje skal genererast (liten forbokstav ved namn). Det er altså ein feil i fst-en, og ikkje med testoppsettet. Det ser ut som om alle namn får feilaktig liten fyrstebokstav. I tillegg er det nokre namn som er feil tagga: Test 131: Bieva (Lexical/Generation)[1/3][FAIL] Bieva+N+Prop+Plc+Sg+Nom => Missing results: Bieva Test 131 - Passes: 0, Fails: 3, Total: 3 Dvs Sg vs Pl. I og med at alle namn blir råka, kan det sjå ut som om det er ein feil i bygginga heller enn ein feil i lexc- eller twolc-koden. |
Comment 9517Date: 2014-06-30 16:33:28 +0200 Eg og Trond såg på desse i dag, men fann ikkje ut av det. Det gjeld alle fst-ar i smj (bortsett frå -raw-), og det gjeld berre smj, ingen andre språk. Det er riktig mystisk, fordi byggjestega for alle språk er like. Eg sjekka dei delane som er spesifikke for smj, men det var ingen ting der - problemet finst òg i dei temporære fst-ane som blir laga likt for alle språk, før den språkspesifikke prosesseringa byrjar. Inga: |
Comment 9544Date: 2014-08-06 13:40:24 +0200 I hand this over to Tomi, and add myself and Trond to the Cc list. Also changed Product and Component. |
Comment 9545Date: 2014-08-06 15:16:55 +0200 Inga's testing has been done using Xerox, but Hfst has some issues as well, as it turns out. It might be a hint at what the problem is: $ hfst-lookup src/analyser-gt-desc.hfst
$ hfst-lookup src/analyser-gt-desc.hfst
That is, something is inserting an unlimited number of an/some optional symbol(s) in the analysing fst, causing it to err as above. Generation works fine: $ hfst-lookup src/generator-gt-desc.hfst
|
Comment 9549Date: 2014-08-07 09:05:28 +0200 Forget comment #5, I had local modifications that destroyed some of the results in unpredictable ways. The following is produced with clean data directly from the svn repository, using svn revision 98060: Xerox: $ lookup src/analyser-gt-desc.xfst oslo ^C That is, Xerox has the bug with lower-case initial caps of proper nouns. Hfst: $ hfst-lookup src/analyser-gt-desc.hfst
That is, everything works as expected using Hfst. The questions are:
|
Comment 9550Date: 2014-08-07 10:47:13 +0200 The bug is related to the regex for handling downcasing of derived proper strings. If I remove the following line from am-shared/src-dir-include.am:
both Hfst and Xerox behave properly. Hfst: $ hfst-lookup analyser-gt-desc.hfst
Xerox: $ lookup analyser-gt-desc.xfst oslo ^C That is, without the downcasing of derived proper nouns, everything is working ok for both Xerox and Hfst. But when adding the downcasing, things break for Xerox but not for Hfst. |
Comment 9551Date: 2014-08-07 10:59:12 +0200 The first question remains: Why only smj? |
Comment 9552Date: 2014-08-07 11:40:49 +0200 (In reply to comment #8)
Both questions remain, and we have a third one: How can we get downcasing of proper nouns to work properly and identically for both Xerox and Hfst, without affecting analysis of regular proper nouns? And preferably by using exactly the same build instructions for both Xerox and Hfst. |
Comment 9560Date: 2014-08-20 12:36:08 +0200 In affix file for smj propernouns the obligatory uppercase strings need to be defined also with @U.Cap.Obl@ The creation of proper nouns has two paths, one with obligatory initial uppercase, and with optional initial uppercase: @U.Cap.Obl@ ProperNoun ; ! These flags are for Oslo example is directed to ACCRA-plc lexicon: LEXICON ACCRA-plc !!= @code@ Vowel-final names where caseendings are added directly, no cg. Place names LEXICON ACCRADECL-PLC LEXICON ACCRA-DC In above example @U.Cap.Opt@ path with 'regular' cases is not filtered out from fst since there is no @U.Cap.Obl@ flag. |
Comment 9566Date: 2014-08-27 15:30:36 +0200 The build system is working as it should, and the basic flag setup is correct, as demonstrated by the following: $ lookup -q src/generator-gt-desc.xfst The problem is that several/many of the continuation lexicons for proper nouns are lacking the required flag(s). The job is thus to go through the file src/morphology/affixes/propernouns.lexc, and make sure all lexicons are set up as e.g. EATNAMAT-plc (the one for Vuolleednama), which works as intended. This is tested against svn revision 98779. |
Comment 9567Date: 2014-08-27 15:42:53 +0200 Actually, the main reason is that most derivations are Sub-marked, e.g. all LASJ derivations from placenames. I have removed some of the sub-marking for cases where the yaml tests say that they should be accepted by the normative transducer. |
Comment 9568Date: 2014-08-27 16:56:34 +0200 Handing this one over to Sandra. |
Comment 9569Date: 2014-08-28 08:15:55 +0200 One more observation: -lasj is Sub-marked in its own lexicon: LEXICON LASJ !SUB !+CmpN/SgN +CmpN/SgG +CmpN/PlG !from placenames This should probably be reevaluated, since subbing all lasj derivations is probably too strong. I will remove the sub-tagging now, but leave a note for future work. |
Comment 9575Date: 2014-09-11 22:49:35 +0200 This comment is only indirectly related to this bug, but it still is worth mentioning. "oslolasj" type derivations (with initial lowercase letter) have not been working with hfst, because of a bug / xerox incompatibility related to composition with flag diacritics. In the very latest hfst code this is now fixed: $ hfst-lookup src/analyser-gt-desc.hfst
That is, Xerox and Hfst behave the same. NB! This really requires the very latest hfst code, as of today. |
Comment 9610Date: 2014-10-07 08:21:19 +0200 As of Hfst 3.8 we have an official release with all required bug fixes included. Summary after all backs and forths:
Sandra, you can close this bug as fixed as soon as you have verified that all proper noun continuation lexicons have the correct flag diacritics. |
This issue was created automatically with bugzilla2github
Bugzilla Bug 1874
Date: 2014-06-24T11:52:16+02:00
From: Inga Lill Sigga Mikkelsen <<inga.l.mikkelsen>>
To: Sandra Nystø Rahka <<sandra.rahka>>
CC: inga.l.mikkelsen, lene.antonsen, sjur.n.moshagen, thomas.omma, tomi.k.pieski, trond.trosterud
Last updated: 2014-10-08T10:21:26+02:00
The text was updated successfully, but these errors were encountered: