-
Notifications
You must be signed in to change notification settings - Fork 10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement text validation by default, even for shallow validation #24
Comments
Please clarify what is exactly 'invalid' in this case: |
The head is part of the paragraph, so the text of the paragraph is assumed to also cover the head. |
It still isn't clear what you mean here: On the fly testing is quite hard then... |
So the conclusion is: |
Agreed..
|
…ent instantiation) (proycon/folia#24)
…cter at some point in the future) #24
Should still be expanded with offset/ref checks as well |
see also LanguageMachines/libfolia#9 |
Proposal for Text ValidationFoLiA allows for text on multiple structural levels. Text on words higher than the word level (e.g sentence level, paragraph level) by definition reflect untokenised text. When text is specified on multiple levels, there is a level of redundancy and it may occur that the text on one level conflicts with the text on another level, this is a problem that needs to be detected at validation time. FoLiA v1.5 intends to enforce this. Text validation has two aspects:
The main issue in text checking is how to deal with whitespaces: We can go for the very strict approach, which seems to be the road we've been on until now, where whitespace has to be exactly matched ( A linebreak can be considered a special kind of whitespace. For the more lenient proposal on text validation, I think we can consider newlines (and carriage returns) again the same as spaces. We're obviously not the first ones to run into this issue, the W3C has thought about it too and there is a When it comes to XML parsing and serialisation, I think the output should match the input, so retaining all tabs/spaces/newlines as-is. If we don't do that may be forced to solve issue #12 (CDATA). My proposal for FoLiA v1.5 would be:
Implementation-wise, @kosloot identifies an important issue in LanguageMachines/libfolia#9:
Another thing that complicates text validation is corrections, the authoritative route through <s><t>This is a test</t></s>
<w>
<correction>
<new><t>test</t></new>
<original auth="no"><t>test</t></original>
</correction>
</w> This however does raise some issues which need to be addressed still and are not covered here yet, I'll write up on it in a separate comment. Thoughts and comments? |
Most of this make sense. <s>
<t>This is odd</t>
<w id=1 offset=0>
<t>This</t>
</w>
<w id=2 offset=6>
<t>is</t>
</w>
<w id=13 offset=8>
<t>odd</t>
</w>
</s>`
I would think odd has an offset of 24. not 8 |
Remark: it is doable to have NEWLINES explicit in FoLiA text using <br/> tags but multiple spaces and tabs will still be a problem. So may just forget about the <br/> |
… , using normalize_spaces(), added a normalize_spaces attribute on text() methods.
Text validation should not descend into morphology, as complex morphemes may not correspond entirely with higher levels. Example: <w xml:id="WR-P-E-J-0000000001.p.1.s.2.w.16">
<t>genealogie</t>
<pos class="N(soort,ev,basis,zijd,stan)" set="https://raw.githubusercontent.com/proycon/folia/master/setdefinitions/frog-mbpos-cgn"/>
<lemma class="genealogie"/>
<morphology>
<morpheme class="complex">
<t>genealogie</t>
<feat class="[[genealogisch]adjective[ie]]noun/singular" subset="structure"/>
<pos class="N" set="http://ilk.uvt.nl/folia/sets/frog-mbpos-clex"/>
<morpheme class="complex">
<feat class="N_A*" subset="applied_rule"/>
<feat class="[[genealogisch]adjective[ie]]noun" subset="structure"/>
<pos class="N" set="http://ilk.uvt.nl/folia/sets/frog-mbpos-clex"/>
<morpheme class="stem">
<t>genealogisch</t>
<pos class="A" set="http://ilk.uvt.nl/folia/sets/frog-mbpos-clex"/>
</morpheme>
<morpheme class="affix">
<t>ie</t>
<feat class="[ie]" subset="structure"/>
</morpheme>
</morpheme>
<morpheme class="inflection">
<feat class="singular" subset="inflection"/>
</morpheme>
</morpheme>
</morphology>
</w> |
@kosloot , I hadn't replied to your offset concern yet. I understand the concern and think I agree with you. It would be a bit too odd indeed and the offset would even lose information. So let's revert this idea and just count each unicode point as 1 (as it was) and not normalize the offsets in any way. This would imply the check is very strict and an error is easily made (but perhaps a nice counterweight to the offset-less text validation which is now more flexible). |
…d (part) [the new text validator discovered it] #24
…extContent.ref attribute is now a string (ID), getreference() method resolves the actual reference and does the final validation
A small remark on the implementation of text validation: I consider it a validation step and not a parse step; so in the Python library text checking is only done when explicitly asked (not on every document load, to save performance). The validation methods and by extension the validator tool of course always explicitly turn text checking on (for FoLiA v1.5+). |
I don't agree (fully). |
…iven a textclass, what kind of correction handling is appropriate. This is used by text validation (proycon/folia#24). The method is limited and can not deal with complex situations (nested corrections, inconsistencies), in such cases text validation will be skipped alltogether for that element.
…ent instantiation) (proycon/folia#24)
… , using normalize_spaces(), added a normalize_spaces attribute on text() methods.
…extContent.ref attribute is now a string (ID), getreference() method resolves the actual reference and does the final validation
…iven a textclass, what kind of correction handling is appropriate. This is used by text validation (proycon/folia#24). The method is limited and can not deal with complex situations (nested corrections, inconsistencies), in such cases text validation will be skipped alltogether for that element.
Fail on invalid constructions like:
and check offsets.
The text was updated successfully, but these errors were encountered: