Skip to content

v0.8.1

Choose a tag to compare

@github-actions github-actions released this 25 Sep 21:30
· 1 commit to main since this release
955447d

Fixed

  • CoNLL-U token spans index the text the sentence declares. A sentence
    that carries a # text comment keeps that text, but spans were still
    computed as though the text were the forms joined by single spaces. Every
    corpus that writes SpaceAfter=No therefore drifted one byte per run-on
    token, and a reader validating spans against the text rejected the file:
    Universal Dependencies English-EWT failed on its first sentence, whose
    spans ran seven bytes past a 129-byte text. Spans now locate forms in the
    declared text and retain UTF-8 byte offsets whatever the corpus's spacing.
    Multiword range rows are preserved by both parser paths and through Layers
    records: each component word shares the range's complete surface span, as in
    the published Layers UD data. A form that cannot be aligned receives no span
    rather than a fabricated or out-of-bounds one.