You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit was created on GitHub.com and signed with GitHub’s verified signature.
Fixed
CoNLL-U token spans index the text the sentence declares. A sentence
that carries a # text comment keeps that text, but spans were still
computed as though the text were the forms joined by single spaces. Every
corpus that writes SpaceAfter=No therefore drifted one byte per run-on
token, and a reader validating spans against the text rejected the file:
Universal Dependencies English-EWT failed on its first sentence, whose
spans ran seven bytes past a 129-byte text. Spans now locate forms in the
declared text and retain UTF-8 byte offsets whatever the corpus's spacing.
Multiword range rows are preserved by both parser paths and through Layers
records: each component word shares the range's complete surface span, as in
the published Layers UD data. A form that cannot be aligned receives no span
rather than a fabricated or out-of-bounds one.