Span with surrounding quotation marks appears as if without ones

[cjer]

Span annotation with beginning or closing quotation mark (and I believe other punctuation marks as well) appears as if the punctuation is not included in the mark. This made annotators miss a lot of these small border issues.

Examples:

"הראל" will visually appear the same as הראל, but as you can see they are annotated differently, as recognized by the curation interface.

Same goes for חטיבת "הראל" and חטיבת "הראל

[reckart]

Sounds like an edge-case. I assume the quotes are not unicode RTL characters, right? Can you verify whether the quotes are detected as separate tokens? So "הראל" should consist of three tokens. You could check that e.g. by exporting the data as TSV and seeing whether ", הראל and " all appear on separate lines. If they are, then we have an unknown bug. If they are not and this is a mixed RTL-LTR token, then it sounds like a known bug (#283).

[reckart]

Actually, re-reading #283 it sounds like the same thing.

[cjer]

Yes these are regular ASCII quotation marks. And yes, they are recognized as separate tokens.