Spacy Integration - "detect an empty sentence"

[mrw-dev]

Using your spacy integration the senticizer for Spacy will sometimes produce an empty sentence (using "en_core_web_sm"). These leads to the SpanMarkerTokenizer throwing an exception. Not sure how active this project is any more, but these seems like an easy fix. Is there a work-around already for this? Would you like the code updated to have one (I might be able to do this fix).

[tomaarsen]

Hello!

I don't think there's a workaround yet. Feel free to make a pull request and I'll try to have a look at it :)

• Tom Aarsen

[mwade-noetic]

Actually, using the en_core_web_lg model instead of the sm model mitigates the problem to a great extent (it just parses the sentences better). May not be worth a pull just yet, but I had to fix the same issue during training by pre-processing the sentences.

[mrw-dev]

Hi,

I have a one-line fix that should work for this as it keeps happening. spacy integration code that resolves this issue fairly nicely. I am not sure how to do the pull request, but the in the "spacy_integration.py" file I would propose the following change:

    def __call__(self, doc: Doc) -> Doc:
        """Fill `doc.ents` and `span.label_` using the chosen SpanMarker model."""
        sents = list(doc.sents)
        inputs = [[token.text if not token.is_space else "" for token in sent] for sent in sents]
        # Remove any sentences where the tokens are all empty strings or the sentence has 0 tokens.
        inputs = [sentence for sentence in inputs if any(sentence) and len(sentence) > 0]

        # use document-level context in the inference if the model was also trained that way
        if self.model.config.trained_with_document_context:
            inputs = self.convert_inputs_to_dataset(inputs)

I have updated the code and run a substantial amount of text through it. I did not however create any unit-tests but will be happy to do so you would consider this addition. It could be merged into the list comprehension above, but I wanted to show the logic clearly here.

[xxyzz]

The pipe() function seems a bit difficult to fix, because if empty sentences are removed it will be tricky to calculate the offset value. Couldn't the tokenizer code somehow ignore empty sentences?

[mwade-noetic]

Yes, I have run into this issue. I ended up dropping the spacy support altogether and just parse the sentences with my own parser that is targeted at my data. This resolved my issue with the empty sentences and the let me deal with better sentence capture. Contracts and other legal documents do not really parse well with Spacy anyways.