Bug report in SequenceTokenizerNew

[Jabher]

SequenceTokenizerNew fails on following call:

sentenceTokenizer.tokenize('"All ticketed passengers should now be in the Blue Concourse sleep lounge. Make sure your validation papers are in order. Thank you". The upstairs lounge was not at all grungy.') (quote from "The Jaunt" by Stephen King)

with following message:

{
    "message": "Expected [ \\t\\n\\r.?!] or [)\\]}\"'`’] but \"M\" found.",
    "expected": [
        {
            "type": "class",
            "parts": [
                " ",
                "\t",
                "\n",
                "\r",
                ".",
                "?",
                "!"
            ],
            "inverted": false,
            "ignoreCase": false
        },
        {
            "type": "class",
            "parts": [
                ")",
                "]",
                "}",
                "\"",
                "'",
                "`",
                "’"
            ],
            "inverted": false,
            "ignoreCase": false
        }
    ],
    "found": "M",
    "location": {
        "start": {
            "offset": 75,
            "line": 1,
            "column": 76
        },
        "end": {
            "offset": 76,
            "line": 1,
            "column": 77
        }
    },
    "name": "SyntaxError"
}

[Hugo-ter-Doest]

Srictly speaking the second double quote is in the wrong place, It should be right behind the period like this:

Thank you."

This results in a syntax error as well, so I will try and make it more robust.

[Hugo-ter-Doest]

Partly solved with #658

It now can handle multiple sentences surrounded by quotes. It cannot handle the position of the quote symbol in your example. So I will leave the issue open and will try later to make it more robust,

[Jabher]

Srictly speaking the second double quote is in the wrong place, It should be right behind the period like this:

Thank you."

This results in a syntax error as well, so I will try and make it more robust.

I totally agree :) that's why I specifically left a note that it is not my text but sci-fi classics I used.

[Hugo-ter-Doest]

Won't fix it further since the parser mechanism underlying this sentence tokenizer is not flexible enough for this.