Token offsets are incorrect

[Trey314159]

I had to build the plugin myself for Elasticsearch v5.3.2.

If I analyze this string sách .. sách ; 東京都 I get the following:

{
  "tokens" : [
    {
      "token" : "sách",
      "start_offset" : 0,
      "end_offset" : 4,
      "type" : "phrase",
      "position" : 0
    },
    {
      "token" : "sách",
      "start_offset" : 5,
      "end_offset" : 9,
      "type" : "phrase",
      "position" : 3
    },
    {
      "token" : "東",
      "start_offset" : 10,
      "end_offset" : 11,
      "type" : "residual",
      "position" : 7
    },
    {
      "token" : "京",
      "start_offset" : 12,
      "end_offset" : 13,
      "type" : "residual",
      "position" : 11
    },
    {
      "token" : "都",
      "start_offset" : 14,
      "end_offset" : 15,
      "type" : "residual",
      "position" : 15
    }
  ]
}

The start and end offsets are incorrect and do not map correctly back into the original string. Each start_offset is just one more than the end_offset of the previous token. This is incorrect when there is more than one character or less than one character between tokens.

I believe that this would be the correct output:

{
  "tokens" : [
    {
      "token" : "sach",
      "start_offset" : 0,
      "end_offset" : 4,
      "type" : "phrase",
      "position" : 0
    },
    {
      "token" : "sach",
      "start_offset" : 8,
      "end_offset" : 12,
      "type" : "phrase",
      "position" : 1
    },
    {
      "token" : "東",
      "start_offset" : 15,
      "end_offset" : 16,
      "type" : "residual",
      "position" : 2
    },
    {
      "token" : "京",
      "start_offset" : 16,
      "end_offset" : 17,
      "type" : "residual",
      "position" : 3
    },
    {
      "token" : "都",
      "start_offset" : 17,
      "end_offset" : 18,
      "type" : "residual",
      "position" : 4
    }
  ]
}

Note that the position values are also not correct.

[duydo]

Thanks for raising this issue @Trey314159, the position value plays an important role in searching phrases, I'll look at it.

Btw, the output tokens should keep their accent marks, this is the goal of the plugin.

[Trey314159]

Oops—I agree that the tokens should keep their accent marks! I ran the text through a different analyzer to get the correct offsets in my sample output. I changed the types to match, but not the tokens.

[Trey314159]

FYI, I think the incorrect offsets or position values are causing StringIndexOutOfBoundsExceptions in my re-indexing script. I've posted a stack trace for one instance if you want to take a look.

[duydo]

I close this issue, it will be fixed in new tokenizer #37