Add some more test cases for tokenization and ascii folding

[SiarheiFedartsou]

👋 I did some awesome work for the Pelias project and would love for everyone to have a look at it and provide feedback.

---

Here's the reason for this change 🚀

There are some questions in discussion of this PR #498, so I'd like to propose to extend a bit existing test harness to kind of document current state of the things. So current schema:

• tokenizes by whitespace, hyphen, slashes (but doesn't by "dash" character for example - I'd expect ICU tokenizer to work differently here btw 🤔 )
• normalizes thai digits to Arabic ones (I think we can extrapolate it to any other digits writing system)
• removes tonal marks in Thai script
• we never make digits "glued" to the end of word a separate token
• we don't tokenize Asian languages which don't use whitespaces properly

---

Here's what actually got changed 👏

• Added tests

---

Here's how others can test the changes 👀

Run tests :)

[orangejulius]

Looks good to me, these will be helpful tests to have as we take a look at #498

[missinglink]

Hi @SiarheiFedartsou I just rebased origin/master after merging your other PR and these cases now fail due to differences between the two tokenizers.

Would you mind updating it to reflect the behaviour you were expecting from the icuTokenizer 🙏

[missinglink]

Are these failures expected?

✖ thai_tonemarks
-----------------
  operator: deepEqual
  expected: |-
    { '@pos0': [ 'กกกกขขขขคคคคฆฆฆฆ' ] }
  actual: |-
    {}

✖ chinese_address
------------------
  operator: deepEqual
  expected: |-
    { '@pos0': [ '北京市朝阳区东三环中路1号国际大厦a座1001室' ] }
  actual: |-
    {}

[SiarheiFedartsou]

Are these failures expected?

✖ thai_tonemarks
-----------------
  operator: deepEqual
  expected: |-
    { '@pos0': [ 'กกกกขขขขคคคคฆฆฆฆ' ] }
  actual: |-
    {}

✖ chinese_address
------------------
  operator: deepEqual
  expected: |-
    { '@pos0': [ '北京市朝阳区东三环中路1号国际大厦a座1001室' ] }
  actual: |-
    {}

I doubt it :) I will look into it nearest days. Most likely we should have different expected results for ICU and no-ICU...

[SiarheiFedartsou]

@missinglink done. PTAL again.