Fix Japanese handling

[polm]

This changes the Japanese tokenizer to use versions of mecab-python3 1.0
or greater. This means the package will work on Windows.

However, since the Japanese tokenizer pulls in heavy dependencies and
isn't necessary unless you're dealing with Japanese, I moved it to
optional dependencies. You can install sacrebleu with Japanese support
like below:

pip install sacrebleu[ja]

That will install mecab-python3 and ipadic.

This also includes basic tests to check that the tokenization is as
exepcted for IPAdic.

You may need to make changes to your automated testing setup to install the Japanese dependencies.

[polm]

I made the necessary changes to Travis and it looks like the tests are passing now.

[ozancaglayan]

Thank you for this nice patch! I made a small comment. So pip install sacrebleu[ja] does a full install with mecab, right?

[ozancaglayan]

Can you remove these comments related to 0.996.5 then?

[ozancaglayan]

Also, can you rebase your branch as master moved forward by 1 commit in the meanwhile?

[martinpopel]

@ozancaglayan I think you can merge master into this polm:japanese-fix branch yourself, e.g. by clicking on the "Update branch" button on GitHub. @mjpost prefers to squash all commits in the PR, so in the end it does not matter if you rebase the PR or merge in master.

[polm]

I removed the comments and updated the branch.

[polm]

So pip install sacrebleu[ja] does a full install with mecab, right?

That's correct.

[ozancaglayan]

Thanks for the PR. I will test this tomorrow a bit further and then @mjpost could do a release to fix Windows installation issue.

[ozancaglayan]

@polm why does tagger.version() still returns 0.996? Shouldn't this reflect MeCab' version?

[polm]

That's the version of the underlying C++ library, which is not the same as the version number used for mecab-python3, the Python wrapper.