Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot search CJK contents #2033

Closed
evannook opened this issue Feb 1, 2016 · 16 comments

Comments

@evannook
Copy link

commented Feb 1, 2016

Hi,

I've downloaded docker image mattermost/platform:latest and ran it for testing purpose, and found that the current version of mattermost cannot search CJK contents.
2016-02-01 09 07 22
2016-02-01 09 07 43

@it33

This comment has been minimized.

Copy link
Contributor

commented Feb 1, 2016

Please try adding * after your search term and use at least 3 characters? Full text search in your database setup is probably set to English, there would probably need a setting for CJK, if anyone on the community has guidance it would be highly appreciated

@evannook

This comment has been minimized.

Copy link
Author

commented Feb 1, 2016

I've found related discussions here: https://forum.mattermost.org/t/supporting-full-text-for-japanese-and-other-idiographic-languages/779

I'm willing to help solving this issue.

@it33

This comment has been minimized.

Copy link
Contributor

commented Feb 1, 2016

Fantastic! We're open to ideas, which approach do you feel might be best?

@evannook

This comment has been minimized.

Copy link
Author

commented Feb 3, 2016

IMHO, it's better if we can handle this issue in database layer. According to http://dev.mysql.com/doc/refman/5.7/en/fulltext-search.html, MySQL 5.7+ has built-in CJK full text search capability that can solve the issue. Keep to find out how to gracefully solve this issue with PostgreSQL...

@it33

This comment has been minimized.

Copy link
Contributor

commented Feb 4, 2016

Hi @evannook, would you be open to testing the instructions and reporting back if they work?

If so, they could be added to Mattermost documentation to help other CJK users

@it33 it33 closed this Feb 4, 2016

@it33 it33 reopened this Feb 4, 2016

@it33

This comment has been minimized.

Copy link
Contributor

commented Feb 4, 2016

Sorry, hit the wrong button, didn't meant to close,

@artiya4u

This comment has been minimized.

Copy link

commented Feb 10, 2016

I have a workaround with MySql 5.7, using full text index with ngram parser.
DROP INDEX idx_posts_message_txt ON Posts;
CREATE FULLTEXT INDEX idx_posts_message_txt ON Posts (Message) WITH PARSER ngram;

For more information see MySql document:
ngram Full-Text Parser

@it33

This comment has been minimized.

Copy link
Contributor

commented Feb 12, 2016

Thanks @artiya4u!

Really helpful, can anyone in the community verify this approach?

If so, would love to add it to the documentation--and if anyone knows the equivalent for Postgres, please share as well?

@seikichi

This comment has been minimized.

Copy link

commented Feb 14, 2016

I tried the ngram parser approach. It works fine! Thanks @artiya4u!

I uploaded the docker image to DockerHub, so you can try by the following command:

docker run --name mattermost-cjk -d --publish 8065:80 seikichi/mattermost

If you interested the Dockerfile see this commit.

@it33

This comment has been minimized.

Copy link
Contributor

commented Feb 21, 2016

Thanks to everyone for helping solve this issue with CJK search. A note has been added to the database requirements documentation referencing ngram parsing and this thread.

Closing this thread for now. Please file a new issue if there's any further issue with CJK search.

@it33 it33 closed this Feb 21, 2016

@evannook

This comment has been minimized.

Copy link
Author

commented Mar 1, 2016

Sorry for replying so late, the solution provided by @artiya4u works for me, thank you all. :D

@schemacs

This comment has been minimized.

Copy link

commented Apr 10, 2016

@evannook Is there any way to do search CJK on PostgreSQL? Or any help on migration PostgreSQL database to MySQL?

@schemacs

This comment has been minimized.

Copy link

commented Apr 12, 2016

@cometkim cometkim referenced this issue Nov 15, 2016
0 of 1 task complete
@accordtsai

This comment has been minimized.

Copy link

commented Nov 23, 2016

I confirm the solution proposed by @artiya4u works on Mattermost 3.5 and MySQL 5.7 community for searching Tradiational Chinese characters.

Is it possible to include this as the default Mattermost behaviour (i.e. using the ngram parser if mysql 5.7+ is detected?)

@it33

This comment has been minimized.

Copy link
Contributor

commented Nov 23, 2016

Hi @accordtsai, not sure, we're not really experts. Is this something you'd be interested in contributing to the community?

@rock0018

This comment has been minimized.

Copy link

commented Dec 8, 2016

@artiya4u ,search simple chinese phase with mysql.5.7.16 also work .thanks a lot

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.