ii: remove redudant normalize on search as much as possible #1421
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@kou @HashidaTKS Could you review this PR? |
kou
reviewed
Sep 28, 2022
lib/db.c
Outdated
| @@ -3748,6 +3748,34 @@ grn_table_add_by_key(grn_ctx *ctx, grn_obj *table, grn_obj *key, int *added) | |||
| return id; | |||
| } | |||
|
|
|||
| grn_id | |||
| grn_table_get_without_normalize(grn_ctx *ctx, | |||
There was a problem hiding this comment.
I don't want to add this. If we add this, we want to add grn_table_cursor_open_without_nromalize().
I think that this is a ii problem. ii should not re-get token from token id. ii should use source of a token instead of re-getting token. We may be able to use grn_token_get_source_offset() and grn_token_get_source_length() for it. (We can add grn_token_get_source() for convenient.)
kou
force-pushed
the
remove-redundant-normalize
branch
from
f7b9883
to
17069d0
Compare
Currently, a search keyword execute normalize twice.
Therefore, the search keyword is normalized involuntarily depending on
settings NormalizerTable.
For example, If we define NormalizerTable as below "BCD" is normalized
like "BCD" -> normalize -> "bcd" -> tokenize -> "bc"/"cd"/"d" ->
normlizer -> "bk"/"cd"/"d".
load --table ColumnNormalizations
[
{"target_column": "c", "normalized": "k"},
{"target_column": "ＣＤ", "normalized": "cd"}
]
Therefore, if we use a search keyword "BCD", "ABCD" is not hit.
If we use a search keyword "CD", "ABCD" is hit.
kou
force-pushed
the
remove-redundant-normalize
branch
from
17069d0
to
6e078c9
Compare
kou
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Some normalizes such as
NormalizerTablearen't idempotent. So we should not normalize the inputmultiple times.
The current
token_info_open()related codes normalize the input multiple times. This change avoidsmultiple normalizations as much as possible. If a tokenizer enables
report_source_location, we canavoid multiple normalizations all time. But if a tokenizer doesn't enable
report_source_location, we maynormalize the input multiple times.
See the added tests for problem cases.