-
Notifications
You must be signed in to change notification settings - Fork 1k
/
textsearch.txt
100 lines (67 loc) · 6.66 KB
/
textsearch.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
[[index-parameters]]
Index Parameters and Full-Text Search
-------------------------------------
When defining a mixed index, a list of parameters can be optionally specified for each property key added to the index. These parameters control how the particular key is to be indexed. Titan recognizes the following index parameters. Whether these are supported depends on the configured index backend. A particular index backend might also support custom parameters in addition to the ones listed here.
[[text-search]]
Full-Text Search
~~~~~~~~~~~~~~~~
When indexing string values, that is property keys with `String.class` data type, one has the choice to either index those as text or character strings which is controlled by the `mapping` parameter type.
When the value is indexed as text, the string is tokenized into a bag of words which allows the user to efficiently query for all matches that contain one or multiple words. This is commonly referred to as *full-text search*.
When the value is indexed as a character string, the string is index "as-is" without any further analysis or tokenization. This facilitates queries looking for an exact character sequence match. This is commonly referred to as *string search*.
Full-Text Search
^^^^^^^^^^^^^^^^
By default, strings are indexed as text. To make this indexing option explicit, one can define a mapping when indexing a property key as text.
[source, gremlin]
mgmt = graph.openManagement()
summary = mgmt.makePropertyKey('booksummary').dataType(String.class).make()
mgmt.buildIndex('booksBySummary', Vertex.class).addKey(summary, Mapping.TEXT.asParameter()).buildMixedIndex("search")
mgmt.commit()
This is identical to a standard mixed index definition with the only addition of an extra parameter that specifies the mapping in the index - in this case `Mapping.TEXT`.
When a string property is indexed as text, the string value is tokenized into a bag of tokens. The exact tokenization depends on the indexing backend and its configuration. Titan's default tokenization splits the string on non-alphanumeric characters and removes any tokens with less than 2 characters. The tokenization used by an indexing backend may differ (e.g. stop words are removed) which can lead to minor differences in how full-text search queries are handled for modifications inside a transaction and committed data in the indexing backend.
When a string property is indexed as text, only full-text search predicates are supported in graph queries by the indexing backend. Full-text search is case-insensitive.
* textContains: is true if (at least) one word inside the text string matches the query string
* textContainsPrefix: is true if (at least) one word inside the text string begins with the query string
* textContainsRegex: is true if (at least) one word inside the text string matches the given regular expression
String search predicates (see below) may be used in queries, but those require filtering in memory which can be very costly.
String Search
^^^^^^^^^^^^^
To index string properties as character sequences without any analysis or tokenization, specify the mapping as `Mapping.STRING`:
[source, gremlin]
mgmt = graph.openManagement()
name = mgmt.makePropertyKey('bookname').dataType(String.class).make()
mgmt.buildIndex('booksBySummary', Vertex.class).addKey(name, Mapping.STRING.asParameter()).buildMixedIndex("search")
mgmt.commit()
When a string mapping is configured, the string value is indexed and can be queried "as-is" - including stop words and non-letter characters. However, in this case the query must match the entire string value. Hence, the string mapping is useful when indexing short character sequences that are considered to be one token.
When a string property is indexed as string, only the following predicates are supported in graph queries by the indexing backend. String search is case-sensitive.
* eq: if the string is identical to the query string
* neq: if the string is different than the query string
* textPrefix: if the string value starts with the given query string
* textRegex: if the string value matches the given regular expression in its entirety
Full-text search predicates may be used in queries, but those require filtering in memory which can be very costly.
Full text and string search
^^^^^^^^^^^^^^^^^^^^^^^^^^^
If you are using ElasticSearch it is possible to index properties as both text and string allowing you to use all of the predicates for exact and fuzzy matching.
[source, gremlin]
mgmt = graph.openManagement()
summary = mgmt.makePropertyKey('booksummary').dataType(String.class).make()
mgmt.buildIndex('booksBySummary', Vertex.class).addKey(summary, Mapping.TEXTSTRING.asParameter()).buildMixedIndex("search")
mgmt.commit()
Note that the data will be stored in the index twice, once for exact matching and once for fuzzy matching.
Field Mapping
~~~~~~~~~~~~~
[[index-local-field-mapping]]
Individual Field Mapping
^^^^^^^^^^^^^^^^^^^^^^^^
By default, Titan will encode property keys to generate a unique field name for the property key in the mixed index. If one wants to query the mixed index directly in the external index backend can be difficult to deal with and are illegible. For this use case, the field name can be explicitly specified through a parameter.
[source, gremlin]
mgmt = graph.openManagement()
name = mgmt.makePropertyKey('bookname').dataType(String.class).make()
mgmt.buildIndex('booksBySummary', Vertex.class).addKey(name, Parameter.of('mapped-name', 'bookname')).buildMixedIndex("search")
mgmt.commit()
With this field mapping defined as a parameter, Titan will use the same name for the field in the `booksBySummary` index created in the external index system as for the property key. Note, that it must be ensured that the given field name is unique in the index.
[[index-global-field-mapping]]
Global Field Mapping
^^^^^^^^^^^^^^^^^^^^
Instead of individually adjusting the field mapping for every key added to a mixed index, one can instruct Titan to always set the field name in the external index to be identical to the property key name. This is accomplished by enabling the configuration option `map-name` which is configured per indexing backend. If this option is enabled for a particular indexing backend, then all mixed indexes defined against said backend will use field names identical to the property key names.
However, this approach has two limitations: 1) The user has to ensure that the property key names are valid field names for the indexing backend and 2) renaming the property key will NOT rename the field name in the index which can lead to naming collisions that the user has to be aware of and avoid.
Note, that individual field mappings as described above can be used to overwrite the default name for a particular key.