fix: Char covert to String on remove specified unicode #1664

coderzc · 2021-11-22T11:47:18Z

codecov · 2021-11-22T11:54:44Z

Codecov Report

Merging #1664 (5189258) into master (5bd37d8) will increase coverage by 0.40%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##             master    #1664      +/-   ##
============================================
+ Coverage     66.56%   66.97%   +0.40%     
- Complexity     7039     7068      +29     
============================================
  Files           421      421              
  Lines         34673    34675       +2     
  Branches       4803     4803              
============================================
+ Hits          23081    23223     +142     
+ Misses         9247     9107     -140     
  Partials       2345     2345

Impacted Files	Coverage Δ
.../baidu/hugegraph/backend/query/ConditionQuery.java	`85.94% <100.00%> (+0.15%)`	⬆️
...du/hugegraph/backend/tx/GraphIndexTransaction.java	`83.81% <100.00%> (-0.04%)`	⬇️
...c/main/java/com/baidu/hugegraph/task/HugeTask.java	`72.30% <0.00%> (+0.30%)`	⬆️
...va/com/baidu/hugegraph/task/ServerInfoManager.java	`71.34% <0.00%> (+0.56%)`	⬆️
...hugegraph/backend/query/ConditionQueryFlatten.java	`75.63% <0.00%> (+0.84%)`	⬆️
.../java/com/baidu/hugegraph/auth/RolePermission.java	`91.76% <0.00%> (+1.17%)`	⬆️
...in/java/com/baidu/hugegraph/auth/HugeResource.java	`79.22% <0.00%> (+1.29%)`	⬆️
...baidu/hugegraph/backend/store/mysql/MysqlUtil.java	`96.61% <0.00%> (+3.38%)`	⬆️
...va/com/baidu/hugegraph/auth/HugeAuthenticator.java	`40.21% <0.00%> (+5.43%)`	⬆️
...aidu/hugegraph/backend/store/mysql/MysqlTable.java	`81.59% <0.00%> (+5.76%)`	⬆️
... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5bd37d8...5189258. Read the comment docs.

javeme · 2021-11-23T04:07:38Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/tx/GraphIndexTransaction.java

-        }
+
+        // Ignore unicode \u0000 to \u0003
+        words.removeAll(IGNORE_SYM_SET);


can add a test case?

javeme · 2021-11-23T08:22:48Z

hugegraph-test/src/main/java/com/baidu/hugegraph/core/VertexCoreTest.java

@@ -8582,6 +8582,29 @@ public void testQueryByHasIdEmptyListInPage() {
        Assert.assertNull(page);
    }

+    @Test
+    public void testIgnoreSymbol() {


rename to test testQueryBySearchIndexWithSpecialSymbol

javeme · 2021-11-23T08:24:45Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/tx/GraphIndexTransaction.java

@@ -103,6 +103,15 @@
    public static final String INDEX_SYM_NULL = "\u0001";
    public static final String INDEX_SYM_EMPTY = "\u0002";
    public static final char INDEX_SYM_MAX = '\u0003';
+    public static final Set<String> IGNORE_SYM_SET;


add fixed #1638 to commit message

javeme · 2021-11-23T08:26:25Z

hugegraph-test/src/main/java/com/baidu/hugegraph/core/VertexCoreTest.java

+             .by("name").search().ifNotExist().create();
+
+        Vertex vertex = graph.addVertex(T.label, "person", "name",
+                                        "xyz\u0000abc", "city", "Hongkong",


also test \u0003

javeme · 2021-11-23T08:27:25Z

hugegraph-test/src/main/java/com/baidu/hugegraph/core/VertexCoreTest.java

+
+        Vertex vertex = graph.addVertex(T.label, "person", "name",
+                                        "xyz\u0000abc", "city", "Hongkong",
+                                        "age", 15);


what's the behavior if adding a vertex with "name": "\u0000"

javeme · 2021-11-23T09:40:50Z

LGTM

coderzc · 2021-11-23T11:11:11Z

postgresql ci failed

Error:  testQueryBySearchIndexWithSpecialSymbol(com.baidu.hugegraph.core.VertexCoreTest)  Time elapsed: 0.304 s  <<< ERROR!
com.baidu.hugegraph.backend.BackendException: Failed to commit
	at com.baidu.hugegraph.core.VertexCoreTest.testQueryBySearchIndexWithSpecialSymbol(VertexCoreTest.java:8604)
Caused by: java.sql.BatchUpdateException: Batch entry 0 INSERT INTO g_ai (FIELD_VALUES, INDEX_LABEL_ID, ELEMENT_IDS, EXPIRED_TIME) VALUES ('xyz', 276, 'S1454:xyzS1454:xyzabc', 0) ON CONFLICT (FIELD_VALUES, INDEX_LABEL_ID, ELEMENT_IDS) DO UPDATE SET FIELD_VALUES = 'xyz', INDEX_LABEL_ID = 276, ELEMENT_IDS = 'S1454:xyzS1454:xyzabc', EXPIRED_TIME = 0 was aborted: ERROR: invalid byte sequence for encoding "UTF8": 0x00  Call getNextException to see other errors in the batch.
	at com.baidu.hugegraph.core.VertexCoreTest.testQueryBySearchIndexWithSpecialSymbol(VertexCoreTest.java:8604)
Caused by: org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00
	at com.baidu.hugegraph.core.VertexCoreTest.testQueryBySearchIndexWithSpecialSymbol(VertexCoreTest.java:8604)

javeme · 2021-11-24T08:14:49Z

hugegraph-test/src/main/java/com/baidu/hugegraph/core/VertexCoreTest.java

@@ -8590,10 +8590,10 @@ public void testQueryBySearchIndexWithSpecialSymbol() {
             .by("name").search().ifNotExist().create();

        Vertex vertex = graph.addVertex(T.label, "person", "name",
-                                        "xyz\u0000abc", "city", "Hongkong",
+                                        "xyz\u0002abc", "city", "Hongkong",


also add case for "xyz\u0001abc"

javeme · 2021-11-24T08:15:09Z

hugegraph-test/src/main/java/com/baidu/hugegraph/core/VertexCoreTest.java

                                        "age", 15);
        Vertex vertex2 = graph.addVertex(T.label, "person", "name",
-                                         "\u0000", "city", "Hongkong",
+                                         "\u0002", "city", "Hongkong",


also add case for "\u0001"

coderzc · 2021-11-24T08:52:03Z

ci failed

Error:  Tests run: 714, Failures: 0, Errors: 1, Skipped: 13, Time elapsed: 265.664 s <<< FAILURE! - in com.baidu.hugegraph.core.CoreTestSuite
Error:  testQueryBySearchIndexWithSpecialSymbol(com.baidu.hugegraph.core.VertexCoreTest)  Time elapsed: 0.151 s  <<< ERROR!
java.lang.IllegalArgumentException: Illegal value of index property: '�'
	at com.baidu.hugegraph.core.VertexCoreTest.testQueryBySearchIndexWithSpecialSymbol(VertexCoreTest.java:8610)

javeme · 2021-11-26T20:27:11Z

need to rebase master

javeme · 2021-11-29T18:13:08Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/query/ConditionQuery.java

@@ -61,6 +61,15 @@
    public static final String INDEX_VALUE_NULL = new String("<null>");
    public static final String INDEX_VALUE_EMPTY = new String("<empty>");

+    public static final Set<String> IGNORE_SYM_SET;
+    static {
+        List<String> list = new ArrayList<>(INDEX_SYM_MIN - INDEX_SYM_MAX);


error Illegal Capacity: -3, expect INDEX_SYM_MAX - INDEX_SYM_MIN

coderzc force-pushed the fix_segment_words branch from 1779753 to 0f4f1eb Compare November 23, 2021 02:07

javeme previously approved these changes Nov 23, 2021

View reviewed changes

javeme reviewed Nov 23, 2021

View reviewed changes

coderzc dismissed javeme’s stale review via 0a0d5ad November 23, 2021 07:39

javeme reviewed Nov 23, 2021

View reviewed changes

coderzc force-pushed the fix_segment_words branch from 0a0d5ad to e22ccd4 Compare November 23, 2021 09:23

javeme previously approved these changes Nov 23, 2021

View reviewed changes

imbajin previously approved these changes Nov 23, 2021

View reviewed changes

coderzc dismissed stale reviews from imbajin and javeme via 048751e November 24, 2021 03:12

coderzc force-pushed the fix_segment_words branch from 048751e to df41dd0 Compare November 24, 2021 05:13

coderzc requested a review from javeme November 24, 2021 05:55

coderzc force-pushed the fix_segment_words branch 2 times, most recently from e21a453 to e5cc8c1 Compare November 24, 2021 07:55

javeme reviewed Nov 24, 2021

View reviewed changes

coderzc force-pushed the fix_segment_words branch from e5cc8c1 to 5111d0a Compare November 24, 2021 08:44

coderzc added 3 commits November 29, 2021 22:19

fixed #1638, Char covert to String on remove specified unicode

55ceb15

fix test: postgresql not allowed '\0'

9da3f87

rebase master

fc6b4e3

coderzc force-pushed the fix_segment_words branch from 5111d0a to fc6b4e3 Compare November 29, 2021 14:24

javeme reviewed Nov 29, 2021

View reviewed changes

fix code

5189258

javeme approved these changes Nov 30, 2021

View reviewed changes

javeme mentioned this pull request Nov 30, 2021

Exception while inserting special char values like '\\u0000' #1638

Closed

1 task

imbajin approved these changes Dec 1, 2021

View reviewed changes

coderzc merged commit 83180fb into master Dec 1, 2021

javeme deleted the fix_segment_words branch December 28, 2021 13:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: Char covert to String on remove specified unicode #1664

fix: Char covert to String on remove specified unicode #1664

coderzc commented Nov 22, 2021 •

edited by javeme

codecov bot commented Nov 22, 2021 •

edited

javeme Nov 23, 2021

coderzc Nov 23, 2021

javeme Nov 23, 2021

javeme Nov 23, 2021

javeme Nov 23, 2021

javeme Nov 23, 2021

javeme commented Nov 23, 2021

coderzc commented Nov 23, 2021 •

edited

javeme Nov 24, 2021

javeme Nov 24, 2021

coderzc commented Nov 24, 2021

javeme commented Nov 26, 2021

javeme Nov 29, 2021

coderzc Nov 30, 2021

fix: Char covert to String on remove specified unicode #1664

fix: Char covert to String on remove specified unicode #1664

Conversation

coderzc commented Nov 22, 2021 • edited by javeme

codecov bot commented Nov 22, 2021 • edited

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

javeme commented Nov 23, 2021

coderzc commented Nov 23, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coderzc commented Nov 24, 2021

javeme commented Nov 26, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coderzc commented Nov 22, 2021 •

edited by javeme

codecov bot commented Nov 22, 2021 •

edited

coderzc commented Nov 23, 2021 •

edited