Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix ci too much exception "pk table not exist" #1098

Merged
merged 9 commits into from
Jul 21, 2020

Conversation

zhoney
Copy link
Contributor

@zhoney zhoney commented Jul 16, 2020

fix #1075
Change-Id: Ie9a1ba689ae4aec876bf9793970a1d62e3c69abf

@codecov
Copy link

codecov bot commented Jul 16, 2020

Codecov Report

Merging #1098 into master will increase coverage by 0.07%.
The diff coverage is 97.43%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master    #1098      +/-   ##
============================================
+ Coverage     69.23%   69.30%   +0.07%     
- Complexity     5558     5586      +28     
============================================
  Files           331      331              
  Lines         26970    26997      +27     
  Branches       3861     3865       +4     
============================================
+ Hits          18673    18711      +38     
+ Misses         6463     6457       -6     
+ Partials       1834     1829       -5     
Impacted Files Coverage Δ Complexity Δ
...va/com/baidu/hugegraph/task/ServerInfoManager.java 69.54% <66.66%> (-1.56%) 47.00 <2.00> (+1.00) ⬇️
...in/java/com/baidu/hugegraph/StandardHugeGraph.java 80.40% <100.00%> (+0.30%) 108.00 <0.00> (ø)
...om/baidu/hugegraph/task/StandardTaskScheduler.java 79.39% <100.00%> (+0.05%) 90.00 <1.00> (+1.00)
...ain/java/com/baidu/hugegraph/task/TaskManager.java 74.30% <100.00%> (+0.62%) 29.00 <6.00> (+3.00)
...c/main/java/com/baidu/hugegraph/util/LockUtil.java 84.72% <100.00%> (+0.89%) 17.00 <2.00> (+2.00)
...baidu/hugegraph/exception/ConnectionException.java 33.33% <0.00%> (-33.34%) 1.00% <0.00%> (-1.00%)
.../hugegraph/backend/store/AbstractBackendStore.java 72.22% <0.00%> (-16.67%) 5.00% <0.00%> (-1.00%)
.../com/baidu/hugegraph/backend/BackendException.java 53.84% <0.00%> (-15.39%) 4.00% <0.00%> (-1.00%)
...aidu/hugegraph/backend/tx/AbstractTransaction.java 76.51% <0.00%> (-4.70%) 53.00% <0.00%> (ø%)
...u/hugegraph/backend/store/hbase/HbaseSessions.java 59.93% <0.00%> (-1.81%) 21.00% <0.00%> (ø%)
... and 18 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b266391...e312e7e. Read the comment docs.

houzhizhen
houzhizhen previously approved these changes Jul 17, 2020
this.storeProvider.clear();
} finally {
this.loadGraphStore().close();
this.loadSystemStore().close();
this.loadSchemaStore().close();
LockUtil.unlock(this.name, TaskManager.HEARTBEAT);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move to line 258

@@ -250,11 +252,13 @@ public void clearBackend() {
this.loadSystemStore().open(this.configuration);
this.loadGraphStore().open(this.configuration);
try {
LockUtil.lock(this.name, TaskManager.HEARTBEAT);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move out of try

@@ -231,12 +231,14 @@ public void initBackend() {
this.loadSystemStore().open(this.configuration);
this.loadGraphStore().open(this.configuration);
try {
LockUtil.lock(this.name, TaskManager.HEARTBEAT);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rename to a common name like GRAPH_LOCK

@@ -268,14 +272,15 @@ public void clearBackend() {
public void truncateBackend() {
this.waitUntilAllTasksCompleted();

LockUtil.lock(this.name, TaskManager.GRAPH_LOCK);

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove empty line

@@ -45,7 +45,7 @@

private static final Logger LOG = Log.logger(TaskManager.class);

public static final String HEARTBEAT = "heartbeat";
public static final String GRAPH_LOCK = "graph_lock";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move to LockUtil

Copy link
Contributor

@javeme javeme left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

executeTasksOnWorker

@@ -276,6 +288,11 @@ private void scheduleOrExecuteJob() {
}
} catch (Throwable e) {
LOG.error("Exception occurred when schedule job", e);
} finally {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only do executeTasksOnWorker and cancelTasksOnWorker when !server.master() or server.onlySingleNode(), and rename server to serverManager

Copy link
Contributor

@javeme javeme left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The following query is triggered each time:

2020-07-18 13:33:26 2589143 [server-info-db-worker-1] [INFO ] com.baidu.hugegraph.backend.Transaction [] - Transaction query: `Query * from VERTEX where id in [server1]`
2020-07-18 13:33:26 2589146 [server-info-db-worker-1] [INFO ] com.baidu.hugegraph.backend.Transaction [] - Transaction query: `Query * from VERTEX where [LABEL == -38]`
2020-07-18 13:33:26 2589149 [task-db-worker-1] [INFO ] com.baidu.hugegraph.backend.Transaction [] - Transaction query: `Query * from SECONDARY_INDEX page '', limit 2 where [INDEX_LABEL_ID == -54, FIELD_VALUES == 12]`
2020-07-18 13:33:26 2589150 [task-db-worker-1] [INFO ] com.baidu.hugegraph.backend.Transaction [] - Transaction query: `Query * from SECONDARY_INDEX page '', limit 2 where [INDEX_LABEL_ID == -54, FIELD_VALUES == 13]`
2020-07-18 13:33:26 2589152 [task-db-worker-1] [INFO ] com.baidu.hugegraph.backend.Transaction [] - Transaction query: `Query * from SECONDARY_INDEX page '', limit 2 where [INDEX_LABEL_ID == -54, FIELD_VALUES == 18]`

@zhoney
Copy link
Contributor Author

zhoney commented Jul 21, 2020

The following query is triggered each time:

2020-07-18 13:33:26 2589143 [server-info-db-worker-1] [INFO ] com.baidu.hugegraph.backend.Transaction [] - Transaction query: `Query * from VERTEX where id in [server1]`
2020-07-18 13:33:26 2589146 [server-info-db-worker-1] [INFO ] com.baidu.hugegraph.backend.Transaction [] - Transaction query: `Query * from VERTEX where [LABEL == -38]`
2020-07-18 13:33:26 2589149 [task-db-worker-1] [INFO ] com.baidu.hugegraph.backend.Transaction [] - Transaction query: `Query * from SECONDARY_INDEX page '', limit 2 where [INDEX_LABEL_ID == -54, FIELD_VALUES == 12]`
2020-07-18 13:33:26 2589150 [task-db-worker-1] [INFO ] com.baidu.hugegraph.backend.Transaction [] - Transaction query: `Query * from SECONDARY_INDEX page '', limit 2 where [INDEX_LABEL_ID == -54, FIELD_VALUES == 13]`
2020-07-18 13:33:26 2589152 [task-db-worker-1] [INFO ] com.baidu.hugegraph.backend.Transaction [] - Transaction query: `Query * from SECONDARY_INDEX page '', limit 2 where [INDEX_LABEL_ID == -54, FIELD_VALUES == 18]`

this is triggered by serverStarted():

  1. first two queries is for server not exist check
  2. last three queries is for restoring tasks

Change-Id: Ie9a1ba689ae4aec876bf9793970a1d62e3c69abf
Change-Id: Ifbaa77b45177c668b61b90eeca5b436ea970060f
Change-Id: I09dfb1a993ae382e590e2116ead6bcb14b3ebce0
Change-Id: I3c4e3330c8b9fb916f3fe1db591cbb74950bb6af
Change-Id: I6222493175e1c3438eb6023149ca598b8d583b7f
Change-Id: I30149b43146e0e82f775357181c5455928ae463a
@zhoney zhoney force-pushed the fix-ci-pk-table-not-exist branch from 94672c0 to 98d8fe0 Compare July 21, 2020 02:28
@javeme
Copy link
Contributor

javeme commented Jul 21, 2020

cassandra:

[ERROR] Tests run: 673, Failures: 2, Errors: 1, Skipped: 16, Time elapsed: 495.409 s <<< FAILURE! - in com.baidu.hugegraph.core.CoreTestSuite
[ERROR] testCreateGraphsWithDifferentNameDifferentBackends(com.baidu.hugegraph.core.MultiGraphsTest)  Time elapsed: 4.482 s  <<< FAILURE!
java.lang.AssertionError: java.lang.IllegalStateException: Ensure tx closed in all threads when closing graph 'g1'
	at com.baidu.hugegraph.core.MultiGraphsTest.destoryGraphs(MultiGraphsTest.java:344)
	at com.baidu.hugegraph.core.MultiGraphsTest.testCreateGraphsWithDifferentNameDifferentBackends(MultiGraphsTest.java:299)
[ERROR] testCreateGraphsWithMultiDisksForRocksDB(com.baidu.hugegraph.core.MultiGraphsTest)  Time elapsed: 0.797 s  <<< FAILURE!
java.lang.AssertionError: No exception was thrown(expected com.baidu.hugegraph.backend.BackendException)
	at com.baidu.hugegraph.core.MultiGraphsTest.testCreateGraphsWithMultiDisksForRocksDB(MultiGraphsTest.java:313)
[ERROR] testCreateGraphWithSameNameDifferentBackends(com.baidu.hugegraph.core.MultiGraphsTest)  Time elapsed: 0.006 s  <<< ERROR!
java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [class com.baidu.hugegraph.HugeFactory]
	at com.baidu.hugegraph.core.MultiGraphsTest.openGraphWithBackend(MultiGraphsTest.java:365)
	at com.baidu.hugegraph.core.MultiGraphsTest.testCreateGraphWithSameNameDifferentBackends(MultiGraphsTest.java:277)
Caused by: java.lang.reflect.InvocationTargetException
	at com.baidu.hugegraph.core.MultiGraphsTest.openGraphWithBackend(MultiGraphsTest.java:365)
	at com.baidu.hugegraph.core.MultiGraphsTest.testCreateGraphWithSameNameDifferentBackends(MultiGraphsTest.java:277)
Caused by: java.lang.IllegalStateException: Graph name 'graph' has been used by backend 'cassandra'
	at com.baidu.hugegraph.core.MultiGraphsTest.openGraphWithBackend(MultiGraphsTest.java:365)
	at com.baidu.hugegraph.core.MultiGraphsTest.testCreateGraphWithSameNameDifferentBackends(MultiGraphsTest.java:277)

hbase:

[ERROR] Tests run: 673, Failures: 2, Errors: 1, Skipped: 13, Time elapsed: 884.3 s <<< FAILURE! - in com.baidu.hugegraph.core.CoreTestSuite
[ERROR] testCreateGraphsWithDifferentNameDifferentBackends(com.baidu.hugegraph.core.MultiGraphsTest)  Time elapsed: 52.635 s  <<< FAILURE!
java.lang.AssertionError: java.lang.IllegalStateException: Ensure tx closed in all threads when closing graph 'g1'
	at com.baidu.hugegraph.core.MultiGraphsTest.destoryGraphs(MultiGraphsTest.java:344)
	at com.baidu.hugegraph.core.MultiGraphsTest.testCreateGraphsWithDifferentNameDifferentBackends(MultiGraphsTest.java:299)
[ERROR] testCreateGraphsWithMultiDisksForRocksDB(com.baidu.hugegraph.core.MultiGraphsTest)  Time elapsed: 0.792 s  <<< FAILURE!
java.lang.AssertionError: No exception was thrown(expected com.baidu.hugegraph.backend.BackendException)
	at com.baidu.hugegraph.core.MultiGraphsTest.testCreateGraphsWithMultiDisksForRocksDB(MultiGraphsTest.java:313)
[ERROR] testCreateGraphWithSameNameDifferentBackends(com.baidu.hugegraph.core.MultiGraphsTest)  Time elapsed: 0.009 s  <<< ERROR!
java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [class com.baidu.hugegraph.HugeFactory]
	at com.baidu.hugegraph.core.MultiGraphsTest.openGraphWithBackend(MultiGraphsTest.java:365)
	at com.baidu.hugegraph.core.MultiGraphsTest.testCreateGraphWithSameNameDifferentBackends(MultiGraphsTest.java:277)
Caused by: java.lang.reflect.InvocationTargetException
	at com.baidu.hugegraph.core.MultiGraphsTest.openGraphWithBackend(MultiGraphsTest.java:365)
	at com.baidu.hugegraph.core.MultiGraphsTest.testCreateGraphWithSameNameDifferentBackends(MultiGraphsTest.java:277)
Caused by: java.lang.IllegalStateException: Graph name 'graph' has been used by backend 'hbase'
	at com.baidu.hugegraph.core.MultiGraphsTest.openGraphWithBackend(MultiGraphsTest.java:365)
	at com.baidu.hugegraph.core.MultiGraphsTest.testCreateGraphWithSameNameDifferentBackends(MultiGraphsTest.java:277)

ServerInfoManager server = scheduler.serverManager();
ServerInfoManager serverManager = scheduler.serverManager();

String graph = scheduler.graph().name();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add graphName() to call graphparams.name() to avoid auth issue

Change-Id: I4fed921ad3519e9ad8ecdc2df265eb568ada832d
Change-Id: Ide2df71f51ccd9ab381db371c0297450b1ebe3bc
@@ -94,7 +94,7 @@ public void addScheduler(HugeGraphParams graph) {
this.schedulers.put(graph, scheduler);
}

public void closeScheduler(HugeGraphParams graph) {
public synchronized void closeScheduler(HugeGraphParams graph) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add some comments

Change-Id: I01caba3f1c6fe9ebef6b948b4b90c367e272e026
@Linary Linary merged commit 125b2ef into master Jul 21, 2020
@Linary Linary deleted the fix-ci-pk-table-not-exist branch July 21, 2020 11:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

server-info-db-worker: Invalid column family specified in write batch
4 participants