Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix hang when closeTaskTx if current thread is task worker #1111

Merged
merged 2 commits into from
Jul 28, 2020
Merged

Fix hang when closeTaskTx if current thread is task worker #1111

merged 2 commits into from
Jul 28, 2020

Conversation

javeme
Copy link
Contributor

@javeme javeme commented Jul 25, 2020 •
edited

related #1075
Change-Id: I5dcde954d846d0c9235ee575749cc7d6fc294e63

@codecov
Copy link

codecov bot commented Jul 27, 2020 •
edited

Codecov Report

Merging #1111 into master will decrease coverage by 7.85%.
The diff coverage is 45.45%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master    #1111      +/-   ##
============================================
- Coverage     66.43%   58.57%   -7.86%     
+ Complexity     5646     4818     -828     
============================================
  Files           347      347              
  Lines         28521    28713     +192     
  Branches       4028     4068      +40     
============================================
- Hits          18947    16819    -2128     
- Misses         7716    10073    +2357     
+ Partials       1858     1821      -37
Impacted Files Coverage Δ Complexity Δ
...ain/java/com/baidu/hugegraph/task/TaskManager.java 71.89% <45.45%> (-2.43%) 25.00 <2.00> (-5.00)
...va/com/baidu/hugegraph/io/HugeGraphIoRegistry.java 0.00% <0.00%> (-100.00%) 0.00% <0.00%> (-4.00%)
...aidu/hugegraph/exception/HugeGremlinException.java 0.00% <0.00%> (-100.00%) 0.00% <0.00%> (-3.00%)
...ugegraph/backend/store/memory/InMemoryMetrics.java 0.00% <0.00%> (-100.00%) 0.00% <0.00%> (-2.00%)
...ava/com/baidu/hugegraph/analyzer/WordAnalyzer.java 0.00% <0.00%> (-92.31%) 0.00% <0.00%> (-4.00%)
.../com/baidu/hugegraph/backend/cache/LevelCache.java 0.00% <0.00%> (-92.11%) 0.00% <0.00%> (-19.00%)
...idu/hugegraph/backend/store/raft/StoreCommand.java 0.00% <0.00%> (-88.24%) 0.00% <0.00%> (-6.00%)
.../com/baidu/hugegraph/analyzer/SmartCNAnalyzer.java 0.00% <0.00%> (-87.50%) 0.00% <0.00%> (-4.00%)
...baidu/hugegraph/plugin/HugeGraphGremlinPlugin.java 0.00% <0.00%> (-86.67%) 0.00% <0.00%> (-4.00%)
...ph/backend/serializer/BinaryScatterSerializer.java 0.00% <0.00%> (-86.21%) 0.00% <0.00%> (-8.00%)
... and 126 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update df1d89b...1f01f8c. Read the comment docs.

Linary
Linary previously approved these changes Jul 27, 2020
View reviewed changes
@javeme
Copy link
Contributor Author

javeme commented Jul 27, 2020 •
edited

init-store error when enabled auth:

2020-07-27 22:13:35 48976 [main] [INFO ] com.baidu.hugegraph.HugeGraph [] - Close graph standardhugegraph[hugegraph]
Exception in thread "main" com.baidu.hugegraph.HugeException: Failed to update/query server info: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Missing authentication context when verifying resource permission
	at com.baidu.hugegraph.task.ServerInfoManager.call(ServerInfoManager.java:320)
	at com.baidu.hugegraph.task.ServerInfoManager.initialized(ServerInfoManager.java:208)
	at com.baidu.hugegraph.task.ServerInfoManager.removeSelfServerInfo(ServerInfoManager.java:341)
	at com.baidu.hugegraph.task.ServerInfoManager.close(ServerInfoManager.java:121)
	at com.baidu.hugegraph.task.StandardTaskScheduler.close(StandardTaskScheduler.java:478)
	at com.baidu.hugegraph.task.TaskManager.closeScheduler(TaskManager.java:112)
	at com.baidu.hugegraph.StandardHugeGraph.close(StandardHugeGraph.java:696)
	at com.baidu.hugegraph.auth.HugeGraphAuthProxy.close(HugeGraphAuthProxy.java:479)
	at com.baidu.hugegraph.auth.StandardAuthenticator.initAdminUser(StandardAuthenticator.java:55)
	at com.baidu.hugegraph.auth.StandardAuthenticator.initAdminUser(StandardAuthenticator.java:122)
	at com.baidu.hugegraph.cmd.InitStore.main(InitStore.java:99)
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Missing authentication context when verifying resource permission
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at com.baidu.hugegraph.task.ServerInfoManager.call(ServerInfoManager.java:318)
	... 10 more
Caused by: java.lang.IllegalStateException: Missing authentication context when verifying resource permission
	at com.google.common.base.Preconditions.checkState(Preconditions.java:531)
	at com.baidu.hugegraph.util.E.checkState(E.java:68)
	at com.baidu.hugegraph.auth.HugeGraphAuthProxy.verifyResPermission(HugeGraphAuthProxy.java:774)
	at com.baidu.hugegraph.auth.HugeGraphAuthProxy.verifyResPermission(HugeGraphAuthProxy.java:765)
	at com.baidu.hugegraph.auth.HugeGraphAuthProxy.verifyPermission(HugeGraphAuthProxy.java:638)
	at com.baidu.hugegraph.auth.HugeGraphAuthProxy.verifyAdminPermission(HugeGraphAuthProxy.java:624)
	at com.baidu.hugegraph.auth.HugeGraphAuthProxy.backendStoreSystemInfo(HugeGraphAuthProxy.java:543)
	at com.baidu.hugegraph.task.ServerInfoManager.lambda$2(ServerInfoManager.java:208)
	at com.baidu.hugegraph.task.TaskManager$ContextCallable.call(TaskManager.java:374)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

@javeme
Copy link
Contributor Author

javeme commented Jul 27, 2020 •
edited

start error after fix init-store error:
(this is because the first time init-store fails, but not init server-info schema due to bug the above, still no schema even set admin pass success after bugfix. clear and re init is ok)

2020-07-28 00:06:34 76567 [task-scheduler-1] [ERROR] com.baidu.hugegraph.task.TaskManager [] - Exception occurred when schedule job
com.baidu.hugegraph.HugeException: Failed to update/query server info: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Undefined vertex label: '~server'
	at com.baidu.hugegraph.task.ServerInfoManager.call(ServerInfoManager.java:320) ~[classes/:?]
	at com.baidu.hugegraph.task.ServerInfoManager.serverInfos(ServerInfoManager.java:383) ~[classes/:?]
	at com.baidu.hugegraph.task.ServerInfoManager.serverInfos(ServerInfoManager.java:378) ~[classes/:?]
	at com.baidu.hugegraph.task.ServerInfoManager.allServerInfos(ServerInfoManager.java:369) ~[classes/:?]
	at com.baidu.hugegraph.task.StandardTaskScheduler.scheduleTasks(StandardTaskScheduler.java:329) ~[classes/:?]
	at com.baidu.hugegraph.task.TaskManager.scheduleOrExecuteJobForGraph(TaskManager.java:329) ~[classes/:?]
	at com.baidu.hugegraph.task.TaskManager.scheduleOrExecuteJob(TaskManager.java:288) ~[classes/:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_111]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_111]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_111]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_111]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Undefined vertex label: '~server'
	at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_111]
	at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_111]
	at com.baidu.hugegraph.task.ServerInfoManager.call(ServerInfoManager.java:318) ~[classes/:?]
	... 13 more
Caused by: java.lang.IllegalArgumentException: Undefined vertex label: '~server'
	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:163) ~[guava-25.1-jre.jar:?]
	at com.baidu.hugegraph.util.E.checkArgument(E.java:56) ~[hugegraph-common-1.7.8.jar:1.7.8.0]
	at com.baidu.hugegraph.StandardHugeGraph.vertexLabel(StandardHugeGraph.java:578) ~[classes/:?]
	at com.baidu.hugegraph.auth.HugeGraphAuthProxy.lambda$2(HugeGraphAuthProxy.java:221) ~[classes/:?]
	at com.baidu.hugegraph.auth.HugeGraphAuthProxy.lambda$23(HugeGraphAuthProxy.java:754) ~[classes/:?]
	at com.baidu.hugegraph.auth.HugeGraphAuthProxy.verifyResPermission(HugeGraphAuthProxy.java:779) ~[classes/:?]
	at com.baidu.hugegraph.auth.HugeGraphAuthProxy.verifyResPermission(HugeGraphAuthProxy.java:765) ~[classes/:?]
	at com.baidu.hugegraph.auth.HugeGraphAuthProxy.verifySchemaPermission(HugeGraphAuthProxy.java:752) ~[classes/:?]
	at com.baidu.hugegraph.auth.HugeGraphAuthProxy.verifySchemaPermission(HugeGraphAuthProxy.java:745) ~[classes/:?]
	at com.baidu.hugegraph.auth.HugeGraphAuthProxy.vertexLabel(HugeGraphAuthProxy.java:220) ~[classes/:?]
	at com.baidu.hugegraph.task.ServerInfoManager.lambda$7(ServerInfoManager.java:390) ~[classes/:?]
	at com.baidu.hugegraph.task.TaskManager$ContextCallable.call(TaskManager.java:374) ~[classes/:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_111]
	... 3 more
2020-07-28 00:06:34 76576 [gremlin-server-stop] [WARN ] org.apache.tinkerpop.gremlin.server.GremlinServer [] - Exception while closing Graph instance [hugegraph]
java.lang.IllegalStateException: Ensure tx closed in all threads when closing graph 'hugegraph'
	at com.google.common.base.Preconditions.checkState(Preconditions.java:531) ~[guava-25.1-jre.jar:?]
	at com.baidu.hugegraph.util.E.checkState(E.java:68) ~[hugegraph-common-1.7.8.jar:1.7.8.0]
	at com.baidu.hugegraph.StandardHugeGraph.close(StandardHugeGraph.java:705) ~[classes/:?]
	at com.baidu.hugegraph.auth.HugeGraphAuthProxy.close(HugeGraphAuthProxy.java:479) ~[classes/:?]
	at org.apache.tinkerpop.gremlin.server.GremlinServer.lambda$null$7(GremlinServer.java:298) ~[gremlin-server-3.4.3.jar:3.4.3]
	at java.util.concurrent.ConcurrentHashMap$KeySetView.forEach(ConcurrentHashMap.java:4649) ~[?:1.8.0_111]
	at org.apache.tinkerpop.gremlin.server.GremlinServer.lambda$stop$8(GremlinServer.java:295) ~[gremlin-server-3.4.3.jar:3.4.3]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]

@javeme
let other thread close task tx if current is task thread
0a8a04e 
Change-Id: Iac11eeeaaaf0be5d01a85496a629ec5b2a387216
@javeme
just call closeTx by one task thread if current is task thread
1f01f8c 
Change-Id: I5dcde954d846d0c9235ee575749cc7d6fc294e63
javeme
javeme commented Jul 28, 2020
View reviewed changes
hugegraph-dist/src/main/java/com/baidu/hugegraph/cmd/InitStore.java Outdated
// Forced set to false when initializing backend
config.setProperty(CoreOptions.RAFT_MODE.name(), "false");
HugeGraph graph = HugeFactory.open(config);
HugeGraph graph = (HugeGraph) GraphFactory.open(config);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

give HugeFactoryAuthProxy a chance to load as soon as possible

@javeme javeme mentioned this pull request Jul 28, 2020
Merged
@zhoney zhoney merged commit b6d16e7 into master Jul 28, 2020
@zhoney zhoney deleted the fix-ap-auth-manager branch July 28, 2020 12:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Linary Linary Linary approved these changes

@zhoney zhoney zhoney approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@javeme @Linary @zhoney