Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] [EC]EC tryShutdown will have blocking problems #1765

Closed
2 tasks done
peacewong opened this issue Mar 21, 2022 · 0 comments
Closed
2 tasks done

[Bug] [EC]EC tryShutdown will have blocking problems #1765

peacewong opened this issue Mar 21, 2022 · 0 comments
Labels
bug Something isn't working

Comments

@peacewong
Copy link
Contributor

Search before asking

  • I searched the issues and found no similar issues.

Linkis Component

linkis-cg-engineConnplugin

What happened + What you expected to happen

EC tryShutdown has a blocking problem. When the task is running and triggers shutdown, it will block. The jstack log is as follows:
EC tryShutdown 会存在阻塞问题,当又任务正在运行触发shutdown时会阻塞,jstack日志如下:

SpringContextShutdownHook" #28 prio=5 os_prio=0 tid=0x00007fc16402c800 nid=0x789e waiting for monitor entry [0x00007fc1ed5e0000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.linkis.engineconn.executor.entity.SensibleExecutor$class.transition(SensibleExecutor.scala:36)
        - waiting to lock <0x00000000c19bf838> (a org.apache.linkis.manager.engineplugin.python.executor.PythonEngineConnExecutor)
        at org.apache.linkis.engineconn.acessible.executor.entity.AccessibleExecutor.transition(AccessibleExecutor.scala:27)
        at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor.tryShutdown(ComputationExecutor.scala:97)
        at org.apache.linkis.engineconn.acessible.executor.service.DefaultAccessibleService$$anonfun$executorShutDownHook$1.apply$mcZ$sp(DefaultAccessibleService.scala:83)
        at org.apache.linkis.engineconn.acessible.executor.service.DefaultAccessibleService$$anonfun$executorShutDownHook$1.apply(DefaultAccessibleService.scala:83)
        at org.apache.linkis.engineconn.acessible.executor.service.DefaultAccessibleService$$anonfun$executorShutDownHook$1.apply(DefaultAccessibleService.scala:83)
        at org.apache.linkis.common.utils.Utils$.tryCatch(Utils.scala:40)
        at org.apache.linkis.common.utils.Utils$.tryAndWarn(Utils.scala:69)
        at org.apache.linkis.engineconn.acessible.executor.service.DefaultAccessibleService.executorShutDownHook(DefaultAccessibleService.scala:82)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:305)
        at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:190)
        at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:153)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139)
        at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:404)
        at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:361)
        at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1013)
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.doClose(ServletWebServerApplicationContext.java:170)
        at org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:949)
        - locked <0x00000000c0aef2f0> (a java.lang.Object)

Relevent platform

all

Reproduction script

no

Anything else

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!
@peacewong peacewong added the bug Something isn't working label Mar 21, 2022
peacewong added a commit to WeDataSphere/linkis that referenced this issue Mar 21, 2022
Alexkun pushed a commit that referenced this issue Mar 23, 2022
* add close to shutdown

* Fix issue that EC tryShutdown will have blocking problems #1765

* update HIVE_RESULTSET_USE_UNIQUE_COLUMN_NAMES default value

* add todo msg

* should judge the execute user

* ECMNodeHeartbeatMonitor monitoring ECM should consider heartbeat reporting time #1779
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant