Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Orphan threads block JVM termination #13821

Closed
quentingodeau opened this issue Sep 21, 2018 · 3 comments

Comments

Projects
None yet
5 participants
@quentingodeau
Copy link

commented Sep 21, 2018

Hi,

We are using Hazelcast client version 3.10.3 with a kubernetes discovery service (TODO URL) and we notice an issue when the discoveryService fails to initialize, in this case the JVM does not terminate because it still wait for three threads:

  • hz.client_0.lifecycle-1
  • hz.hz.client_0.MetricsRegistry.thread-1
  • hz.hz.client_0.MetricsRegistry.thread-2

Our analyses is that creating thread inside class constructor can lead, if exception are not handle, to orphan thread with no references on it which give us no mean to kill them.

Bellow a part of the stack

// Hazelcast client creation
HazelcastClient.newHazelcastClient(clientConfig);
...
final HazelcastClientInstanceImpl client = hazelcastClientFactory.createHazelcastInstanceClient(config, clientConnectionManagerFactory);
...
// Create the first thread hz.client_0.lifecycle-1
lifecycleService = new LifecycleServiceImpl(this);
...
// Create the two other threads (MetricsRegistry)
metricsRegistry = initMetricsRegistry();
...
// Finally the code bellow throw an exception and the previous thread become orphan, and thus they can not be deleted by the JVM
discoveryService = initDiscoveryService(config);

Did you ever reproduce this and if so is there a workaround ?


Other information:
JVM: 1.8
Hazelcast client: 3.10.3

stack:

Exception in thread "main" java.lang.RuntimeException: Failed to configure discovery strategies
	at com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.loadDiscoveryStrategies(DefaultDiscoveryService.java:150)
	at com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.<init>(DefaultDiscoveryService.java:57)
	at com.hazelcast.spi.discovery.impl.DefaultDiscoveryServiceProvider.newDiscoveryService(DefaultDiscoveryServiceProvider.java:29)
	at com.hazelcast.client.impl.HazelcastClientInstanceImpl.initDiscoveryService(HazelcastClientInstanceImpl.java:430)
	at com.hazelcast.client.impl.HazelcastClientInstanceImpl.<init>(HazelcastClientInstanceImpl.java:253)
	at com.hazelcast.client.HazelcastClient$1.createHazelcastInstanceClient(HazelcastClient.java:55)
	at com.hazelcast.client.HazelcastClientManager.newHazelcastClient(HazelcastClientManager.java:75)
	at com.hazelcast.client.HazelcastClient.newHazelcastClient(HazelcastClient.java:72)
	at com.example.hazelcast.Main.main(Main.java:21)
Caused by: java.lang.RuntimeException: Could not get token file
	at com.hazelcast.kubernetes.ServiceEndpointResolver.readFileContents(ServiceEndpointResolver.java:169)
	at com.hazelcast.kubernetes.ServiceEndpointResolver.getAccountToken(ServiceEndpointResolver.java:157)
	at com.hazelcast.kubernetes.ServiceEndpointResolver.buildKubernetesClient(ServiceEndpointResolver.java:71)
	at com.hazelcast.kubernetes.ServiceEndpointResolver.<init>(ServiceEndpointResolver.java:66)
	at com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy.<init>(HazelcastKubernetesDiscoveryStrategy.java:79)
	at com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategyFactory.newDiscoveryStrategy(HazelcastKubernetesDiscoveryStrategyFactory.java:57)
	at com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.buildDiscoveryStrategy(DefaultDiscoveryService.java:184)
	at com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.loadDiscoveryStrategies(DefaultDiscoveryService.java:142)
	... 8 more
Caused by: java.io.FileNotFoundException: /var/run/secrets/kubernetes.io/serviceaccount/token (No such file or directory)
	at java.io.FileInputStream.open0(Native Method)
	at java.io.FileInputStream.open(FileInputStream.java:195)
	at java.io.FileInputStream.<init>(FileInputStream.java:138)
	at com.hazelcast.kubernetes.ServiceEndpointResolver.readFileContents(ServiceEndpointResolver.java:165)
	... 15 more
@mmedenjak

This comment has been minimized.

Copy link
Contributor

commented Sep 21, 2018

Hi @quentingodeau !

Can you try running with Hazelcast 3.11-BETA-1? The issue might have been already fixed in the upcoming release.

Possibly related: #13345

@mmedenjak mmedenjak added this to the 3.11 milestone Sep 21, 2018

@quentingodeau

This comment has been minimized.

Copy link
Author

commented Sep 21, 2018

@mmedenjak Thanks for you response ! I just test it but it doesn't solve my issue :(


Also, in my previous message I forget to change the TODO URL to https://github.com/hazelcast/hazelcast-kubernetes

Just in case I add the configuration that I use:

<?xml version="1.0" encoding="UTF-8"?>
<hazelcast-client  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                   xsi:schemaLocation="http://www.hazelcast.com/schema/client-config
                               http://www.hazelcast.com/schema/client-config/hazelcast-client-config-3.11.xsd"
                   xmlns="http://www.hazelcast.com/schema/client-config">
    <properties>
        <property name="hazelcast.discovery.enabled">true</property>
    </properties>
    
    <network>
        <discovery-strategies>
            <discovery-strategy enabled="true"
                                class="com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy">
                <properties>
                    <property name="service-name">${serviceName}</property>
                    <property name="namespace">${namespace}</property>
                </properties>
            </discovery-strategy>
        </discovery-strategies>
    </network>
</hazelcast-client>

@jerrinot jerrinot self-assigned this Sep 24, 2018

@mmedenjak mmedenjak added Team: Client and removed Team: Core labels Oct 2, 2018

@sancar sancar modified the milestones: 3.11, 3.11.1 Oct 5, 2018

@mmedenjak mmedenjak modified the milestones: 3.11.1, 3.11.2 Nov 14, 2018

@mmedenjak mmedenjak modified the milestones: 3.11.2, 3.11.3 Jan 31, 2019

@jerrinot jerrinot removed their assignment Mar 21, 2019

@ihsandemir ihsandemir self-assigned this Mar 21, 2019

@ihsandemir

This comment has been minimized.

Copy link
Contributor

commented Mar 21, 2019

On initial examination, we see that there is no shutdown in case of client instance creation for cleanup of threads. I will try to write a test to test for this.

ihsandemir added a commit to ihsandemir/hazelcast that referenced this issue Mar 25, 2019

Rearranged the client instance thread starts. The constructor should …
…not start any threads, otherwise there may be orphan threads leaking. Moved the thread starts to start or init methods, and recapture the exceptions and do shutdown the client in such a case to avoid any resource leakage.

fixes hazelcast#13821

ihsandemir added a commit to ihsandemir/hazelcast that referenced this issue Mar 26, 2019

Rearranged the client instance thread starts. The constructor should …
…not start any threads, otherwise there may be orphan threads leaking. Moved the thread starts to start or init methods, and recapture the exceptions and do shutdown the client in such a case to avoid any resource leakage.

fixes hazelcast#13821

ihsandemir added a commit to ihsandemir/hazelcast that referenced this issue Mar 26, 2019

Rearranged the client instance thread starts. The constructor should …
…not start any threads, otherwise there may be orphan threads leaking. Moved the thread starts to start or init methods, and recapture the exceptions and do shutdown the client in such a case to avoid any resource leakage.

fixes hazelcast#13821

ihsandemir added a commit to ihsandemir/hazelcast that referenced this issue Mar 27, 2019

Rearranged the client instance thread starts. The constructor should …
…not start any threads, otherwise there may be orphan threads leaking. Moved the thread starts to start or init methods, and recapture the exceptions and do shutdown the client in such a case to avoid any resource leakage.

fixes hazelcast#13821

ihsandemir added a commit to ihsandemir/hazelcast that referenced this issue Mar 27, 2019

Rearranged the client instance thread starts. The constructor should …
…not start any threads, otherwise there may be orphan threads leaking. Moved the thread starts to start or init methods, and recapture the exceptions and do shutdown the client in such a case to avoid any resource leakage.

fixes hazelcast#13821

ihsandemir added a commit that referenced this issue Mar 28, 2019

Client initialization failure should not leave any orphan threads - a…
…void thread leakage (#14762)

* Rearranged the client instance thread starts. The constructor should not start any threads, otherwise there may be orphan threads leaking. Moved the thread starts to start or init methods, and recapture the exceptions and do shutdown the client in such a case to avoid any resource leakage.

fixes #13821

@mmedenjak mmedenjak modified the milestones: 3.11.3, 3.12 Mar 28, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.