Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to connect to any address in the config #11288

Closed
abhinavsinha09 opened this issue Sep 5, 2017 · 8 comments
Closed

Unable to connect to any address in the config #11288

abhinavsinha09 opened this issue Sep 5, 2017 · 8 comments

Comments

@abhinavsinha09
Copy link

@abhinavsinha09 abhinavsinha09 commented Sep 5, 2017

Hi All,

I'm using Hazelcast 3.8 and the issue occurs in Production after 4-5 hours of continuous execution.
I created 4 IMaps to cache the values and it worked fine for 4-5 hours. But after 5 hours the client was not able to connect to the cache. I even reproduced this in lower environment with the use of Jmeter.

What I can make out is that it is occurring only when the client is invoked continuously for 4 hours. If I give a break like front user trying to invoke client and there is no delta load/initial load occurring then it works fine.

Request your help.

Error:

Error while creating Hazelcast client: Unable to connect to any address in the config! The following addresses were tried: [localhost/127.0.0.1:5703, localhost/127.0.0.1:5702, localhost/127.0.0.1:5701]
Hazelcast.xml is configured like below:
<?xml version="1.0" encoding="UTF-8"?>
<hazelcast
	xsi:schemaLocation="http://www.hazelcast.com/schema/config hazelcast-config-3.8.xsd"
	xmlns="http://www.hazelcast.com/schema/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

	<properties>
		<property name="hazelcast.health.monitoring.level">OFF</property>
		<property name="hazelcast.logging.type">none</property>
		<property name="hazelcast.property.foo">value</property>
		<property name="hazelcast.health.monitoring.level">OFF</property>
		<property name="hazelcast.client.max.no.heartbeat.seconds">5000</property>
		<property name="hazelcast.client.heartbeat.interval">5000</property>
		<property name="hazelcast.client.heartbeat.timeout">60000</property>
		<property name="hazelcast.client.event.thread.count">5</property>
               <property name="hazelcast.client.event.queue.capacity">1000000</property>
                <property name="hazelcast.client.invocation.timeout.seconds">120</property>
		<property name="hazelcast.operation.priority.generic.thread.count">1</property>
	</properties>
	<management-center enabled="false" />
	<network>
		<port auto-increment="true">5701</port>
		<join>
			<multicast enabled="false">
				<multicast-group>224.2.2.3</multicast-group>
				<multicast-port>54327</multicast-port>
			</multicast>
			<tcp-ip enabled="true">
				<interface>localhost</interface>
			</tcp-ip>
			<aws enabled="false" />
		</join>
		<interfaces enabled="true">
			<interface>127.0.0.*</interface>       
		</interfaces>

	</network>

	<map name="usergroupMap">
		<backup-count>0</backup-count>
		<async-backup-count>1</async-backup-count>
		<time-to-live-seconds>21600</time-to-live-seconds>
		<max-idle-seconds>21600</max-idle-seconds>
		<eviction-policy>NONE</eviction-policy>
		<eviction-percentage>25</eviction-percentage>
		<merge-policy>com.hazelcast.map.merge.PassThroughMergePolicy
		</merge-policy>
	</map>
	<map name="jurisdictionMap">
		<backup-count>0</backup-count>
		<async-backup-count>1</async-backup-count>
		<time-to-live-seconds>21600</time-to-live-seconds>
		<max-idle-seconds>21600</max-idle-seconds>
		<eviction-policy>NONE</eviction-policy>
		<eviction-percentage>25</eviction-percentage>
		<merge-policy>com.hazelcast.map.merge.PassThroughMergePolicy
		</merge-policy>
	</map>
	<map name="restrictedPartyMap">
		<backup-count>0</backup-count>
		<async-backup-count>1</async-backup-count>
		<time-to-live-seconds>21600</time-to-live-seconds>
		<max-idle-seconds>21600</max-idle-seconds>
		<eviction-policy>NONE</eviction-policy>
		<eviction-percentage>25</eviction-percentage>
		<merge-policy>com.hazelcast.map.merge.PassThroughMergePolicy
		</merge-policy>
	</map>
	<map name="restrictedSourceMap">
		<backup-count>0</backup-count>
		<async-backup-count>1</async-backup-count>
		<time-to-live-seconds>21600</time-to-live-seconds>
		<max-idle-seconds>21600</max-idle-seconds>
		<eviction-policy>NONE</eviction-policy>
		<eviction-percentage>25</eviction-percentage>
		<merge-policy>com.hazelcast.map.merge.PassThroughMergePolicy
		</merge-policy>
	</map>
</hazelcast>

Root Cause - I've set the time to live seconds as 21600 (6 hours). At 21600 seconds when my cache is automatically getting refreshed and at the same time if client is invoked by multiple threads then cache is not able to create and result in error:

[9/1/17 12:01:06:335 AEST] 000149d4 ClusterListen W com.hazelcast.client.spi.impl.ClusterListenerSupport hz.client_6615 [dev] [3.8] Unable to get alive cluster connection, try in 2998 ms later, attempt 2 of 2.

I was able to reproduce this scenario in non-prod env. by setting the below configuration:
Number of threads = 80

Below is the code:

static class UserGroupListener implements
EntryEvictedListener<String, String> {

    @Override
    public void entryEvicted(EntryEvent<String, String> arg0) {
        try {
            IMap<String, ArrayList<String>> mapping = instance
                    .getMap("usergroupMap");
            createUserGroupIMap(mapping);
        } catch (IOException | NamingException | SQLException e) {
            log.error("User Group Map - entryEvicted] "
                    + "Error while re-creating maps: " + e.getMessage());
        }
    }
}

Can someone please let me know how to fix this issue permanently.

@mmedenjak
Copy link
Contributor

@mmedenjak mmedenjak commented Sep 5, 2017

Hi @abhinavsinha09 !
Is this issue different than this one : #11194?

@abhinavsinha09
Copy link
Author

@abhinavsinha09 abhinavsinha09 commented Sep 5, 2017

No, it's the same issue but I found the root cause now. Need a solution to fix it.

@mmedenjak
Copy link
Contributor

@mmedenjak mmedenjak commented Sep 5, 2017

Great, can you please close either of the issues then so we don't have duplicates. Thanks!

@mmedenjak mmedenjak added this to the 3.8.6 milestone Sep 5, 2017
@abhinavsinha09
Copy link
Author

@abhinavsinha09 abhinavsinha09 commented Sep 6, 2017

Any help is much appreciated, please look into this issue.

@sancar
Copy link
Member

@sancar sancar commented Sep 7, 2017

Hi @abhinavsinha09 ,
I don't really see a connection with time to live second and unable to get alive cluster connection. Can you send a small full application for us to run and reproduce the problem? And preferably not something that reproduce in 6 hours but in seconds.

@ihsandemir ihsandemir modified the milestones: 3.8.6, 3.8.7 Sep 14, 2017
@mdogan
Copy link
Contributor

@mdogan mdogan commented Sep 14, 2017

@abhinavsinha09:
Can you please send the logs from servers and the client including the time span between when connection is healthy and when you got the failure?

@sancar
Copy link
Member

@sancar sancar commented Oct 3, 2017

@abhinavsinha09
Closing the issue, because we could not hear from you.
If you can provide logs or a way to reproduce the issue, don't hesitate to write to this issue. We will look into it.

@sancar sancar closed this Oct 3, 2017
@philnate
Copy link

@philnate philnate commented Jan 17, 2018

It kind of surprises me that this ticket is listed as bugfixed with 3.8.7, while there was actually no code change and the issue was only closed due to lack of feedback. If this is a "configuration issue" or no real issue it shouldn't appear in the changelog or if it's indeed an issue it can't simply be considered fixed without at least referencing something changed attributed to fix it. Could you please adjust the changelog for 3.8.7, @sancar?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants
You can’t perform that action at this time.