Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hazelcast Client APIs (3.7.3) not able to execute get/put/delete operations on maps when used in AWS with smart-routing enabled where as values & clear operation works #9419

Closed
AjitDas opened this issue Dec 10, 2016 · 11 comments

Comments

@AjitDas
Copy link

@AjitDas AjitDas commented Dec 10, 2016

I am using hazelcast 3.7.3 both server & client APIs with AWS discovery, on both server & client side I am able to identify members. Server side all the operations put/get/delete works fine but on client side when it tries to do put operation it fails when uses get()/put()/delete() operations on map complaining that it can't open a connection where as values() and clear() operations works just fine. So I can safely conclude that it's not a connectivity issue for sure as I can see the members in the log in client application. It fails only when client tries to do some operations on the map.

I did some googling and people have recommended of many things like setting enabled=true, inside-aws=true etc but everything I have configured looks just right as per documentation. I have tested both with configuration and programmatic way but same result.

This looks like a bug to me, can someone please take a quick look at it and let me know what's wrong.

<beans xmlns="http://www.springframework.org/schema/beans"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:context="http://www.springframework.org/schema/context"
	xmlns:hz="http://www.hazelcast.com/schema/spring"
	xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
              http://www.hazelcast.com/schema/spring http://www.hazelcast.com/schema/spring/hazelcast-spring.xsd">

	<hz:client id="sampleHazelcastClient">

		<hz:group name="${sample.hz.group.name}" password="${sample.hz.group.pwd}" />

		<hz:properties>
			<hz:property name="hazelcast.socket.bind.any">${sample.hz.socket.bind.any:true}</hz:property>
			<hz:property name="hazelcast.socket.client.bind.any">${sample.hz.socket.client.bind.any:true}</hz:property>

			<hz:property name="hazelcast.client.shuffle.member.list">${sample.hz.client.shuffle.member.list:true}</hz:property>
			<hz:property name="hazelcast.client.heartbeat.timeout">${sample.hz.client.heartbeat.timeout:60000}</hz:property>
			<hz:property name="hazelcast.client.heartbeat.interval">${sample.hz.client.heartbeat.interval:5000}</hz:property>
			<hz:property name="hazelcast.client.event.thread.count">${sample.hz.client.event.thread.count:5}</hz:property>
			<hz:property name="hazelcast.client.event.queue.capacity">${sample.hz.client.event.queue.capacity:1000000}</hz:property>
			<hz:property name="hazelcast.client.invocation.timeout.seconds">${sample.hz.client.invocation.timeout.seconds:120}</hz:property>
			<hz:property name="hazelcast.logging.type">${sample.hz.logging.type:slf4j}</hz:property>
		</hz:properties>

		<hz:network connection-attempt-limit="${sample.hz.client.connection.attempt.limit}"
			connection-attempt-period="${sample.hz.client.connection.attempt.period}"
			connection-timeout="${sample.hz.client.connection.timeout}"
			redo-operation="true" smart-routing="true">
            	
			<hz:aws enabled="${sample.hz.aws.enabled:true}"
				inside-aws="true"
				access-key="${sample.hz.aws.access.key}"
				secret-key="${sample.hz.aws.secret.key}" 
				iam-role="${sample.hz.aws.iam.role}"
				region="${sample.hz.aws.region:us-east-1}" 
				host-header="${sample.hz.aws.host.header:ec2.amazonaws.com}"
				security-group-name="${sample.hz.aws.security.group.name}" 
				tag-key="${sample.hz.aws.tag.key}"
				tag-value="${sample.hz.aws.tag.value}" 
				connection-timeout-seconds="${sample.hz.aws.conn.timeout:30}" />

		</hz:network>
	</hz:client>

</beans>

In client code HazelcastInstance on client is auto wired as a spring bean and

	@Autowired @Qualifier("sampleHazelcastClient")
	private HazelcastInstance hazelcastClient;

        public void someMethod(){
               // DO SOME MAP UPDATES USING 'hazelcastClient' spring bean auto wired 

        }

Also with programmatic approach

               ClientConfig clientConfig = new ClientConfig();
               ClientNetworkConfig clientNetworkConfig = new ClientNetworkConfig();
	       ClientAwsConfig config =new ClientAwsConfig(); 
               config.setEnabled(true);
               config.setInsideAws(true);

               PropertiesReaderUtil propsReader = PropertiesReaderUtil.getInstance();

		String secKey = propsReader.getAsString("sample.hz.aws.secret.key", null);
		if(secKey!=null && !secKey.isEmpty()){
			config.setSecretKey(secKey) ;
		}

		String accKey = propsReader.getAsString("sample.hz.aws.access.key", null);
		if(accKey !=null && !accKey.isEmpty()){
			config.setAccessKey(accKey);
		}

		String iAMRole = propsReader.getAsString("sample.hz.aws.iam.role", null);
		if(iAMRole !=null && !iAMRole.isEmpty()){
			config.setIamRole(iAMRole);
		}

		String region = propsReader.getAsString("sample.hz.aws.region", "us-east-1");
		config.setRegion(region);

		String hostHeader = propsReader.getAsString("sample.hz.aws.host.header", null);
		if(hostHeader !=null && !hostHeader.isEmpty()){
			config.setHostHeader(hostHeader);
		}

		String secGrpName = propsReader.getAsString("sample.hz.aws.security.group.name", null);
		if(secGrpName !=null && !secGrpName.isEmpty()){
			config.setSecurityGroupName(secGrpName);
		}

		String tagKey = propsReader.getAsString("sample.hz.aws.tag.key", null);
		if(tagKey!=null && !tagKey.isEmpty()){
			config.setTagKey(tagKey);
		}

		String tagVal =propsReader.getAsString("sample.hz.aws.tag.value", null);
		if(tagVal !=null && !tagVal.isEmpty()){
			config.setTagValue(tagVal);
		}

		int connTimeout =propsReader.getAsInteger("sample.hz.aws.conn.timeout", 30);
		config.setConnectionTimeoutSeconds(connTimeout);
		
		GroupConfig groupConfig = new GroupConfig();
		String grpName = propsReader.getAsString("sample.hz.group.name", "dev");
		groupConfig.setName(grpName);
		
		String grpPwd = propsReader.getAsString("sample.hz.group.pwd", "dev-pass");
		groupConfig.setPassword(grpPwd);
		
		clientConfig.setGroupConfig(groupConfig);
		clientConfig.setNetworkConfig(clientNetworkConfig.setAwsConfig(awsConfig));

               HazelcastInstance hazelcastInstance = HazelcastClient.newHazelcastClient(clientConfig);
              
               // DO SOME MAP UPDATES HERE

Here is the error that's thrown while doing the map updates from client APIs ( NOTE : it gives error for client on get()/put()/delete() where as values()/clear() works well, all operations on server side instance works perfectly fine though)

java.io.IOException: No available connection to address [10.1.50.168]:5701 "}
com.hazelcast.core.HazelcastException: java.io.IOException: No available connection to address [10.1.50.168]:5701
        at com.hazelcast.util.ExceptionUtil.peel(ExceptionUtil.java:73) ~[hazelcast-all-3.7.3.jar:3.7.3]
        at com.hazelcast.util.ExceptionUtil.peel(ExceptionUtil.java:63) ~[hazelcast-all-3.7.3.jar:3.7.3]
        at com.hazelcast.util.ExceptionUtil.peel(ExceptionUtil.java:52) ~[hazelcast-all-3.7.3.jar:3.7.3]
        at com.hazelcast.util.ExceptionUtil.rethrow(ExceptionUtil.java:83) ~[hazelcast-all-3.7.3.jar:3.7.3]
        at com.hazelcast.client.spi.ClientProxy.invokeOnPartition(ClientProxy.java:155) ~[hazelcast-all-3.7.3.jar:3.7.3]
        at com.hazelcast.client.spi.ClientProxy.invoke(ClientProxy.java:147) ~[hazelcast-all-3.7.3.jar:3.7.3]
        at com.hazelcast.client.proxy.ClientMapProxy.putInternal(ClientMapProxy.java:457) ~[hazelcast-all-3.7.3.jar:3.7.3]
        at com.hazelcast.client.proxy.ClientMapProxy.put(ClientMapProxy.java:451) ~[hazelcast-all-3.7.3.jar:3.7.3]
        at com.hazelcast.client.proxy.ClientMapProxy.put(ClientMapProxy.java:253) ~[hazelcast-all-3.7.3.jar:3.7.3]


Caused by: java.io.IOException: No available connection to address [10.1.50.168]:5701
        at com.hazelcast.client.spi.impl.ClientSmartInvocationServiceImpl.getOrTriggerConnect(ClientSmartInvocationServiceImpl.java:87) ~[hazelcast-all-3.7.3.jar:3.7.3]
        at com.hazelcast.client.spi.impl.ClientSmartInvocationServiceImpl.invokeOnPartitionOwner(ClientSmartInvocationServiceImpl.java:46) ~[hazelcast-all-3.7.3.jar:3.7.3]
        at com.hazelcast.client.spi.impl.ClientInvocation.invokeOnSelection(ClientInvocation.java:142) ~[hazelcast-all-3.7.3.jar:3.7.3]
        at com.hazelcast.client.spi.impl.ClientInvocation.invoke(ClientInvocation.java:122) ~[hazelcast-all-3.7.3.jar:3.7.3]
        at com.hazelcast.client.spi.impl.ClientInvocation.run(ClientInvocation.java:153) ~[hazelcast-all-3.7.3.jar:3.7.3]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_102]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_102]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_102]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_102]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_102]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_102]
        at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_102]
        at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) ~[hazelcast-all-3.7.3.jar:3.7.3]
        at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:92) ~[hazelcast-all-3.7.3.jar:3.7.3]
        at ------ submitted from ------.(Unknown Source) ~[?:?]
        at com.hazelcast.client.spi.impl.ClientInvocationFuture.resolveAndThrow(ClientInvocationFuture.java:74) ~[hazelcast-all-3.7.3.jar:3.7.3]
        at com.hazelcast.client.spi.impl.ClientInvocationFuture.resolveAndThrow(ClientInvocationFuture.java:30) ~[hazelcast-all-3.7.3.jar:3.7.3]
        at com.hazelcast.spi.impl.AbstractInvocationFuture.get(AbstractInvocationFuture.java:158) ~[hazelcast-all-3.7.3.jar:3.7.3]
        at com.hazelcast.client.spi.ClientProxy.invokeOnPartition(ClientProxy.java:153) ~[hazelcast-all-3.7.3.jar:3.7.3]
        ... 56 more

@jerrinot
Copy link
Contributor

@jerrinot jerrinot commented Dec 11, 2016

hi @AjitDas,

can you try it with 3.7.4? There was a possible related bugfix.

@AjitDas
Copy link
Author

@AjitDas AjitDas commented Dec 12, 2016

@jerrinot Hi Jeromir - as I mentioned this is not a issue of connectivity as get and delete operation are working fine only put operation not working with this error.

I have not tried in version 3.7.4 though, I will give a try and let you know how does it go.

@AjitDas
Copy link
Author

@AjitDas AjitDas commented Dec 12, 2016

@jerrinot : tried with 3.7.4 also it's giving same error. One correction to my earlier observation none of the operations like get/put/delete works, it works only for operation like values() & clear().

This is a very critical issue and it's blocking clients to be able to any of the operation on Map, please take a look at it, let me know if you need further infos and logs.

@AjitDas AjitDas changed the title Hazelcast Client APIs (3.7.3) not able to write to maps when used in AWS where as get & delete operation works Hazelcast Client APIs (3.7.3) not able to execute get/put/delete operations on maps when used in AWS where as values & clear operation works Dec 12, 2016
@AjitDas
Copy link
Author

@AjitDas AjitDas commented Dec 13, 2016

@bilalyasar Hi Bilal - do you know anything about this issue ? as you had fixed something related to this, can you please help me out here, am stuck and need a solution to move ahead with hazelcast APIs.

I can clearly see from my hazelcast client that it identifies the members with the hazelcast public address (which are nothing but private IP of my EC2 instances) which means it can discover via AWS client APIs and it can do certain operations like values(), clear() etc but when am trying to access get(), put() or delete() on the Map instance it gives me this error.

Logs from hazelcast client which says that it is connected to the hazelcast cluster , NOTE : here 10.1.46.31 & 10.1.50.168 are private IPs of my EC2 instance where my docker containers are running.

12-Dec-2016 22:26:25.943 INFO [localhost-startStop-1] com.hazelcast.core.LifecycleService.null hz.client_1 [local-sample-hazelcast] [3.7.4] HazelcastClient 3.7.4 (20161209 - 3df1bb5) is STARTING
12-Dec-2016 22:26:25.980 INFO [localhost-startStop-1] com.hazelcast.core.LifecycleService.null hz.client_1 [local-sample-hazelcast] [3.7.4] HazelcastClient 3.7.4 (20161209 - 3df1bb5) is STARTED
12-Dec-2016 22:26:29.459 INFO [hz.client_1.internal-1] com.hazelcast.client.connection.ClientConnectionManager.null hz.client_1 [local-sample-hazelcast] [3.7.4] Authenticated with server [10.1.46.31]:5701, server version:3.7.4 Local address: /172.17.0.4:45357
12-Dec-2016 22:26:29.468 INFO [hz.client_1.event-9] com.hazelcast.client.spi.impl.ClientMembershipListener.null hz.client_1 [local-sample-hazelcast] [3.7.4]

Members [2] {
        Member [10.1.46.31]:5701 - 8a8904d9-4386-4c84-9e2d-9342911d0d08
        Member [10.1.50.168]:5701 - 896213ef-1e41-445d-87d4-ef9d073a7c19
}

12-Dec-2016 22:26:29.468 INFO [localhost-startStop-1] com.hazelcast.core.LifecycleService.null hz.client_1 [local-sample-hazelcast] [3.7.4] HazelcastClient 3.7.4 (20161209 - 3df1bb5) is CLIENT_CONNECTED
@AjitDas
Copy link
Author

@AjitDas AjitDas commented Dec 13, 2016

After setting smart-routing="false" in hz:network section now am able to access it properly.

This means that it will connect to any one member and then internally identifies the partition which might be less performant that searching for key with correct hazelcast member node. To me this looks like a bug in the code somewhere, I can try with older versions to see if this issue is introduced in recent versions or it's a bug from inception.

<hz:network connection-attempt-limit="${sample.hz.client.connection.attempt.limit}"
			connection-attempt-period="${sample.hz.client.connection.attempt.period}"
			connection-timeout="${sample.hz.client.connection.timeout}"
			redo-operation="true" smart-routing="false">
@AjitDas AjitDas changed the title Hazelcast Client APIs (3.7.3) not able to execute get/put/delete operations on maps when used in AWS where as values & clear operation works Hazelcast Client APIs (3.7.3) not able to execute get/put/delete operations on maps when used in AWS with smart-routing enabled where as values & clear operation works Dec 13, 2016
@bilalyasar
Copy link
Contributor

@bilalyasar bilalyasar commented Dec 16, 2016

@AjitDas
Do your clients are inside AWS, right?
If it is possible, can you please try with tcp/ip configuration rather that AWS discovery? Seems this is not related to AWS discovery. Because it finds ip addresses but couldn't connect them somehow.

In tcp/ip scenario, if problem continues then we can sure about that it is not aws discovery problem...

@AjitDas
Copy link
Author

@AjitDas AjitDas commented Dec 20, 2016

@bilalyasar : my client is inside AWS, and it's not a discovery problem because after the client starts up I can see all the members in logs and I can even get the members details programetically. I could retrieve values() and do clear() operations from client only get(), put() & delete() is not working , as I mentioned in my earlier note when I disabled the smart routing with 'false' then it started working with all operations.

@bilalyasar
Copy link
Contributor

@bilalyasar bilalyasar commented Dec 27, 2016

@AjitDas, seems client couldn't connect Member of [10.1.50.168]:5701.
Can you please use tcp/ip connection rather than aws discovery and try same scenario?

@AjitDas
Copy link
Author

@AjitDas AjitDas commented Jan 12, 2017

@bilalyasar - as I mentioned earlier it's not an issue with connective when smart routing is disabled (i.e set to false) on client side, only when it's enabled it's not able to connect.

@sancar sancar added this to the 3.9 milestone Aug 1, 2017
@mesutcelik mesutcelik self-assigned this Aug 21, 2017
@mesutcelik
Copy link
Contributor

@mesutcelik mesutcelik commented Aug 21, 2017

Hi @AjitDas
Is It possible for you to test with hazelcast 3.8.4? Are you still having issues?

@mesutcelik
Copy link
Contributor

@mesutcelik mesutcelik commented Aug 29, 2017

Hi,

I am closing this issue. please feel free to reopen if you still see the problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants
You can’t perform that action at this time.