Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spring cloud function on aws getting connection refused exception from netty #367

Open
kleung opened this issue May 19, 2019 · 8 comments

Comments

Projects
None yet
3 participants
@kleung
Copy link

commented May 19, 2019

Using version 2.1.0.RELEASE of spring-cloud-function-(core, context, adapter-aws, webflux and web) cause the following stack trace when deployed on aws:

NOTE: I tried both functional and non-function bean registration still ends up the same error.

. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: 

2019-05-19 12:49:53.866 INFO 1 --- [ main] lambdainternal.LambdaRTEntry : Starting LambdaRTEntry on 169.254.103.29 with PID 1 (/var/runtime/lib/LambdaJavaRTEntry-1.0.jar started by sbx_user1051 in /)
2019-05-19 12:49:53.867 INFO 1 --- [ main] lambdainternal.LambdaRTEntry : No active profile set, falling back to default profiles: default
2019-05-19 12:49:56.207 INFO 1 --- [ main] c.t.T.cloudFunction.AuthorizerFunction : context: lambdainternal.api.LambdaContext@4f6ee6e4
2019-05-19 12:49:56.263 INFO 1 --- [ main] c.t.T.cloudFunction.AuthorizerFunction : context: lambdainternal.api.LambdaContext@4f6ee6e4
2019-05-19 12:49:57.968 INFO 1 --- [ main] o.s.c.f.web.source.SupplierExporter : Starting
2019-05-19 12:49:58.347 INFO 1 --- [ main] reactor.Flux.MonoRepeatPredicate.1 : onSubscribe(FluxRepeatPredicate.RepeatPredicateSubscriber)
2019-05-19 12:49:58.350 INFO 1 --- [ main] reactor.Flux.MonoRepeatPredicate.1 : request(256)
2019-05-19 12:49:58.584 INFO 1 --- [ main] lambdainternal.LambdaRTEntry : Started LambdaRTEntry in 6.159 seconds (JVM running for 8.113)
Started
2019-05-19 12:49:58.627 INFO 1 --- [ main] c.t.T.cloudFunction.AuthorizerFunction : ARN: arn:aws:lambda:XXXXXX:XXXXXX:function:XXXXXX
2019-05-19 12:49:58.691 ERROR 1 --- [or-http-epoll-3] reactor.Flux.MonoRepeatPredicate.1 : onError(io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: localhost/127.0.0.1:80)
2019-05-19 12:49:58.705 ERROR 1 --- [or-http-epoll-3] reactor.Flux.MonoRepeatPredicate.1 : 

io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: localhost/127.0.0.1:80
at io.netty.channel.unix.Socket.finishConnect(..)(Unknown Source) ~[task/:na]
Caused by: io.netty.channel.unix.Errors$NativeConnectException: syscall:getsockopt(..) failed: Connection refused
... 1 common frames omitted

I think this is the same issue as stackoverflow post: https://stackoverflow.com/questions/56138363/why-is-my-spring-cloud-function-attempting-to-open-local-http-connections

@kleung

This comment has been minimized.

Copy link
Author

commented May 20, 2019

I noticed this only happens when the cloud function actually throws exception

@drissamri

This comment has been minimized.

Copy link

commented May 21, 2019

After generating from the Spring Initializr and taking a look at the cloud-function-aws-samples I get the same error. Haven't been able to get a working version up yet.

[2019-05-22 00:02:41.524] - 68992 INFO [reactor-http-nio-3] --- reactor.Flux.MonoRepeatPredicate.1: onSubscribe(FluxRepeatPredicate.RepeatPredicateSubscriber)
[2019-05-22 00:02:41.524] - 68992 INFO [reactor-http-nio-3] --- reactor.Flux.MonoRepeatPredicate.1: request(256)
[2019-05-22 00:02:41.526] - 68992 SEVERE [reactor-http-nio-3] --- reactor.Flux.MonoRepeatPredicate.1: onError(io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:80)
[2019-05-22 00:02:41.526] - 68992 SEVERE [reactor-http-nio-3] --- reactor.Flux.MonoRepeatPredicate.1: 
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:80
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327)
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:670)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:617)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:534)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.lang.Thread.run(Thread.java:748)
        Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
        |_ checkpoint ⇢ Request to GET http://localhost/2018-06-01/runtime/invocation/next [DefaultWebClient]

Caused by: java.net.ConnectException: Connection refused
        ... 11 more
@Nxtra

This comment has been minimized.

Copy link

commented May 25, 2019

I cannot reproduce this problem. Functional Bean Registration is working for me and decreasing the startup time on AWS with around 50%. Do you have a class eg. public class AwsRequestHandler extends SpringBootRequestHandler<String, String> { ... } to which you point as the handler for the Lambda Function?

@drissamri

This comment has been minimized.

Copy link

commented May 25, 2019

My example can be found here, tried a few different things already: https://github.com/drissamri/aws-lambda-runtime-coldstart/tree/master/5-java8-spring

I didn't get the exception locally anymore but I still did when deployed on AWS

@Nxtra

This comment has been minimized.

Copy link

commented May 25, 2019

I got the same exception. For me it was fixed with the following steps.
I added a class that extends SpringBootRequestHandler with concrete types for the input and output of the lambda function. Specify this class as the handler of your function so AWS finds the handleRequest method with the correct types.
Next, Spring needs to know which function to execute since you can register multiple beans. So I added the function.name property.
Now the function generates a new Product with a random UUID.
drissamri/aws-lambda-runtime-coldstart#1

@Nxtra

This comment has been minimized.

Copy link

commented May 25, 2019

@kleung Does the above help you out? If not, can you show us your code?

@kleung

This comment has been minimized.

Copy link
Author

commented May 26, 2019

My code is as follows:

Application:

@SpringBootConfiguration
public class TestAwsLambdaAuthorizerApplication implements ApplicationContextInitializer<GenericApplicationContext> {
	
	private static final Logger LOG = LoggerFactory.getLogger(TestAwsLambdaAuthorizerApplication.class);
	
	private Context context;
	
	@Autowired
	public void setContext(Context context) {
		LOG.info("Context: {}", context);
		
		this.context = context;
	}
	
	public static void main(String[] args) {
		FunctionalSpringApplication.run(TestAwsLambdaAuthorizerApplication.class, args);
	}

	public AuthorizerFunction authorizer() {
		return new AuthorizerFunction();
	}
	
	public TestFunction testFunction() {
		return new TestFunction();
	}
	
	@Override
	  public void initialize(GenericApplicationContext context) {
	    context.registerBean("authorizer", FunctionRegistration.class,
	        () -> 
	        	new FunctionRegistration<>(authorizer())
	            .type(FunctionType.from(APIGatewayProxyRequestEvent.class).to(AuthPolicy.class))
	        );
	    
	    context.registerBean("testFunction", FunctionRegistration.class,
		        () -> 
		        	new FunctionRegistration<>(testFunction())
		            .type(FunctionType.from(String.class).to(String.class))
		        );
	  }
}

Request Handler

public class AuthorizerEventHandler extends SpringBootRequestHandler<APIGatewayProxyRequestEvent, AuthPolicy> {

}

Function

public class AuthorizerFunction implements Function<APIGatewayProxyRequestEvent, AuthPolicy> {
	
	private static final Logger LOG = LoggerFactory.getLogger(AuthorizerFunction.class);
	
	private Context context;
	
	public AuthorizerFunction() {
		super();
	}

	@Autowired
	@Qualifier("targetExecutionContext")
	public void setContext(Context context) {
		LOG.info("Context: {}", context);
		this.context = context;
	}
	
	@Override
	public AuthPolicy apply(APIGatewayProxyRequestEvent event) {		
		String functionArn = context.getInvokedFunctionArn();
		LOG.info("ARN: {}", functionArn);
		
		String[] arnPartials = functionArn.split(":");
		String region = arnPartials[3];
		
		APIGatewayProxyRequestEvent.ProxyRequestContext requestContext = event.getRequestContext();
		
		String awsAccountId = requestContext.getAccountId();
    	String restApiId = requestContext.getApiId();
    	String stage = requestContext.getStage();
		
		AuthPolicy result = new AuthPolicy("fake-principal", 
												AuthPolicy.PolicyDocument.getDenyAllPolicy(region, 
																							awsAccountId, 
																							restApiId, 
																							stage));
		
		Map<String, String> headers = event.getHeaders();
		String apiKey = headers.get("x-api-key");
		LOG.info("api key: {}", apiKey);
		
		if((!StringUtils.isEmpty(apiKey)) && ("abcd1234".equalsIgnoreCase(apiKey))) {
			result = new AuthPolicy("fake-principal",
										AuthPolicy.PolicyDocument.getAllowAllPolicy(region, 
																						awsAccountId, 
																						restApiId, 
																						stage));
		}
				
		return result;
	}
}

And another function (skipped here), then I point the bean with FUNCTION_NAME environment variable, I did exactly the same as you mentioned but that error happened.

@Nxtra

This comment has been minimized.

Copy link

commented May 26, 2019

I took a look at the code in your repository.
You need to let Spring Boot manage the dependencies. Remove the <version> tag from the spring-cloud-function-adapter-aws dependency.
I see that you don't need the spring-cloud-function-core dependency.
In the application properties mention the name of the function that you want executed: function.name=authorizer. Optionally mention the package to scan for functions: spring.cloud.function.scan.packages=com.test.TestAwsLambdaAuthorizer

These are the changes: kleung/SpringCloudFunctionAwsAuthorizerTest#1

After these changes I get an expected NullPointerException from your logic since I am not invoking the function with your correct event.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.