Spring cloud function on aws getting connection refused exception from netty

[kleung]

Using version 2.1.0.RELEASE of spring-cloud-function-(core, context, adapter-aws, webflux and web) cause the following stack trace when deployed on aws:

NOTE: I tried both functional and non-function bean registration still ends up the same error.

. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: 

2019-05-19 12:49:53.866 INFO 1 --- [ main] lambdainternal.LambdaRTEntry : Starting LambdaRTEntry on 169.254.103.29 with PID 1 (/var/runtime/lib/LambdaJavaRTEntry-1.0.jar started by sbx_user1051 in /)
2019-05-19 12:49:53.867 INFO 1 --- [ main] lambdainternal.LambdaRTEntry : No active profile set, falling back to default profiles: default
2019-05-19 12:49:56.207 INFO 1 --- [ main] c.t.T.cloudFunction.AuthorizerFunction : context: lambdainternal.api.LambdaContext@4f6ee6e4
2019-05-19 12:49:56.263 INFO 1 --- [ main] c.t.T.cloudFunction.AuthorizerFunction : context: lambdainternal.api.LambdaContext@4f6ee6e4
2019-05-19 12:49:57.968 INFO 1 --- [ main] o.s.c.f.web.source.SupplierExporter : Starting
2019-05-19 12:49:58.347 INFO 1 --- [ main] reactor.Flux.MonoRepeatPredicate.1 : onSubscribe(FluxRepeatPredicate.RepeatPredicateSubscriber)
2019-05-19 12:49:58.350 INFO 1 --- [ main] reactor.Flux.MonoRepeatPredicate.1 : request(256)
2019-05-19 12:49:58.584 INFO 1 --- [ main] lambdainternal.LambdaRTEntry : Started LambdaRTEntry in 6.159 seconds (JVM running for 8.113)
Started
2019-05-19 12:49:58.627 INFO 1 --- [ main] c.t.T.cloudFunction.AuthorizerFunction : ARN: arn:aws:lambda:XXXXXX:XXXXXX:function:XXXXXX
2019-05-19 12:49:58.691 ERROR 1 --- [or-http-epoll-3] reactor.Flux.MonoRepeatPredicate.1 : onError(io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: localhost/127.0.0.1:80)
2019-05-19 12:49:58.705 ERROR 1 --- [or-http-epoll-3] reactor.Flux.MonoRepeatPredicate.1 : 

io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: localhost/127.0.0.1:80
at io.netty.channel.unix.Socket.finishConnect(..)(Unknown Source) ~[task/:na]
Caused by: io.netty.channel.unix.Errors$NativeConnectException: syscall:getsockopt(..) failed: Connection refused
... 1 common frames omitted

I think this is the same issue as stackoverflow post: https://stackoverflow.com/questions/56138363/why-is-my-spring-cloud-function-attempting-to-open-local-http-connections

[kleung]

I noticed this only happens when the cloud function actually throws exception

[drissamri]

After generating from the Spring Initializr and taking a look at the cloud-function-aws-samples I get the same error. Haven't been able to get a working version up yet.

[2019-05-22 00:02:41.524] - 68992 INFO [reactor-http-nio-3] --- reactor.Flux.MonoRepeatPredicate.1: onSubscribe(FluxRepeatPredicate.RepeatPredicateSubscriber)
[2019-05-22 00:02:41.524] - 68992 INFO [reactor-http-nio-3] --- reactor.Flux.MonoRepeatPredicate.1: request(256)
[2019-05-22 00:02:41.526] - 68992 SEVERE [reactor-http-nio-3] --- reactor.Flux.MonoRepeatPredicate.1: onError(io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:80)
[2019-05-22 00:02:41.526] - 68992 SEVERE [reactor-http-nio-3] --- reactor.Flux.MonoRepeatPredicate.1: 
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:80
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327)
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:670)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:617)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:534)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.lang.Thread.run(Thread.java:748)
        Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
        |_ checkpoint ⇢ Request to GET http://localhost/2018-06-01/runtime/invocation/next [DefaultWebClient]

Caused by: java.net.ConnectException: Connection refused
        ... 11 more

[Nxtra]

I cannot reproduce this problem. Functional Bean Registration is working for me and decreasing the startup time on AWS with around 50%. Do you have a class eg. public class AwsRequestHandler extends SpringBootRequestHandler<String, String> { ... } to which you point as the handler for the Lambda Function?

[drissamri]

My example can be found here, tried a few different things already: https://github.com/drissamri/aws-lambda-runtime-coldstart/tree/master/5-java8-spring

I didn't get the exception locally anymore but I still did when deployed on AWS

[Nxtra]

I got the same exception. For me it was fixed with the following steps.
I added a class that extends SpringBootRequestHandler with concrete types for the input and output of the lambda function. Specify this class as the handler of your function so AWS finds the handleRequest method with the correct types.
Next, Spring needs to know which function to execute since you can register multiple beans. So I added the function.name property.
Now the function generates a new Product with a random UUID.
drissamri/aws-lambda-runtime-coldstart#1

[Nxtra]

@kleung Does the above help you out? If not, can you show us your code?

[kleung]

My code is as follows:

Application:

@SpringBootConfiguration
public class TestAwsLambdaAuthorizerApplication implements ApplicationContextInitializer<GenericApplicationContext> {
	
	private static final Logger LOG = LoggerFactory.getLogger(TestAwsLambdaAuthorizerApplication.class);
	
	private Context context;
	
	@Autowired
	public void setContext(Context context) {
		LOG.info("Context: {}", context);
		
		this.context = context;
	}
	
	public static void main(String[] args) {
		FunctionalSpringApplication.run(TestAwsLambdaAuthorizerApplication.class, args);
	}

	public AuthorizerFunction authorizer() {
		return new AuthorizerFunction();
	}
	
	public TestFunction testFunction() {
		return new TestFunction();
	}
	
	@Override
	  public void initialize(GenericApplicationContext context) {
	    context.registerBean("authorizer", FunctionRegistration.class,
	        () -> 
	        	new FunctionRegistration<>(authorizer())
	            .type(FunctionType.from(APIGatewayProxyRequestEvent.class).to(AuthPolicy.class))
	        );
	    
	    context.registerBean("testFunction", FunctionRegistration.class,
		        () -> 
		        	new FunctionRegistration<>(testFunction())
		            .type(FunctionType.from(String.class).to(String.class))
		        );
	  }
}

Request Handler

public class AuthorizerEventHandler extends SpringBootRequestHandler<APIGatewayProxyRequestEvent, AuthPolicy> {

}

Function

public class AuthorizerFunction implements Function<APIGatewayProxyRequestEvent, AuthPolicy> {
	
	private static final Logger LOG = LoggerFactory.getLogger(AuthorizerFunction.class);
	
	private Context context;
	
	public AuthorizerFunction() {
		super();
	}

	@Autowired
	@Qualifier("targetExecutionContext")
	public void setContext(Context context) {
		LOG.info("Context: {}", context);
		this.context = context;
	}
	
	@Override
	public AuthPolicy apply(APIGatewayProxyRequestEvent event) {		
		String functionArn = context.getInvokedFunctionArn();
		LOG.info("ARN: {}", functionArn);
		
		String[] arnPartials = functionArn.split(":");
		String region = arnPartials[3];
		
		APIGatewayProxyRequestEvent.ProxyRequestContext requestContext = event.getRequestContext();
		
		String awsAccountId = requestContext.getAccountId();
    	String restApiId = requestContext.getApiId();
    	String stage = requestContext.getStage();
		
		AuthPolicy result = new AuthPolicy("fake-principal", 
												AuthPolicy.PolicyDocument.getDenyAllPolicy(region, 
																							awsAccountId, 
																							restApiId, 
																							stage));
		
		Map<String, String> headers = event.getHeaders();
		String apiKey = headers.get("x-api-key");
		LOG.info("api key: {}", apiKey);
		
		if((!StringUtils.isEmpty(apiKey)) && ("abcd1234".equalsIgnoreCase(apiKey))) {
			result = new AuthPolicy("fake-principal",
										AuthPolicy.PolicyDocument.getAllowAllPolicy(region, 
																						awsAccountId, 
																						restApiId, 
																						stage));
		}
				
		return result;
	}
}

And another function (skipped here), then I point the bean with FUNCTION_NAME environment variable, I did exactly the same as you mentioned but that error happened.

[Nxtra]

I took a look at the code in your repository.
You need to let Spring Boot manage the dependencies. Remove the <version> tag from the spring-cloud-function-adapter-aws dependency.
I see that you don't need the spring-cloud-function-core dependency.
In the application properties mention the name of the function that you want executed: function.name=authorizer. Optionally mention the package to scan for functions: spring.cloud.function.scan.packages=com.test.TestAwsLambdaAuthorizer

These are the changes: kleung/SpringCloudFunctionAwsAuthorizerTest#1

After these changes I get an expected NullPointerException from your logic since I am not invoking the function with your correct event.

[olegz]

I don't think this is an issue anymore. That said, there were also some enhancements. One of them is that you no longer need to specify an empty AWS request handler as we provide generic one that handles everything from API gateway, to stream, to Kinesis. We have updated user guide, so feedback is welcome

[ripper2hl]

I have the same error but i fixed when add an profile in my pom.xml for the aws dependencies and run the test without aws profile.

mvn test

This is the part of aws profile dependencies

    <profiles>
        <profile>
            <id>aws</id>
            <dependencies>
                <dependency>
                    <groupId>org.springframework.cloud</groupId>
                    <artifactId>spring-cloud-function-adapter-aws</artifactId>
                </dependency>
                <dependency>
                    <groupId>com.amazonaws</groupId>
                    <artifactId>aws-lambda-java-events</artifactId>
                    <version>2.0.2</version>
                </dependency>
                <dependency>
                    <groupId>com.amazonaws</groupId>
                    <artifactId>aws-lambda-java-core</artifactId>
                    <version>1.1.0</version>
                    <scope>provided</scope>
                </dependency>
            </dependencies>
            <build>
                <plugins>
                    <plugin>
                        <groupId>org.apache.maven.plugins</groupId>
                        <artifactId>maven-deploy-plugin</artifactId>
                        <configuration>
                            <skip>true</skip>
                        </configuration>
                    </plugin>
                    <plugin>
                        <groupId>org.springframework.boot</groupId>
                        <artifactId>spring-boot-maven-plugin</artifactId>
                        <dependencies>
                            <dependency>
                                <groupId>org.springframework.boot.experimental</groupId>
                                <artifactId>spring-boot-thin-layout</artifactId>
                                <version>1.0.17.RELEASE</version>
                            </dependency>
                        </dependencies>
                    </plugin>
                    <plugin>
                        <groupId>org.apache.maven.plugins</groupId>
                        <artifactId>maven-shade-plugin</artifactId>
                        <version>3.2.4</version>
                        <configuration>
                            <createDependencyReducedPom>false</createDependencyReducedPom>
                            <shadedArtifactAttached>true</shadedArtifactAttached>
                            <shadedClassifierName>aws</shadedClassifierName>
                        </configuration>
                    </plugin>
                </plugins>
            </build>
        </profile>
    </profiles>

This is the pom.xml , ignore rekognition libraries.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.3.4.RELEASE</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.some</groupId>
    <artifactId>x</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>x</name>
    <description>Demo project for Spring Boot</description>

    <properties>
        <java.version>1.8</java.version>
        <spring-cloud.version>Hoxton.SR8</spring-cloud.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-function-webflux</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-function-compiler</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-configuration-processor</artifactId>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-devtools</artifactId>
            <scope>runtime</scope>
            <optional>true</optional>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
            <exclusions>
                <exclusion>
                    <groupId>org.junit.vintage</groupId>
                    <artifactId>junit-vintage-engine</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-rekognition</artifactId>
            <version>1.11.860</version>
        </dependency>
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-transcribe</artifactId>
            <version>1.11.864</version>
        </dependency>

        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-sns</artifactId>
            <version>1.11.860</version>
        </dependency>
    </dependencies>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.cloud</groupId>
                <artifactId>spring-cloud-dependencies</artifactId>
                <version>${spring-cloud.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <profiles>
        <profile>
            <id>aws</id>
            <dependencies>
                <dependency>
                    <groupId>org.springframework.cloud</groupId>
                    <artifactId>spring-cloud-function-adapter-aws</artifactId>
                </dependency>
                <dependency>
                    <groupId>com.amazonaws</groupId>
                    <artifactId>aws-lambda-java-events</artifactId>
                    <version>2.0.2</version>
                </dependency>
                <dependency>
                    <groupId>com.amazonaws</groupId>
                    <artifactId>aws-lambda-java-core</artifactId>
                    <version>1.1.0</version>
                    <scope>provided</scope>
                </dependency>
            </dependencies>
            <build>
                <plugins>
                    <plugin>
                        <groupId>org.apache.maven.plugins</groupId>
                        <artifactId>maven-deploy-plugin</artifactId>
                        <configuration>
                            <skip>true</skip>
                        </configuration>
                    </plugin>
                    <plugin>
                        <groupId>org.springframework.boot</groupId>
                        <artifactId>spring-boot-maven-plugin</artifactId>
                        <dependencies>
                            <dependency>
                                <groupId>org.springframework.boot.experimental</groupId>
                                <artifactId>spring-boot-thin-layout</artifactId>
                                <version>1.0.17.RELEASE</version>
                            </dependency>
                        </dependencies>
                    </plugin>
                    <plugin>
                        <groupId>org.apache.maven.plugins</groupId>
                        <artifactId>maven-shade-plugin</artifactId>
                        <version>3.2.4</version>
                        <configuration>
                            <createDependencyReducedPom>false</createDependencyReducedPom>
                            <shadedArtifactAttached>true</shadedArtifactAttached>
                            <shadedClassifierName>aws</shadedClassifierName>
                        </configuration>
                    </plugin>
                </plugins>
            </build>
        </profile>
    </profiles>


    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

</project>

[olegz]

There have been many improvements, updates and enhancements to AWS recently, so given the age of this issue I will close it with no action, but feel free to give it try and raise new issues as you discover them