AWS Adapter: CustomRuntimeEventLoop sticks with first function definiton header when invoked

[ArnauAregall]

Describe the bug

I have a Spring Cloud Function application deployed as an AWS Lamda on a provided.al2 Runtime (the application is a GraalVM native executable).

The Spring Boot and Spring Cloud dependencies I am using:

springBootVersion=3.1.0
springCloudVersion=2022.0.3

implementation("org.springframework.boot:spring-boot-starter-webflux")
implementation("org.springframework.cloud:spring-cloud-function-context")
implementation("org.springframework.cloud:spring-cloud-function-web")
implementation("org.springframework.cloud:spring-cloud-function-adapter-aws")

The application exposes two different functions as @Beans:

@Bean
fun forecast(weatherService: WeatherService): (Message<GeoLocation>) -> Mono<Forecast> = weatherService::getForecast

@Bean
fun question(questionService: QuestionService): (Message<Question>) -> Mono<Answer> = questionService::answerQuestion

After deploying the AWS Lambda, I can correctly invoke the "question" function using HTTP message header spring.cloud.function.definition: question with a JSON request body that maps successfully as a Question domain object, and the application returns the desired response. Everything OK until here.

But after that, if I want to invoke the just theforecast function, now with the HTTP header spring.cloud.function.definition: forecast and corresponding request body message that can be mapped to a GeoLocation object, the CustomRuntimeEventLoop detects the resolved function is still question.

Therefore then the message conversion fails as it tries to convert the request body to a Question object with a body aimed for a GeoLocation object.

Evidences from CloudWatch logs (removed AWS account sensitive parts):

2023/06/19/[$LATEST]c646dfa4d77c49c3a462b769ebf7ebcf 2023-06-19T08:49:02.108000 2023-06-19T08:49:02.108Z  INFO 9 --- [pool-5-thread-1] o.s.c.f.a.aws.CustomRuntimeEventLoop     : Located function question
2023/06/19/[$LATEST]c646dfa4d77c49c3a462b769ebf7ebcf 2023-06-19T08:49:02.108000 2023-06-19T08:49:02.108Z  INFO 9 --- [pool-5-thread-1] o.s.c.f.adapter.aws.AWSLambdaUtils       : Received: {"version":"1.0","resource":"$default","path":"/","httpMethod":"POST","headers":{"Content-Length":"48","Content-Type":"application/json","Host":"XXXXXXXXXX.amazonaws.com","User-Agent":"Apache-HttpClient/4.5.14 (Java/17.0.6)","X-Amzn-Trace-Id":"Root=1-......","X-Forwarded-For":".......","X-Forwarded-Port":"443","X-Forwarded-Proto":"https","accept":"application/json","accept-encoding":"br,deflate,gzip,x-gzip","spring.cloud.function.definition":"forecast"},"multiValueHeaders":{"Content-Length":["48"],"Content-Type":["application/json"],"Host":["XXXXXXXXXX.amazonaws.com"],"User-Agent":["Apache-HttpClient/4.5.14 (Java/17.0.6)"],"X-Amzn-Trace-Id":["Root=1-6490167e-6b78cfea23bc44e72647d12b"],"X-Forwarded-For":["XXXXXXXXXX"],"X-Forwarded-Port":["443"],"X-Forwarded-Proto":["https"],"accept":["application/json"],"accept-encoding":["br,deflate,gzip,x-gzip"],"spring.cloud.function.definition":["forecast"]} ....}

2023/06/19/[$LATEST]c646dfa4d77c49c3a462b769ebf7ebcf 2023-06-19T08:49:02.109000 java.lang.IllegalStateException: Failed to convert. Possible bug as the conversion probably shouldn't have been attempted here
2023/06/19/[$LATEST]c646dfa4d77c49c3a462b769ebf7ebcf 2023-06-19T08:49:02.109000 at org.springframework.cloud.function.json.JacksonMapper.doFromJson(JacksonMapper.java:65) ~[na:na]
2023/06/19/[$LATEST]c646dfa4d77c49c3a462b769ebf7ebcf 2023-06-19T08:49:02.109000 at org.springframework.cloud.function.json.JsonMapper.fromJson(JsonMapper.java:66) ~[aws-lambda-spring-cloud-function:4.0.3]
2023/06/19/[$LATEST]c646dfa4d77c49c3a462b769ebf7ebcf 2023-06-19T08:49:02.109000 at org.springframework.cloud.function.adapter.aws.AWSTypesMessageConverter.convertFromInternal(AWSTypesMessageConverter.java:90)
...

Note: after waiting the lambda cooldown period, invoking again just the question function, it works correctly.

I am not sure if it could be somehow related to the heads up of the reference documentation, as I am using Mono<?> return types.

https://cloud.spring.io/spring-cloud-function/reference/html/spring-cloud-function.html#_function_routing

Disclaimer: did not try with a java17 runtime though.

[olegz]

Actually yes, if your function is reactive you can NOT use routing based on headers. The RoutingFunction won't allow that primarily due to the streaming issues where in streaming environments a reactive function is only invoked once.
However, i'll give it another look in the context of AWS.

Also, not a fan of using reactive with AWS since aside from API you are not benefitting from core features of the reactive especially with API gateway or custom runtime where the source request is actually blocking.

[ArnauAregall]

Thanks for the advice, now that I have understood a bit better how AWS adapter and custom runtime work I can understand your reasoning here.

[olegz]

FWIW, it works fine locally and that is why i said I'll revisit it with regard to AWS, especially with API gateway where each request is different, so a function is invoked on each request as a new thing.
Just to clarify, you are attempting to invoke two different functions with two different HTTP requests, thus different definition headers, correct?

[olegz]

So, i just modified your previous sample, by:

• Added new function:

@Bean
public Function<Flux<GeoLocation>, Flux<GeoLocation>> echo() {
        return x -> x;
}

• Removed function.definition from the yaml file
  Build, deploy as per your instructions. .
  Than I curled it with

curl -X POST -H "Content-Type: application/json" -H "spring.cloud.function.definition: echo" -d '[{"latitude": 41.34, "longitude": 2.78}]' https://emcdxu5ijj.execute-api.us-east-2.amazonaws.com/question

and then with

curl -X POST -H "Content-Type: application/json" -H "spring.cloud.function.definition: forecast" -d '[{"latitude": 41.34, "longitude": 2.78}]' https://emcdxu5ijj.execute-api.us-east-2.amazonaws.com/question

And all went as expected - two different functions were invoked

So, I am wondering as to what are the steps to reproduce your issue

[ArnauAregall]

The issue was reproduced invoking two functions that accept different payload.

I guess the "echo" one you posted works because the payload object is effectively the same and can be accepted by the two functions.

I'll try to update the sample app by adding a new function to reproduce the issue.

[olegz]

I just changed to

 @Bean
 public Function<Flux<String>, Flux<String>> echo() {
        return x -> x;
 }

will let you know in a few

[olegz]

So all works fine with these two requests

curl -X POST -H "Content-Type: application/json" -H "spring.cloud.function.definition: echo" -d '"blah"' https://emcdxu5ijj.execute-api.us-east-2.amazonaws.com/question

and

curl -X POST -H "Content-Type: application/json" -H "spring.cloud.function.definition: forecast" -d '[{"latitude": 41.34, "longitude": 2.78}]' https://emcdxu5ijj.execute-api.us-east-2.amazonaws.com/question

[ArnauAregall]

Thanks for checking @olegz

I added two extra functions to the sample app to double check it, and I am also not not able to reproduce it again on AWS with that Java sample.

However, the original issue was opened when I managed to reproduce it in another sample project that has roughly the same implementation but using Kotlin, and I can still reproduce the issue on AWS with that Kotlin SCF application.

The Kotlin sample app is the following: https://github.com/ArnauAregall/aws-lambda-spring-cloud-function

Build instructions are also provided, except OPENAI_API_KEY env var can be defined as foo, not needed to reproduce the issue.

Once deployed, invoke the function with the following requests:

1. Invoke forecast function

curl -X POST --location "https://xxxxxxxxxx.execute-api.yyyyyyyy.amazonaws.com" \
    -H "Accept: application/json" \
    -H "Content-Type: application/json" \
    -H "spring.cloud.function.definition: forecast" \
    -d "{
          \"latitude\": 41.3874,
          \"longitude\": 2.1686
        }"

Response should be the already known Forecast.

2. Invoke a function compositon: forecast,forecastToQuestion

curl -X POST --location "https://xxxxxxxxxx.execute-api.yyyyyyyy.amazonaws.com" \
    -H "Accept: application/json" \
    -H "Content-Type: application/json" \
    -H "spring.cloud.function.definition: forecast,forecastToQuestion" \
    -d "{
          \"latitude\": 41.3874,
          \"longitude\": 2.1686
        }"

The response is still the result of forecast only.

The CloudWatch logs

2023-07-14T06:09:51.497Z DEBUG 9 --- [pool-5-thread-1] o.s.c.f.a.aws.CustomRuntimeEventLoop     : New Event received: <200 OK OK,{"version":"1.0","resource":"$default","path":"/","httpMethod":"POST","headers":{"Content-Length":"48","Content-Type":"application/json","Host":"xxxxxxxxxx.execute-api.yyyyyyyy.amazonaws.com","User-Agent":"Apache-HttpClient/4.5.14 (Java/17.0.7)","X-Amzn-Trace-Id":"Root=1-64b0e6af-49bcf7120f153aa960c18ca3","X-Forwarded-For":"90.68.82.53","X-Forwarded-Port":"443","X-Forwarded-Proto":"https","accept":"application/json","accept-encoding":"br,deflate,gzip,x-gzip","spring.cloud.function.definition":"forecast,forecastToQuestion"},"multiValueHeaders":{"Content-Length":["48"],"Content-Type":["application/json"],"Host":["xxxxxxxxxx.execute-api.yyyyyyyy.amazonaws.com"],"User-Agent":["Apache-HttpClient/4.5.14 (Java/17.0.7)"],"X-Amzn-Trace-Id":["Root=1-64b0e6af-49bcf7120f153aa960c18ca3"],"X-Forwarded-For":["90.68.82.53"],"X-Forwarded-Port":["443"],"X-Forwarded-Proto":["https"],"accept":["application/json"],"accept-encoding":["br,deflate,gzip,x-gzip"],"spring.cloud.function.definition":["forecast,forecastToQuestion"]},"queryStringParameters":null,"multiValueQueryStringParameters":null,"requestContext":{"accountId":"741919747208","apiId":"fdwvp3wqoh","domainName":"xxxxxxxxxx.execute-api.yyyyyyyy.amazonaws.com","domainPrefix":"fdwvp3wqoh","extendedRequestId":"ICj7dgoQFiAEJbA=","httpMethod":"POST","identity":{"accessKey":null,"accountId":null,"caller":null,"cognitoAmr":null,"cognitoAuthenticationProvider":null,"cognitoAuthenticationType":null,"cognitoIdentityId":null,"cognitoIdentityPoolId":null,"principalOrgId":null,"sourceIp":"XXXXXXXXXX","user":null,"userAgent":"Apache-HttpClient/4.5.14 (Java/17.0.7)","userArn":null},"path":"/","protocol":"HTTP/1.1","requestId":"ICj7dgoQFiAEJbA=","requestTime":"14/Jul/2023:06:09:51 +0000","requestTimeEpoch":1689314991468,"resourceId":"$default","resourcePath":"$default","stage":"$default"},"pathParameters":null,"stageVariables":null,"body":"{\n  \"latitude\": 41.3874,\n  \"longitude\": 2.1686\n}","isBase64Encoded":false},[Content-Type:"application/json", Lambda-Runtime-Aws-Request-Id:"79b50de9-2708-4c1d-9f02-b869b07f61ed", Lambda-Runtime-Deadline-Ms:"1689315011496", Lambda-Runtime-Invoked-Function-Arn:"XXXXXXXX", Lambda-Runtime-Trace-Id:"Root=1-64b0e6af-2f15c7d95e699a0470759efd;Parent=b160849799657d27;Sampled=1;Lineage=8643e627:0", Date:"Fri, 14 Jul 2023 06:09:51 GMT", Content-Length:"1987"]>
2023-07-14T06:09:51.497Z DEBUG 9 --- [pool-5-thread-1] o.s.c.f.a.aws.CustomRuntimeEventLoop     : Value of DEFAULT_HANDLER env: null
2023-07-14T06:09:51.497Z  INFO 9 --- [pool-5-thread-1] o.s.c.f.a.aws.CustomRuntimeEventLoop     : Located function forecast

As you can see the header is received as expected "spring.cloud.function.definition":["forecast,forecastToQuestion"]} but the runtime event loop states Located function forecast.

Would be great if you could check, in the meantime I will prepare a simpler Kotlin sample app and try to reproduce it with that one.