Unable to use DynamoDB SDK in a Graal native image because of Cannot construct instance of com.amazonaws.partitions.model.Partitions (no Creators, like default construct, exist)

[codependent]

This problem is similar to the issue: #848

The full context is available in a StackOverflow question, where there's also a link to a Github sample project: https://stackoverflow.com/questions/54931286/micronaut-serverless-application-using-dynamodb-and-graal-custom-runtime-throws

Summing up:

In my native image running on a AWS lambda custom runtime I call some DynamoDB tables. The problem is in com.amazonaws.partition.PartitionsLoader.loadPartitionFromStream() they invoke jackson to unmarshal an endpoints.json file into a com.amazonaws.partitions.model.Partitions object.

This ends up throwing this exception:

Caused by: com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of `com.amazonaws.partitions.model.Partitions` (no Creators, like default construct, exist): cannot deserialize from Object value (no delegate- or property-based Creator)
at [Source: (ByteArrayInputStream); line: 2, column: 3]
at com.fasterxml.jackson.databind.DeserializationContext.reportBadDefinition(DeserializationContext.java:1452)
at com.fasterxml.jackson.databind.DeserializationContext.handleMissingInstantiator(DeserializationContext.java:1028)
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1297)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:326)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:159)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4013)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3070)
at com.amazonaws.partitions.PartitionsLoader.loadPartitionFromStream(PartitionsLoader.java:92)
... 68 more

I have tried adding into reflect.json file, as suggested in the other issue:

...
}, {
  "name" : "com.amazonaws.partitions.model.Partitions",
  "allPublicMethods" : true,
  "allDeclaredConstructors" : true
}] 

[cstancu]

I was able to succesfully execute the

$ docker build . -t micronaut-aws-lambda-proxy-graal
$ ./sam-local.sh

steps. How can I test this locally, without requiring an aws account. I tried to go to http://127.0.0.1:3000/ and I get {"message":"Missing Authentication Token"} as a response.

[codependent]

Hi @cstancu, I've prepared a simplified project so that you can test it in a local environment: https://github.com/codependent/graal-app

Project setup:

1. export AWS_DEFAULT_REGION=eu-central-1
2. docker build . -t graal-app

To run the app just execute ./sam-local.sh and access http://localhost:3000/ping.

The first time the request may take a while until it downloads the lambda docker image:

Fetching lambci/lambda:provided Docker container image...............
..................

I've extracted the functionality into a controller that passes a json to the object mapper getting the same exception:

@Controller("/")
public class ExampleController {

    private static final Logger LOG = LoggerFactory.getLogger(ExampleController.class);
    private ObjectMapper mapper = new ObjectMapper()
            .disable(MapperFeature.CAN_OVERRIDE_ACCESS_MODIFIERS)
            .disable(MapperFeature.ALLOW_FINAL_FIELDS_AS_MUTATORS)
            .enable(JsonParser.Feature.ALLOW_COMMENTS)
            .disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES);

    @Get("/ping")
    public String index() throws IOException {
        LOG.info("Local Test");

        String json = "{\n" +
                "  \"partitions\" : [ {\n" +
                "    \"defaults\" : {\n" +
                "      \"hostname\" : \"{service}.{region}.{dnsSuffix}\",\n" +
                "      \"protocols\" : [ \"https\" ],\n" +
                "      \"signatureVersions\" : [ \"v4\" ]\n" +
                "    }\n" +
                "  }]\n" +
                "}";

        ByteArrayInputStream stream = new ByteArrayInputStream(json.getBytes());
        final Partitions partitions = mapper.readValue(stream, Partitions.class);
        LOG.info("Local Test partitions {}", partitions);
        return "{\"pong\":true, \"graal\": true}";
    }
}

Exception:

Caused by: com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of `com.amazonaws.partitions.model.Partitions` (no Creators, like default construct, exist): cannot deserialize from Object value (no delegate- or property-based Creator)
 at [Source: (ByteArrayInputStream); line: 2, column: 3]
        at com.fasterxml.jackson.databind.DeserializationContext.reportBadDefinition(DeserializationContext.java:1452)
        at com.fasterxml.jackson.databind.DeserializationContext.handleMissingInstantiator(DeserializationContext.java:1028)
        at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1297)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:326)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:159)
        at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4013)
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3070)
        at graal.app.ExampleController.index(ExampleController.java:44)
        at graal.app.$ExampleControllerDefinition$$exec1.invokeInternal(Unknown Source)
        at io.micronaut.context.AbstractExecutableMethod.invoke(AbstractExecutableMethod.java:144)
        at io.micronaut.context.DefaultBeanContext$BeanExecutionHandle.invoke(DefaultBeanContext.java:2545)
        at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:236)
        at io.micronaut.web.router.RouteMatch.execute(RouteMatch.java:122)
        at io.micronaut.function.aws.proxy.MicronautLambdaContainerHandler.lambda$null$1(MicronautLambdaContainerHandler.java:240)
        at io.reactivex.internal.operators.flowable.FlowableDefer.subscribeActual(FlowableDefer.java:35)
        at io.reactivex.Flowable.subscribe(Flowable.java:14805)
        at io.reactivex.Flowable.subscribe(Flowable.java:14752)
        at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68)
        at io.reactivex.Flowable.subscribe(Flowable.java:14805)
        at io.reactivex.Flowable.subscribe(Flowable.java:14755)
        at io.micronaut.http.context.ServerRequestTracingPublisher.lambda$subscribe$0(ServerRequestTracingPublisher.java:52)
        at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:52)
        at io.micronaut.http.context.ServerRequestTracingPublisher.subscribe(ServerRequestTracingPublisher.java:52)
        at io.reactivex.internal.operators.flowable.FlowableFromPublisher.subscribeActual(FlowableFromPublisher.java:29)
        at io.reactivex.Flowable.subscribe(Flowable.java:14805)
        at io.reactivex.Flowable.subscribe(Flowable.java:14752)
        at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68)
        at io.reactivex.Flowable.subscribe(Flowable.java:14805)
        at io.reactivex.Flowable.subscribe(Flowable.java:14752)
        at io.reactivex.internal.operators.flowable.FlowableSubscribeOn$SubscribeOnSubscriber.run(FlowableSubscribeOn.java:82)
        at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:52)
        at io.micronaut.http.context.ServerRequestContext.lambda$instrument$0(ServerRequestContext.java:68)
        at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker$BooleanRunnable.run(ExecutorScheduler.java:288)
        at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker.run(ExecutorScheduler.java:253)
        at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:52)
        at io.micronaut.http.context.ServerRequestContext.lambda$instrument$0(ServerRequestContext.java:68)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:52)
        at io.micronaut.http.context.ServerRequestContext.lambda$instrument$0(ServerRequestContext.java:68)
        at java.lang.Thread.run(Thread.java:748)
        at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:481)
        at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:193)
END RequestId: 52fdfc07-2182-154f-163f-5f0f9a621d72

[cstancu]

Thank you for the easy to use reproducible. I updated it to use InputStream stream = ExampleController.class.getClassLoader().getResourceAsStream("com/amazonaws/partitions/endpoints.json"); instead of the stream from the json string, and also updated the native-image command to register the resource: -H:IncludeResources="logback.xml|application.yml|com/amazonaws/partitions/endpoints.json".

I was able to get past the jackson errors by adding these entries to reflect.json:

, {
  "name" : "com.amazonaws.partitions.model.Partitions",
  "allPublicMethods" : true,
  "allDeclaredConstructors" : true
}, {
  "name" : "com.amazonaws.partitions.model.Partition",
  "allPublicMethods" : true,
  "allDeclaredConstructors" : true
}, {
  "name" : "com.amazonaws.partitions.model.Endpoint",
  "allPublicMethods" : true,
  "allDeclaredConstructors" : true
}, {
  "name" : "com.amazonaws.partitions.model.Region",
  "allPublicMethods" : true,
  "allDeclaredConstructors" : true
}, {
  "name" : "com.amazonaws.partitions.model.Service",
  "allPublicMethods" : true,
  "allDeclaredConstructors" : true
}, {
  "name" : "com.amazonaws.partitions.model.CredentialScope",
  "allPublicMethods" : true,
  "allDeclaredConstructors" : true
}, {
  "name" : "java.util.HashSet",
  "allPublicMethods" : true,
  "allDeclaredConstructors" : true
}

I came up with the list of classes that need reflective access by painstakingly following the Caused by: com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of ... errors one at the time. Unfortunatelly currently there is no automated way to detect reflectively accessed elements; we are working on it. The jackson error messages are sometimes cryptic but in this case they were straightforward, although quite noisy, so it is easy to miss the essence of the issue if you don't know what you're looking for.

When I go to http://localhost:3000/ping I get: {"pong":true, "graal": true} and no error on the server side.

[codependent]

I appreciate your taking the time to help me work this out. I'll be great I you end up detecting reflectively accessed elements automatically. In the meantime I'll take this into account for similar cases.

Thanks again!!

[abhishekmishra17]

The same issue occurring for aws-java-sdk-cognitoidentity:1.11.887.Is issue fixed for all or specficy to DynamoDb?

[jiu-z]

stream

do you have the neweast source? i want to try your source

[jiu-z]

stream

do you have the neweast source? i want to try your source
@codependent