Dyanmodb Enchanced client nested Objects as json #2265

tkindra2 · 2021-01-29T20:56:04Z

I have nested Object (composition) use to represent data that i want to store and using Dyanmodb enhanced client as part of AWS Java version 2 api. In the readme it explains how to flatten the objects. In the version one of the api was able to store list of objects as json documents in dyanmodb. Would like it to be stored as Json not flattened for backward compatibility:

Steps to Reproduce

public class Customer{
  private String name;
  private List<GenericRecord> recordMetadata;
  //getters and setters for all attributes
}


public class GenericRecord {
  private String id;
  private String details;
  //getters and setters for all attributes
}

Would like it to be stored as below not flattened for backward compatibility:

{
  "name": "ABC",
 "recordMetadata": [
  {
    "id":"123",
    "details":"hello"

   },
  {
    "id":"456",
    "details":"yellow"

   }
 ]

}

Current Behavior

In the readme it only describe how to flatten

https://github.com/aws/aws-sdk-java-v2/blob/master/services-custom/dynamodb-enhanced/README.md

Your Environment

AWS Java SDK version used: 2.15.72

The text was updated successfully, but these errors were encountered:

catchin · 2021-02-02T09:02:25Z

I'm wondering why this should not work. I stumbled upon a problem with nested Java classes, but the exception was triggered by a different problem.
Here is an example from the test suite: https://github.com/aws/aws-sdk-java-v2/blob/master/services-custom/dynamodb-enhanced/src/test/java/software/amazon/awssdk/enhanced/dynamodb/mapper/testbeans/DocumentBean.java#L29

Did you add @DynamoDbBean to the GenericRecord class?

tkindra2 · 2021-02-02T21:14:13Z

I am using StaticTableSchema builder to reduce startup time, so no annotations. One work around would be to create custom attribute converter, but thats tedious. If i don't provide attribute converter, i get exception

public static final TableSchema<School> TABLE_SCHEMA = StaticTableSchema
		.builder(Customer.class).newItemSupplier(Customer::new)
                   .addAttribute(List.class,
				a -> a.name("recordMetadata").getter(Customer::getRecordMetadata)
						.setter(School::setRecordMetadata)

java.lang.ExceptionInInitializerError
at com.childcare.lambda.school.SchoolIntegrationTest.(SchoolIntegrationTest.java:46)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:64)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
at org.junit.runners.BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java:250)
at org.junit.runners.BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java:260)
at org.junit.runners.BlockJUnit4ClassRunner$2.runReflectiveCall(BlockJUnit4ClassRunner.java:309)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:306)
at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:89)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:41)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:542)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:770)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:464)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:210)
Caused by: java.lang.IndexOutOfBoundsException: Index: 0
at java.base/java.util.Collections$EmptyList.get(Collections.java:4553)
at software.amazon.awssdk.enhanced.dynamodb.DefaultAttributeConverterProvider.findConverter(DefaultAttributeConverterProvider.java:153)
at software.amazon.awssdk.enhanced.dynamodb.DefaultAttributeConverterProvider.converterFor(DefaultAttributeConverterProvider.java:133)
at software.amazon.awssdk.enhanced.dynamodb.mapper.ImmutableAttribute.converterFrom(ImmutableAttribute.java:167)
at software.amazon.awssdk.enhanced.dynamodb.mapper.ImmutableAttribute.resolve(ImmutableAttribute.java:163)
at software.amazon.awssdk.enhanced.dynamodb.mapper.StaticImmutableTableSchema.lambda$new$0(StaticImmutableTableSchema.java:153)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(StreamSpliterators.java:312)
at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734)
at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658)
at software.amazon.awssdk.enhanced.dynamodb.mapper.StaticImmutableTableSchema.(StaticImmutableTableSchema.java:159)
at software.amazon.awssdk.enhanced.dynamodb.mapper.StaticImmutableTableSchema.(StaticImmutableTableSchema.java:77)
at software.amazon.awssdk.enhanced.dynamodb.mapper.StaticImmutableTableSchema$Builder.build(StaticImmutableTableSchema.java:425)
at software.amazon.awssdk.enhanced.dynamodb.mapper.StaticTableSchema.(StaticTableSchema.java:66)
at software.amazon.awssdk.enhanced.dynamodb.mapper.StaticTableSchema.(StaticTableSchema.java:64)
at software.amazon.awssdk.enhanced.dynamodb.mapper.StaticTableSchema$Builder.build(StaticTableSchema.java:255)
at com.childcare.model.dyanmo.School.(School.java:180)
... 28 more

tkindra2 · 2021-02-04T22:00:15Z

@catchin @scmacdon @beckandros @bmaizels Would you know how to define Static table schema for structure in original post.

tkindra2 · 2021-02-07T20:04:21Z

Ok got this to working
So the type was getting erased at runtime so used EnhancedType.listOf(Customer class). Then it started giving errors on no converter found. So i had to implement custom converter.

In the custom converter i had to make sure there was no empty attribute, of lots of null checks.

.addAttribute(EnhancedType.listOf(GenericRecord.class),
					a -> a.name("recordMetadata").getter(Customer::getRecordMetadata)
							.setter(Customer::setRecordMetadata)
 							.attributeConverter(new ListOfRecordsConverter()))

Is there a way when defining static table schema to avoid defining custom converters for nested documents?

bmaizels · 2021-02-08T21:07:23Z

@tkindra2 There are two ways to do this without having to create a custom converter:

Pass in an enhanced type that uses a TableSchema you have already created for the nested class:

EnhancedType.documentOf(GenericRecord.class, TableSchema.fromClass(GenericRecord.class))

Use annotations and let it do all the work. In this case as long as your GenericRecord class is annotated with @DynamoDbBean or @DynamoDbImmutable and as long as the parent class is also annotated, and you are using TableSchema.fromClass(...) to construct your parent TableSchema it should 'just work'.

@DynamoDbBean
class ParentClass {
ChildClass getAttribute() {...} 
void setAttribute(ChildClass obj) {...}
}

@DynamoDbBean
class ChildClass {
...
}

TableSchema<ParentClass> tableSchema = TableSchema.fromClass(ParentClass.class);

tkindra2 · 2021-02-08T21:40:27Z

@bmaizels Thanks! I like the first option to minimize cold start, but

.addAttribute(EnhancedType.documentOf(GenericRecord.class,TableSchema.fromClass(GenericRecord.class)),
					a -> a.name("recordMetadata").getter(Customer::getRecordMetadata)
							.setter(Customer::setRecordMetadata)  )

The return type of Customer::getRecordMetadata is List not GenericRecord so i get compilation error. How do we fix this?

bmaizels · 2021-02-08T21:45:12Z

@tkindra2 EnhancedType.listOf(EnhancedType.documentOf(GenericRecord.class, genericRecordTableSchema)) maybe?

tkindra2 · 2021-02-08T21:54:59Z

@bmaizels Thanks! that worked!

Also i have nested documents, example if Generic record had another list of documentX, how would that be defined?

does TableSchema.fromClass scan nested @DynamoDbBean?

bmaizels · 2021-02-08T21:56:53Z

@tkindra2

Also i have nested documents, example if Generic record had another list of documentX, how would that be defined?

When you are creating the TableSchema of genericRecord, just do it the same way, use an EnhancedType.documentOf for the attribute definition.

does TableSchema.fromClass scan nested @DynamoDbBean?

Yes, although defining a StaticTableSchema directly does not. So if you want to use nested annotated classes you have to use an annotation scanner on the parent class. I didn't think you wanted to do annotation scanning, so I focused on giving you the direct static examples using EnhancedType.

tkindra2 · 2021-02-08T22:15:50Z

@bmaizels Thanks! I spent several hours getting this right. Could this be added as one of the examples.

Thank you again

scmacdon · 2021-02-08T22:19:09Z

Can you do a Pull Request against our Github repo here please: https://github.com/awsdocs/aws-doc-sdk-examples/tree/master/javav2/example_code/dynamodb Thxs From: tkindra2 <notifications@github.com> Sent: Monday, February 8, 2021 5:16 PM To: aws/aws-sdk-java-v2 <aws-sdk-java-v2@noreply.github.com> Cc: Macdonald, Scott <scmacdon@amazon.com>; Mention <mention@noreply.github.com> Subject: Re: [aws/aws-sdk-java-v2] Dyanmodb Enchanced client nested Objects as json (#2265) @bmaizels<https://github.com/bmaizels> Thanks! I spent several hours getting this right. Could this be added as one of the examples. Thank you again — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#2265 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ANUKOT4NXAF4BJ6ONQPCIBTS6BPCLANCNFSM4WZQZJLQ>.

kevcodez · 2021-02-10T19:13:57Z

I added @ DynamoDbBean to both classes. When retreiving the entry, all entries in the list are null, though, any idea?

Same case as above. ParentClass with the annotation, has a list of items by ChildClass (has the annotation too)

debora-ito · 2021-02-16T03:12:54Z

@tkindra2 thank you for the feedback, I agree that we should have more examples about this case, marking as a documentation issue.

@kevcodez I saw your comment in #2277 (comment), let's track this issue over there. Please provide a minimal working code we can use to reproduce the issue.

Prsna23 · 2023-02-22T06:35:16Z

Can you do a Pull Request against our Github repo here please: https://github.com/awsdocs/aws-doc-sdk-examples/tree/master/javav2/example_code/dynamodb Thxs From: tkindra2 notifications@github.com Sent: Monday, February 8, 2021 5:16 PM To: aws/aws-sdk-java-v2 aws-sdk-java-v2@noreply.github.com Cc: Macdonald, Scott scmacdon@amazon.com; Mention mention@noreply.github.com Subject: Re: [aws/aws-sdk-java-v2] Dyanmodb Enchanced client nested Objects as json (#2265) @bmaizels https://github.com/bmaizels Thanks! I spent several hours getting this right. Could this be added as one of the examples. Thank you again — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#2265 (comment)>, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ANUKOT4NXAF4BJ6ONQPCIBTS6BPCLANCNFSM4WZQZJLQ.

@scmacdon Can you please confirm if the example for having nested documents, with nested one being a list, has been added to the repo? It would be great to have one though

khayouge · 2023-09-21T09:58:31Z

@tkindra2 There are two ways to do this without having to create a custom converter:

Pass in an enhanced type that uses a TableSchema you have already created for the nested class:
EnhancedType.documentOf(GenericRecord.class, TableSchema.fromClass(GenericRecord.class))
Use annotations and let it do all the work. In this case as long as your GenericRecord class is annotated with @DynamoDbBean or @DynamoDbImmutable and as long as the parent class is also annotated, and you are using TableSchema.fromClass(...) to construct your parent TableSchema it should 'just work'.
@DynamoDbBean
class ParentClass {
ChildClass getAttribute() {...} 
void setAttribute(ChildClass obj) {...}
}

@DynamoDbBean
class ChildClass {
...
}

TableSchema<ParentClass> tableSchema = TableSchema.fromClass(ParentClass.class);

The second is the best way to do that.

tkindra2 added guidance Question that needs advice or information. needs-triage This issue or PR still needs to be triaged. labels Jan 29, 2021

debora-ito removed the needs-triage This issue or PR still needs to be triaged. label Feb 8, 2021

debora-ito added the dynamodb-enhanced label Feb 11, 2021

debora-ito added the documentation This is a problem with documentation. label Feb 16, 2021

debora-ito removed the guidance Question that needs advice or information. label Mar 19, 2022

yasminetalby added the p3 This is a minor priority issue label Nov 12, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dyanmodb Enchanced client nested Objects as json #2265

Dyanmodb Enchanced client nested Objects as json #2265

tkindra2 commented Jan 29, 2021

catchin commented Feb 2, 2021

tkindra2 commented Feb 2, 2021

tkindra2 commented Feb 4, 2021 •

edited

tkindra2 commented Feb 7, 2021 •

edited

bmaizels commented Feb 8, 2021 •

edited

tkindra2 commented Feb 8, 2021

bmaizels commented Feb 8, 2021

tkindra2 commented Feb 8, 2021

bmaizels commented Feb 8, 2021 •

edited

tkindra2 commented Feb 8, 2021

scmacdon commented Feb 8, 2021 via email

kevcodez commented Feb 10, 2021

debora-ito commented Feb 16, 2021

Prsna23 commented Feb 22, 2023

khayouge commented Sep 21, 2023

Dyanmodb Enchanced client nested Objects as json #2265

Dyanmodb Enchanced client nested Objects as json #2265

Comments

tkindra2 commented Jan 29, 2021

Steps to Reproduce

Current Behavior

Your Environment

catchin commented Feb 2, 2021

tkindra2 commented Feb 2, 2021

tkindra2 commented Feb 4, 2021 • edited

tkindra2 commented Feb 7, 2021 • edited

bmaizels commented Feb 8, 2021 • edited

tkindra2 commented Feb 8, 2021

bmaizels commented Feb 8, 2021

tkindra2 commented Feb 8, 2021

bmaizels commented Feb 8, 2021 • edited

tkindra2 commented Feb 8, 2021

scmacdon commented Feb 8, 2021 via email

kevcodez commented Feb 10, 2021

debora-ito commented Feb 16, 2021

Prsna23 commented Feb 22, 2023

khayouge commented Sep 21, 2023

tkindra2 commented Feb 4, 2021 •

edited

tkindra2 commented Feb 7, 2021 •

edited

bmaizels commented Feb 8, 2021 •

edited

bmaizels commented Feb 8, 2021 •

edited