associated_data_implicit_conversion.md

title	perex	date	author	proofreading
Associated data implicit conversion process	Related data represents complex unstructured or semi-structured documents that can be automatically converted from/to Java POJO classes automatically using implicit conversion mechanisms.	15.12.2022	Ing. Jan Novotný	done

The complex types are all types that don't qualify as simple evitaDB types (or an array of simple evitaDB types) and don't belong to a java package (i.e. java.lang.URL is forbidden to be stored in evitaDB, even if it is Serializable, because is in the package java, and is not directly supported by the basic data types). The complex types are targeted for the client POJO classes to carry bigger data or associate simple logic along with the data.

Associated data may even contain array of POJOs. Such data will be automatically converted to an array of `ComplexDataObject` types - i.e. `ComplexDataObject[]`.

The complex type can contain the properties of:

• any simple evitaDB types
• any other complex types (additional inner POJOs)
• generic Lists
• generic Sets
• generic Maps
• any array of simple evitaDB types or complex types

Collection generics must be resolvable to an exact class (meaning that wildcard generics are not supported). The complex type may also be an immutable class, accepting properties via the constructor parameters. Immutable classes must be compiled with the javac -parameters argument, and their names in the constructor must match their property names of the getter fields. This plays really well with Lombok @Data annotation.

Serialization

Storing a complex type to entity is executed as follows:

new InitialEntityBuilder('product')
	.setAssociatedData('stockAvailability', new ProductStockAvailabilityDTO());

All properties that comply with JavaBean naming rules and have both an accessor, a mutator method (i.e. get and set methods for the property) and are not annotated with NonSerializedData.java annotation, are serialized into a complex type. See the following example:

public class ProductStockAvailabilityDTO implements Serializable {
    private int id;
    private String stockName;
    @NonSerializedData
    private URL stockUrl;
    private URL stockMotive;
    
    // id gets serialized - both methods are present and are valid JavaBean property methods
    public int getId() { return id; }
    public void setId(int id) { this.id = id; }
    
    // id gets serialized - both methods are present and are valid JavaBean property methods
    public String getStockName() { return stockName; }
    public void setStockName(String stockName) { this.stockName = stockName; }
    
    // stockUrl will not be serialized - corresponding field is annotated with @NonSerializedData
    public URL getStockUrl() { return stockUrl; }
    public void setStockUrl(URL stockUrl) { this.stockUrl = stockUrl; }
    
    // active will not be serialized - it has no corresponding mutator method
    public isActive() { return false; }
    
    // stock motive will not be serialized because getter method is marked with @NonSerializedData
    @NonSerializedData
    public URL getStockMotive() { return stockMotive; }
    public void setStockMotive(URL stockMotive) { this.stockMotive = stockMotive; }
    
}

As you can see, annotations can be placed either on methods or property fields, so that if you use Lombok support, you can still easily define the class:

@Data
public class ProductStockAvailabilityDTO implements Serializable {
    private int id;
    private String stockName;
    @NonSerializedData private URL stockUrl;
    @NonSerializedData private URL stockMotive;
    
    public isActive() { return false; }
}

If the serialization process encounters any property that cannot be serialized, the SerializationFailedException.java is thrown.

Generic collections

You can use collections in complex types, but the specific collection types must be extractable from the collection generics in deserialization time. Look at the following example:

@Data
public class SomeDataWithCollections implements Serializable {
    private List<String> names;
    private Map<String, Integer> index;
    private Set<BigDecimal> amounts;
    private SomeDataWithCollections[] innerContainers;
}

Recommended test coverage

Because methods that don't follow the JavaBeans contract are silently skipped, it is highly recommended to always store and retrieve associated data in the unit test and check that all important data is actually stored:

@Test
void verifyProductStockAvailabilityDTOIsProperlySerialized() {
    final EntityBuilder entity = new InitialEntityBuilder('product');
    final ProductStockAvailabilityDTO stockAvailabilityBeforeStore = new ProductStockAvailabilityDTO(); 
    entity.setAssociatedData('stockAvailability', stockAvailabilityBeforeStore);
    final SealedEntity loadedEntity = entity(); //some custom logic to load proper entity
    final ProductStockAvailabilityDTO stockAvailabilityAfterLoad = loadedEntity.getAssociatedData(
        'stockAvailability', ProductStockAvailabilityDTO.class
    );
    assertEquals(
        stockAvailabilityBeforeStore, stockAvailabilityAfterLoad, 
        "ProductStockAvailabilityDTO was not entirely serialized!"
    );
}

Deserialization, model evolution support

Retrieving a complex type from an entity is executed as follows:

final SealedEntity entity = entity(); //some custom logic to load proper entity
final ProductStockAvailabilityDTO stockAvailability = entity.getAssociatedData(
    'stockAvailability', ProductStockAvailabilityDTO.class
);

Complex types are internally converted to a ComplexDataObject.java type, that can be safely stored in evitaDB storage. The (de)serialization process is also designed to prevent data loss, and allow model evolution.

The deserialization process may fail with two exceptions:

• UnsupportedDataTypeException.java is raised when certain property cannot be deserialized due to an incompatibility with the specified contract
• IncompleteDeserializationException.java is raised when any of the serialized data was not deserialized due to a lack of a mutator method on the class it's being converted to

Field removal

The IncompleteDeserializationException.java exception protects developers from unintentional data loss by making a mistake in the Java model and then executing:

• a fetch of existing complex type
• altering a few properties
• storing it back again to evitaDB

If there is legal reason for dropping some data stored along with its complex type in the previous versions of the application, you can use DiscardedData.java annotation on any complex type class to declare that it is ok to throw away data during deserialization.

Example:

Associated data were stored with this class definition:

@Data
public class ProductStockAvailabilityDTO implements Serializable {
    private int id;
    private String stockName;
}

In future versions, developer will decide that the id field is not necessary anymore and may be dropped. But there is a lot of data written by the previous version of the application. So, when dropping a field, we need to make a note for evitaDB that the presence of any id data is ok, even if there is no field for it anymore. This data will be discarded when the associated data gets rewritten by the new version of the class:

@Data
@DiscardedData("id");
public class ProductStockAvailabilityDTO implements Serializable {
    private String stockName;
}

Field renaming and controlled migration

There are also situations when you need to rename the field (for example you made a typo in the previous version of the Java Bean type). In such case you'd also experience the IncompleteDeserializationException.java when you try to deserialize the type with the corrected Java Bean definition. In this situation, you can use the RenamedData.java annotation to migrate old versions of data.

Example:

First version of the Java type with the mistake:

@Data
public class ProductStockAvailabilityDTO implements Serializable {
    private String stockkName;
}

Next time we'll try to fix the typo:

@Data
public class ProductStockAvailabilityDTO implements Serializable {
    @RenamedData("stockkName")
    private String stockname;
}

But we make yet another mistake, so we need another correction:

@Data
public class ProductStockAvailabilityDTO implements Serializable {
    @RenamedData({"stockkName", "stockname"})
    private String stockName;
}

We may get rid of those annotations when we're confident there is no data with the old contents in evitaDB. Annotation RenamedData.java can also be used for model evolution - i.e. automatic translation of an old data format to the new one.

Example:

Old model:

@Data
public class ProductStockAvailabilityDTO implements Serializable {
    private String stockName;
}

New model:

@Data
public class ProductStockAvailabilityDTO implements Serializable {
    private String upperCasedStockName;
    
    @RenamedData
    public void setStockName(String stockName) {
        this.upperCasedStockName = stockName == null ? null : stockName.toUpperCase();
    }
}