Azure cosmos joins and persistence #1790

radovanradic · 2022-10-18T07:19:23Z

Added support for embedded and one to many joins, since this is only what makes sense in Cosmos Db documents. Cross document or container joins are not supported
Integrated with pre persist/post persist logic to populate created, updated date etc

...odel/src/main/java/io/micronaut/data/document/model/query/builder/CosmosSqlQueryBuilder.java

dstepanov · 2022-10-18T10:30:46Z

data-model/src/main/java/io/micronaut/data/model/query/builder/AbstractSqlLikeQueryBuilder.java

@@ -1085,7 +1130,7 @@ private void appendCriteriaForOperator(StringBuilder whereClause,
            boolean shouldEscape = propertyPath.shouldEscape();

            boolean[] needsTrimming = {false};
-            traversePersistentProperties(propertyPath.getAssociations(), propertyPath.getProperty(), (associations, property) -> {
+            traversePersistentProperties(propertyPath.getAssociations(), propertyPath.getProperty(), true, (associations, property) -> {


Looks like overkill adding a new parameter. Just refactor this code and extract: traverseForCriteriaPropertyPath(propertyPath, lambda)

This parameter is used for different traversal of properties. For Azure Cosmos, how I thought it could/should work, we don't traverse through embedded properties but select whole embedded field since this is how (de)serialization works. Except for where and composite id, then need to traverse them all for comparison (where settlementid.code = @p1 and settlementid.country.id == @p2)

Yeah, so just extract traverse methods based on the use-case so they can be modified in the parent class.

Alright, done in 5fd5483

dstepanov · 2022-10-18T10:33:00Z

Do we already run the document TCK tests?

radovanradic · 2022-10-18T10:36:26Z

Do we already run the document TCK tests?

Not sure what that means, but this does not support all the mappings there (join on other tables etc).

@SInCE

* Azure Cosmos POC * Azure Cosmos POC * Add CosmosSqlQueryBuilder * Override default naming strategy * Override default naming strategy * Updated tests package and added javadocs, copyright * Merge some changes from master and fix checkstyle. * Cosmos db config using micronaut azure lib (#1750) * Azure cosmos naming (#1760) * Allow overriding naming strategy in query builder. * Ignore Cosmos test for github * Moved cosmos query test to azure cosmos module * Revert "Moved cosmos query test to azure cosmos module" This reverts commit bdd349d. * Moved cosmos query test to azure cosmos module * Address review comment as possible. * Updated to call sql builder vs directly NamingStrategy * Move changes from NamingStrategy to CosmosSqlQueryBuilder * Undo formatting * Adding cosmos container annotation and map with entities (#1777) * Adding cosmos container annotation and map with entities * Changed annotations and mapping defs per code review comments. * Added test for simple queries * Updates per review suggestion, still need fixes * Fixed initialization bugs * Removed unused classes and configured db throughput * Add container settings to the database config * Fixed throughput settings for azure db and container. * Added shared logic to retrieve partition key. * Added test when db and containers already exist * Replace container properties if storage update requested. * Implementation for deletions in Azure Cosmos DB (#1784) * Implementation for deletions in Azure Cosmos DB * Support batch delete when there is partition key. * Changes in bulk deletions per code review suggestions * More changes per code review comments * Added some comments and logs * More PR changes per review comments. * Azure cosmos update (#1786) * Cosmos DB update operations * Simplified execute update and params resolving. * Change upsert to replace operation. * Verify update query contains fields to update * Fix logic for binding for update. * Azure cosmos read operations (#1787) * Cosmos DB read operations. * Added some fixes for update/delete and partition key logic * PartitionKey can be number, boolean or string. Updated @SInCE * Fixed issue with binding params for raw query * Added missing test and refactor code to remove Sonar/compiler warnings * Added tests for more custom result processing * Added more tests with custom type results * Azure cosmos joins and persistence (#1790) * Initial work for Cosmos relations, joins, embedded * Cosmos entities operations updates and fixed checkstyle * Updated implementation for multiple entities operations * Handle retrieval of Embedded fields, skip joins for now. * Support joins on document/container level * Fixed joins for projections and few other bugs * Added workaround for IN criteria - using ARRAY_CONTAINS * Fix and add test for generic IN criterion handler * Override criteria as suggested in code review * Changed logic around IN criteria for Cosmos Db * Fixed one more bug introduced in latest commit * Simplified code changes, per code review comment * Use constants * Fixed checkstyle and updated method visibility * Reactive operations for Cosmos db (#1797) * Added reactive operations for Cosmos Db * Update copyright, undo test change. * Updated test to verify reactive config flag * Updated misleading comment * Changed logic for reactive implementation as suggested in code review * Removed needless parameter. * Updated formatting * Removed formatting * Changes per code review comments * More code review suggested changes * Added back some exceptions * Added CosmosAccessException * Refactor per code review suggestion * Support for Azure Cosmos arrays and some functions (#1813) * Adding support for some Cosmos functions and arrays * Renamed method to generate SqlQuerySpec after binding * Updated custom functions impl and some small improvements * Improved null/not null criteria and tests * Added support for return Slice/Page and fixed some bugs * Extract serde logic to separate class and manually feed id property * Added test for entities with custom id field names * Fixed querying with partition key * Addressing comments from PR * Prepare query to use expandable query parts for IN/NOT IN * Create naming strategy for reuse * Validate identity field when inspecting partition key * Test @Autopopulated, @transient and @Version * More changes per code review comments * Removed unused test entities and repositories * Cleanup test code a bit * Cleanup some compiler/Sonar warnings * Added support for document _etag version and optimistic locking * Fixed bug when doing bulk operations * Populate etag version field upon entity save * Added test for invalid id type * Support for attribute converters * Renamed converter * Azure cosmos examples and docs (#1827) * Start working on azure-cosmos docs and examples * Added Azure Cosmos groovy example * Added Kotlin example for Azure Cosmos * Java and Kotlin Azure Cosmos examples updated * Completed example tests * Initial docs * Using ETag in Azure Cosmos Examples * Added attribute converter to examples * Added doc references to Azure Cosmos chapters * Azure cosmos docs and support for update specification * More docs and examples, criteria specifications docs * Extended examples and docs with array_contains * Fix bug with getting update query * Use typed queries to get update statement for Azure Cosmos * Removed Azure Cosmos from What's new * Update docs per PR comment * Changes per code review comments and additional cleanup. * Adjust tests and fixed binding for raw query. * Fixed binding for one element in collection. * Check if log level enabled before logging. * Updated @SInCE and added copyright where it was missing * Undo some inadvertent changes. * Removed unused import * Disable binary compatibility for the initial data-azure-cosmos version * Fixed some Sonar warnings. * Updated `@since` to next minor version * Skip cosmos query build tests for java8 Was failing due to: java.lang.NoClassDefFoundError: com/sun/source/util/JavacTask * Re-enable java8 test for query compilation * Adjust azure cosmos reactive implementation according to latest changes * Added `@since` and `@author` tags where were missing. Co-authored-by: Denis Stepanov <denis.s.stepanov@oracle.com>

@SInCE

* Azure Cosmos POC * Azure Cosmos POC * Add CosmosSqlQueryBuilder * Override default naming strategy * Override default naming strategy * Updated tests package and added javadocs, copyright * Merge some changes from master and fix checkstyle. * Cosmos db config using micronaut azure lib (micronaut-projects#1750) * Azure cosmos naming (micronaut-projects#1760) * Allow overriding naming strategy in query builder. * Ignore Cosmos test for github * Moved cosmos query test to azure cosmos module * Revert "Moved cosmos query test to azure cosmos module" This reverts commit bdd349d. * Moved cosmos query test to azure cosmos module * Address review comment as possible. * Updated to call sql builder vs directly NamingStrategy * Move changes from NamingStrategy to CosmosSqlQueryBuilder * Undo formatting * Adding cosmos container annotation and map with entities (micronaut-projects#1777) * Adding cosmos container annotation and map with entities * Changed annotations and mapping defs per code review comments. * Added test for simple queries * Updates per review suggestion, still need fixes * Fixed initialization bugs * Removed unused classes and configured db throughput * Add container settings to the database config * Fixed throughput settings for azure db and container. * Added shared logic to retrieve partition key. * Added test when db and containers already exist * Replace container properties if storage update requested. * Implementation for deletions in Azure Cosmos DB (micronaut-projects#1784) * Implementation for deletions in Azure Cosmos DB * Support batch delete when there is partition key. * Changes in bulk deletions per code review suggestions * More changes per code review comments * Added some comments and logs * More PR changes per review comments. * Azure cosmos update (micronaut-projects#1786) * Cosmos DB update operations * Simplified execute update and params resolving. * Change upsert to replace operation. * Verify update query contains fields to update * Fix logic for binding for update. * Azure cosmos read operations (micronaut-projects#1787) * Cosmos DB read operations. * Added some fixes for update/delete and partition key logic * PartitionKey can be number, boolean or string. Updated @SInCE * Fixed issue with binding params for raw query * Added missing test and refactor code to remove Sonar/compiler warnings * Added tests for more custom result processing * Added more tests with custom type results * Azure cosmos joins and persistence (micronaut-projects#1790) * Initial work for Cosmos relations, joins, embedded * Cosmos entities operations updates and fixed checkstyle * Updated implementation for multiple entities operations * Handle retrieval of Embedded fields, skip joins for now. * Support joins on document/container level * Fixed joins for projections and few other bugs * Added workaround for IN criteria - using ARRAY_CONTAINS * Fix and add test for generic IN criterion handler * Override criteria as suggested in code review * Changed logic around IN criteria for Cosmos Db * Fixed one more bug introduced in latest commit * Simplified code changes, per code review comment * Use constants * Fixed checkstyle and updated method visibility * Reactive operations for Cosmos db (micronaut-projects#1797) * Added reactive operations for Cosmos Db * Update copyright, undo test change. * Updated test to verify reactive config flag * Updated misleading comment * Changed logic for reactive implementation as suggested in code review * Removed needless parameter. * Updated formatting * Removed formatting * Changes per code review comments * More code review suggested changes * Added back some exceptions * Added CosmosAccessException * Refactor per code review suggestion * Support for Azure Cosmos arrays and some functions (micronaut-projects#1813) * Adding support for some Cosmos functions and arrays * Renamed method to generate SqlQuerySpec after binding * Updated custom functions impl and some small improvements * Improved null/not null criteria and tests * Added support for return Slice/Page and fixed some bugs * Extract serde logic to separate class and manually feed id property * Added test for entities with custom id field names * Fixed querying with partition key * Addressing comments from PR * Prepare query to use expandable query parts for IN/NOT IN * Create naming strategy for reuse * Validate identity field when inspecting partition key * Test @Autopopulated, @transient and @Version * More changes per code review comments * Removed unused test entities and repositories * Cleanup test code a bit * Cleanup some compiler/Sonar warnings * Added support for document _etag version and optimistic locking * Fixed bug when doing bulk operations * Populate etag version field upon entity save * Added test for invalid id type * Support for attribute converters * Renamed converter * Azure cosmos examples and docs (micronaut-projects#1827) * Start working on azure-cosmos docs and examples * Added Azure Cosmos groovy example * Added Kotlin example for Azure Cosmos * Java and Kotlin Azure Cosmos examples updated * Completed example tests * Initial docs * Using ETag in Azure Cosmos Examples * Added attribute converter to examples * Added doc references to Azure Cosmos chapters * Azure cosmos docs and support for update specification * More docs and examples, criteria specifications docs * Extended examples and docs with array_contains * Fix bug with getting update query * Use typed queries to get update statement for Azure Cosmos * Removed Azure Cosmos from What's new * Update docs per PR comment * Changes per code review comments and additional cleanup. * Adjust tests and fixed binding for raw query. * Fixed binding for one element in collection. * Check if log level enabled before logging. * Updated @SInCE and added copyright where it was missing * Undo some inadvertent changes. * Removed unused import * Disable binary compatibility for the initial data-azure-cosmos version * Fixed some Sonar warnings. * Updated `@since` to next minor version * Skip cosmos query build tests for java8 Was failing due to: java.lang.NoClassDefFoundError: com/sun/source/util/JavacTask * Re-enable java8 test for query compilation * Adjust azure cosmos reactive implementation according to latest changes * Added `@since` and `@author` tags where were missing. Co-authored-by: Denis Stepanov <denis.s.stepanov@oracle.com>

radovanradic added 6 commits October 14, 2022 09:58

Initial work for Cosmos relations, joins, embedded

47b3a35

Cosmos entities operations updates and fixed checkstyle

33c2481

Updated implementation for multiple entities operations

5165bda

Handle retrieval of Embedded fields, skip joins for now.

aed5fd1

Support joins on document/container level

debf3dc

Fixed joins for projections and few other bugs

6d5d2bf

radovanradic requested a review from dstepanov October 18, 2022 07:19

Added workaround for IN criteria - using ARRAY_CONTAINS

86244d6

radovanradic commented Oct 18, 2022

View reviewed changes

...odel/src/main/java/io/micronaut/data/document/model/query/builder/CosmosSqlQueryBuilder.java Outdated Show resolved Hide resolved

Fix and add test for generic IN criterion handler

da87784

dstepanov requested changes Oct 18, 2022

View reviewed changes

Override criteria as suggested in code review

f939cb0

radovanradic added 3 commits October 18, 2022 12:48

Changed logic around IN criteria for Cosmos Db

350b6ca

Fixed one more bug introduced in latest commit

9daa987

Simplified code changes, per code review comment

5fd5483

radovanradic requested a review from dstepanov October 18, 2022 11:59

radovanradic added 2 commits October 18, 2022 14:02

Use constants

b3bdd1a

Fixed checkstyle and updated method visibility

69131c7

dstepanov approved these changes Oct 24, 2022

View reviewed changes

radovanradic merged commit d5a766e into azure-cosmos Oct 24, 2022

radovanradic deleted the azure-cosmos-joins-persistence branch October 24, 2022 08:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Azure cosmos joins and persistence #1790

Azure cosmos joins and persistence #1790

radovanradic commented Oct 18, 2022

dstepanov Oct 18, 2022

radovanradic Oct 18, 2022

dstepanov Oct 18, 2022

radovanradic Oct 18, 2022

dstepanov commented Oct 18, 2022

radovanradic commented Oct 18, 2022

Azure cosmos joins and persistence #1790

Azure cosmos joins and persistence #1790

Conversation

radovanradic commented Oct 18, 2022

dstepanov Oct 18, 2022

Choose a reason for hiding this comment

radovanradic Oct 18, 2022

Choose a reason for hiding this comment

dstepanov Oct 18, 2022

Choose a reason for hiding this comment

radovanradic Oct 18, 2022

Choose a reason for hiding this comment

dstepanov commented Oct 18, 2022

radovanradic commented Oct 18, 2022