feat: add private properties to contract definition entity #3534

mohannad-ezzo · 2023-10-12T15:04:40Z

What this PR changes/adds

Add private properties to contract definition entity. The implementation supports in-memory and postgres as a store.

Why it does that

The private properties are required to provide the owner with the possibility to add extra properties which should not be exposed outside. This requirement was initiated when we want to add administrative properties like (createdBy, createdAt, chnagedBy, chnagedAt). We assume that the private properties can be used to store such a requirement after registering a listener on the contract definition.
More information is in this discussion:
#2592 (reply in thread)

Further notes

List other areas of code that have changed but are not necessarily linked to the main feature. This could be method
signature changes, package declarations, bugs that were encountered and were fixed inline, etc.

Linked Issue(s)

Closes #3490

…erties

…ohannad-ezzo/Connector into feat-contract-def-private-prop

github-actions

We are always happy to welcome new contributors ❤️ To make things easier for everyone, please make sure to follow our contribution guidelines, check if you have already signed the ECA, and relate this pull request to an existing issue or discussion.

...r/api/management/contractdefinition/transform/JsonObjectToContractDefinitionTransformer.java

paullatzelsperger · 2023-10-13T04:49:13Z

extensions/control-plane/store/sql/contract-definition-store-sql/docs/schema.sql

+    property_name  VARCHAR(255) NOT NULL,
+    property_value TEXT         NOT NULL,
+    property_type  VARCHAR(255) NOT NULL,
+    property_is_private BOOLEAN,


not needed, since contract definitions can only have private properties.

I think this was copied by the sql asset index implementation, but to be fair I'd prefer to follow the transfer process one, where the private properties are serialized into a json field.

I followed the same pattern which is used for Asset. Therefore, I followed this approach. In addition, I intended to add property_is_private as a preparation for adding non-private properties for the contract definition. To be honest, our usecase for contract definition is just to have private properties, but not for non-private properties. So, as requested, the property property_is_private is removed and the corresponding code is adjusted.

.../java/org/eclipse/edc/connector/store/sql/contractdefinition/SqlContractDefinitionStore.java

...pse/edc/connector/contract/spi/testfixtures/offer/store/ContractDefinitionStoreTestBase.java

.../java/org/eclipse/edc/connector/store/sql/contractdefinition/SqlContractDefinitionStore.java

...src/test/java/org/eclipse/edc/connector/contract/spi/types/offer/ContractDefinitionTest.java

paullatzelsperger · 2023-10-13T05:09:01Z

@mohannad-ezzo two more things:

in a PR description, it's fine to reference a discussion, but briefly stating the reason in the desc. is still required so that reviewers don't have to read long threads of discussions.
Also in the PR description, pls use closing keywords when linking the issue, so it gets auto-closed when your PR is merged. I have already done that here.

codecov-commenter · 2023-10-13T05:22:22Z

Codecov Report

Attention: 5 lines in your changes are missing coverage. Please review.

Comparison is base (cba9558) 72.49% compared to head (ba282ff) 72.32%.
Report is 3 commits behind head on main.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3534      +/-   ##
==========================================
- Coverage   72.49%   72.32%   -0.17%     
==========================================
  Files         877      880       +3     
  Lines       17474    17729     +255     
  Branches      994     1021      +27     
==========================================
+ Hits        12667    12822     +155     
- Misses       4391     4471      +80     
- Partials      416      436      +20

Files	Coverage Δ
...ractdefinition/ContractDefinitionApiExtension.java	`90.00% <100.00%> (+1.11%)`	⬆️
...m/JsonObjectFromContractDefinitionTransformer.java	`100.00% <100.00%> (ø)`
...orm/JsonObjectToContractDefinitionTransformer.java	`100.00% <100.00%> (+9.09%)`	⬆️
...contractdefinition/SqlContractDefinitionStore.java	`89.13% <100.00%> (+2.82%)`	⬆️
...actdefinition/schema/BaseSqlDialectStatements.java	`96.15% <100.00%> (-3.85%)`	⬇️
...efinition/schema/ContractDefinitionStatements.java	`100.00% <100.00%> (ø)`
...r/contract/spi/types/offer/ContractDefinition.java	`65.90% <54.54%> (-2.67%)`	⬇️

... and 16 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

ndr-brt

generally, I'd follow the SqlTransferProcessStore approach, putting the properties into a JSON column into the same entity table, it will save a lot of code lines and unneeded overcomplication.

.../eclipse/edc/connector/api/management/contractdefinition/ContractDefinitionApiExtension.java

...api/management/contractdefinition/transform/JsonObjectFromContractDefinitionTransformer.java

...r/api/management/contractdefinition/transform/JsonObjectToContractDefinitionTransformer.java

...management/contractdefinition/transform/JsonObjectFromContractDefinitionTransformerTest.java

...i/management/contractdefinition/transform/JsonObjectToContractDefinitionTransformerTest.java

ndr-brt · 2023-10-13T12:24:25Z

extensions/control-plane/store/sql/contract-definition-store-sql/docs/schema.sql

+    property_name  VARCHAR(255) NOT NULL,
+    property_value TEXT         NOT NULL,
+    property_type  VARCHAR(255) NOT NULL,
+    property_is_private BOOLEAN,


I think this was copied by the sql asset index implementation, but to be fair I'd prefer to follow the transfer process one, where the private properties are serialized into a json field.

.../java/org/eclipse/edc/connector/store/sql/contractdefinition/SqlContractDefinitionStore.java

...src/test/java/org/eclipse/edc/connector/contract/spi/types/offer/ContractDefinitionTest.java

paullatzelsperger · 2023-10-16T07:56:52Z

generally, I'd follow the SqlTransferProcessStore approach, putting the properties into a JSON column into the same entity table, it will save a lot of code lines and unneeded overcomplication.

I agree, that is an even easier way way to do it. @mohannad-ezzo can you adapt your PR accordingly?
@ndr-brt in a future issue, maybe we should also change the SQL-persistence of Assets so that is follows the JSON approach, wdyt?

mohannad-ezzo · 2023-10-16T09:08:48Z

generally, I'd follow the SqlTransferProcessStore approach, putting the properties into a JSON column into the same entity table, it will save a lot of code lines and unneeded overcomplication.

I agree, that is an even easier way way to do it. @mohannad-ezzo can you adapt your PR accordingly? @ndr-brt in a future issue, maybe we should also change the SQL-persistence of Assets so that is follows the JSON approach, wdyt?

@paullatzelsperger @ndr-brt By following the JSON approach, we may loose the flexibility to support search within the private properties across all contract definitions. For sure, for a small set of contract definitions, JSON approach would still work. However, whenever the set becomes bigger and bigger, cross search would be difficult. I still see that the current approach may fit better our current and future requirement. What do you think ?

paullatzelsperger · 2023-10-16T19:02:38Z

@paullatzelsperger @ndr-brt By following the JSON approach, we may loose the flexibility to support search within the private properties across all contract definitions. For sure, for a small set of contract definitions, JSON approach would still work. However, whenever the set becomes bigger and bigger, cross search would be difficult. I still see that the current approach may fit better our current and future requirement. What do you think ?

@mohannad-ezzo Postgres supports JSON operators, and EDC has a way of converting a QuerySpec into such an operator.

mohannad-ezzo · 2023-10-20T15:12:42Z

@paullatzelsperger @ndr-brt By following the JSON approach, we may loose the flexibility to support search within the private properties across all contract definitions. For sure, for a small set of contract definitions, JSON approach would still work. However, whenever the set becomes bigger and bigger, cross search would be difficult. I still see that the current approach may fit better our current and future requirement. What do you think ?

@mohannad-ezzo Postgres supports JSON operators, and EDC has a way of converting a QuerySpec into such an operator.

@paullatzelsperger @ndr-brt ok, I am going to follow the JSON approach of storing the private properties (instead of a having a separate table) and provide you with an update asap.

ndr-brt · 2023-10-20T15:54:27Z

generally, I'd follow the SqlTransferProcessStore approach, putting the properties into a JSON column into the same entity table, it will save a lot of code lines and unneeded overcomplication.

I agree, that is an even easier way way to do it. @mohannad-ezzo can you adapt your PR accordingly? @ndr-brt in a future issue, maybe we should also change the SQL-persistence of Assets so that is follows the JSON approach, wdyt?

@paullatzelsperger @ndr-brt By following the JSON approach, we may loose the flexibility to support search within the private properties across all contract definitions. For sure, for a small set of contract definitions, JSON approach would still work. However, whenever the set becomes bigger and bigger, cross search would be difficult. I still see that the current approach may fit better our current and future requirement. What do you think ?

I agree with paul, plus, using the "separate table" approach will lead to issues in trying to query nested properties (see #3424 )

mohannad-ezzo · 2023-10-24T08:49:17Z

As discussed I adapted the PR so that JSON is used to store private properties of contract definition instead of storing them in a separate table.

ndr-brt

Just avoid those two not necessary changes to the sql store and it's ready to be merged

.../java/org/eclipse/edc/connector/store/sql/contractdefinition/SqlContractDefinitionStore.java

mohannad-ezzo · 2023-10-24T12:49:32Z

Just avoid those two not necessary changes to the sql store and it's ready to be merged

Thank you for mentioning this. This actually was inherited from the old logic when I started to have another table for private properties. Old logic is back now :)

mohannad-ezzo · 2023-10-24T15:11:34Z

End-To-End-Tests and Component-Tests failed now. They were ok before the last commit.
Is there any action still required from my side?

ndr-brt · 2023-10-24T15:18:07Z

@mohannad-ezzo you need to fix the broken tests and the checkstyle as well

mohannad-ezzo · 2023-10-24T17:26:54Z

@mohannad-ezzo you need to fix the broken tests and the checkstyle as well

I just fixed the checksytle issues. Regarding the broken tests, they ran fine this time without any change from me.
As of now, ALL checks are green and have passed :)

paullatzelsperger

LGTM!

mohannad-ezzo added 12 commits September 22, 2023 16:30

feat: add private properties to contract definition

182aabb

feat: adjust JsonObjectFromContractDefinition transformer

4ebadf1

feat: adjust JsonObjectToContractDefinition transformer

f261809

adjust sql schemam and ER diagram

b1edd63

feat create contract definition with private properties

9e430ff

Merge branch 'eclipse-edc:main' into feat-contract-def-private-prop

96ed8b6

feat: add possibility to update contract definition with private prop…

951cbb7

…erties

Merge branch 'feat-contract-def-private-prop' of https://github.com/m…

f02e148

…ohannad-ezzo/Connector into feat-contract-def-private-prop

feat: add possibility to find contract by id with private properties

bb47b3d

feat: retrieve properties when triggering findAll

e4d604b

feat: remove not required code

9fb44d0

Merge branch 'eclipse-edc:main' into feat-contract-def-private-prop

e66c720

mohannad-ezzo requested a review from paullatzelsperger as a code owner October 12, 2023 15:04

mohannad-ezzo marked this pull request as draft October 12, 2023 15:05

github-actions bot reviewed Oct 12, 2023

View reviewed changes

jimmarino requested review from jimmarino and ndr-brt October 12, 2023 15:33

mohannad-ezzo marked this pull request as ready for review October 12, 2023 16:05

paullatzelsperger requested changes Oct 13, 2023

View reviewed changes

paullatzelsperger added enhancement New feature or request core feature labels Oct 13, 2023

paullatzelsperger assigned paullatzelsperger and unassigned paullatzelsperger Oct 13, 2023

Merge branch 'eclipse-edc:main' into feat-contract-def-private-prop

c486717

ndr-brt requested changes Oct 13, 2023

View reviewed changes

mohannad-ezzo added 3 commits October 13, 2023 15:30

fix: address reviewers comments

3dcaec7

Merge branch 'eclipse-edc:main' into feat-contract-def-private-prop

a260e1a

fix: add more test cases

09a9e83

mohannad-ezzo added 2 commits October 15, 2023 16:55

fix: remove isPrivate from contract definition property table

4682385

fix: remove not required import statements

d30401f

mohannad-ezzo requested a review from paullatzelsperger October 15, 2023 15:58

fix: handle checkstyle issues

a997c28

mohannad-ezzo added 4 commits October 23, 2023 11:43

fix: store private properties as JSON instead of having a separate table

b253e75

Merge branch 'eclipse-edc:main' into feat-contract-def-private-prop

db7fc6d

fix: checkstyle and test cases

6629cc5

fix: one more checktyle in import statements

be2076d

mohannad-ezzo requested a review from ndr-brt October 24, 2023 08:49

fix: additional checkstyle issue - avoid import star

ba282ff

ndr-brt reviewed Oct 24, 2023

View reviewed changes

.../java/org/eclipse/edc/connector/store/sql/contractdefinition/SqlContractDefinitionStore.java Outdated Show resolved Hide resolved

.../java/org/eclipse/edc/connector/store/sql/contractdefinition/SqlContractDefinitionStore.java Outdated Show resolved Hide resolved

fix: return the old behavior for for findById and FindAll

1ee6dcd

mohannad-ezzo requested a review from ndr-brt October 24, 2023 12:49

ndr-brt approved these changes Oct 24, 2023

View reviewed changes

fix: checkstyle - indentation issue

c696c85

mohannad-ezzo requested a review from ndr-brt October 24, 2023 17:23

paullatzelsperger approved these changes Oct 25, 2023

View reviewed changes

ndr-brt merged commit 059c668 into eclipse-edc:main Oct 25, 2023
17 checks passed

ndr-brt added the breaking-change Will require manual intervention for version update label Nov 14, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add private properties to contract definition entity #3534

feat: add private properties to contract definition entity #3534

mohannad-ezzo commented Oct 12, 2023 •

edited

github-actions bot left a comment

paullatzelsperger Oct 13, 2023

ndr-brt Oct 13, 2023

mohannad-ezzo Oct 15, 2023

paullatzelsperger commented Oct 13, 2023 •

edited

codecov-commenter commented Oct 13, 2023 •

edited

ndr-brt left a comment

ndr-brt Oct 13, 2023

paullatzelsperger commented Oct 16, 2023 •

edited

mohannad-ezzo commented Oct 16, 2023

paullatzelsperger commented Oct 16, 2023 •

edited

mohannad-ezzo commented Oct 20, 2023

ndr-brt commented Oct 20, 2023

mohannad-ezzo commented Oct 24, 2023 •

edited

ndr-brt left a comment

mohannad-ezzo commented Oct 24, 2023

mohannad-ezzo commented Oct 24, 2023

ndr-brt commented Oct 24, 2023

mohannad-ezzo commented Oct 24, 2023

paullatzelsperger left a comment

feat: add private properties to contract definition entity #3534

feat: add private properties to contract definition entity #3534

Conversation

mohannad-ezzo commented Oct 12, 2023 • edited

What this PR changes/adds

Why it does that

Further notes

Linked Issue(s)

github-actions bot left a comment

Choose a reason for hiding this comment

paullatzelsperger Oct 13, 2023

Choose a reason for hiding this comment

ndr-brt Oct 13, 2023

Choose a reason for hiding this comment

mohannad-ezzo Oct 15, 2023

Choose a reason for hiding this comment

paullatzelsperger commented Oct 13, 2023 • edited

codecov-commenter commented Oct 13, 2023 • edited

Codecov Report

ndr-brt left a comment

Choose a reason for hiding this comment

ndr-brt Oct 13, 2023

Choose a reason for hiding this comment

paullatzelsperger commented Oct 16, 2023 • edited

mohannad-ezzo commented Oct 16, 2023

paullatzelsperger commented Oct 16, 2023 • edited

mohannad-ezzo commented Oct 20, 2023

ndr-brt commented Oct 20, 2023

mohannad-ezzo commented Oct 24, 2023 • edited

ndr-brt left a comment

Choose a reason for hiding this comment

mohannad-ezzo commented Oct 24, 2023

mohannad-ezzo commented Oct 24, 2023

ndr-brt commented Oct 24, 2023

mohannad-ezzo commented Oct 24, 2023

paullatzelsperger left a comment

Choose a reason for hiding this comment

mohannad-ezzo commented Oct 12, 2023 •

edited

paullatzelsperger commented Oct 13, 2023 •

edited

codecov-commenter commented Oct 13, 2023 •

edited

paullatzelsperger commented Oct 16, 2023 •

edited

paullatzelsperger commented Oct 16, 2023 •

edited

mohannad-ezzo commented Oct 24, 2023 •

edited