feat(auth): Fine grained ownership policies #7499

skrydal · 2023-03-04T20:40:42Z

This PR introduces a new optional field for DataHubPolicyInfo - resourceOwnersType. It can only be used with resourceOwners flag set to True. Currently, any owner of the resource will match policy with resourceOwners, this change introduces possibility to specify which particular "types" (i.e. TECHNICAL_OWNER, BUSINESS_OWNER) of ownership will be considered for particular policy - making it possible to, for example, grant rights to change tags of datasets to only their technical owners but not business owners. This is an actual user requirement described here: https://feature-requests.datahubproject.io/p/authorization-policy-restricted-to-_eg_-technical_owners-owners

Beside changes in the PolicyEngine and other parts of backend also UI was amended to allow for new capability to be set as show below:

Policy view pop-up also shows new capabilities:

visible in UI and enforced for users (no groups)

…icies

aditya-radhakrishnan · 2023-03-08T03:53:32Z

Thank you for the contribution! Let me discuss this with the core DataHub team and then get back to you :)

skrydal · 2023-03-08T13:07:13Z

Thank you for the contribution! Let me discuss this with the core DataHub team and then get back to you :)

Appreciate it, let me know if you need any input from me.

aditya-radhakrishnan · 2023-03-16T17:00:55Z

Hi there, after talking with the core DataHub team, we're thinking of doing the following:

We're actually introducing the ability to have custom Ownership Types very soon (cc @pedro93), and we want to make sure this contribution covers that case as well.

Would it be okay with you if we held off on merging this in until custom Ownership Types are supported? At that point either we can work with you on accounting for that case, or we can take over getting this to the finish line.

Let me know your thoughts and thank you so much for this contribution!

skrydal · 2023-03-16T19:35:51Z

Hi there, after talking with the core DataHub team, we're thinking of doing the following:

We're actually introducing the ability to have custom Ownership Types very soon (cc @pedro93), and we want to make sure this contribution covers that case as well.

Would it be okay with you if we held off on merging this in until custom Ownership Types are supported? At that point either we can work with you on accounting for that case, or we can take over getting this to the finish line.

Let me know your thoughts and thank you so much for this contribution!

Hello @aditya-radhakrishnan, thank you for monitoring this issue. Let's wait for the custom Ownership Types then and please let me know once the PR can be updated.

aditya-radhakrishnan · 2023-03-16T21:42:29Z

Sounds good, will definitely do so!

skrydal · 2023-03-31T08:08:59Z

Hello @aditya-radhakrishnan I wanted to ask how is the dependency (custom Ownership Types) doing? It seems it is implemented by this PR: #7623 ? Is it blocked by something?

jjoyce0510 · 2023-06-20T22:35:57Z

datahub-graphql-core/src/main/java/com/linkedin/datahub/graphql/GmsGraphQLEngine.java

@@ -1622,6 +1622,9 @@ private void configurePolicyResolvers(final RuntimeWiring.Builder builder) {
        })).dataFetcher("resolvedRoles", new LoadableTypeBatchResolver<>(dataHubRoleType, (env) -> {
            final ActorFilter filter = env.getSource();
            return filter.getRoles();
+        })).dataFetcher("resolvedOwnershipTypes", new LoadableTypeBatchResolver<>(ownershipType, (env) -> {
+            final ActorFilter filter = env.getSource();
+            return filter.getResourceOwnersTypesUrns();


Can you confirm that we will resolve the entire "OwnershipType" entity from this? It appears that we are just mapping to the URNs themselves, as opposed to the OwnershipType objects that are required in the data model

I think it does, how else would UI work properly? Here is query and the output screensho (please let me know if I misunderstood you)t:

jjoyce0510 · 2023-06-20T22:36:09Z

datahub-graphql-core/src/main/resources/entity.graphql

@@ -8093,6 +8097,8 @@ input ActorFilterInput {
  """
  resourceOwners: Boolean!

+  resourceOwnersTypesUrns: [String!]


Same comments here.

jjoyce0510 · 2023-06-20T22:37:06Z

datahub-web-react/src/app/permissions/policy/PolicyActorForm.tsx

    // Toggle the "Owners" switch
    const onToggleAppliesToOwners = (value: boolean) => {
        setActors({
            ...actors,
            resourceOwners: value,
+            resourceOwnersTypesUrns: value ? actors.resourceOwnersTypesUrns : null,


Same as above - we can remove the "urns" suffix in all of these names

skrydal · 2023-06-21T11:42:26Z

@jjoyce0510 thank you for the review. I applied requested changes and provided the screenshot with a proof that ownership types resolution works. Is there anything left?

skrydal · 2023-06-26T10:31:20Z

I run some performance tests with this feature using locust and it is pretty much fine (note it was done on one host), sharing here (rather limited as everything ran on the same host):

[2023-06-26 12:14:46,550] skrydal-vm/INFO/locust.main: Starting web interface at http://0.0.0.0:8089 (accepting connections from all network interfaces)
[2023-06-26 12:14:46,555] skrydal-vm/INFO/locust.main: Starting Locust 2.15.1
[2023-06-26 12:14:46,555] skrydal-vm/INFO/locust.main: Run time limit set to 150 seconds
[2023-06-26 12:14:46,557] skrydal-vm/INFO/locust.runners: Ramping to 100 users at a rate of 2.00 per second
[2023-06-26 12:15:35,596] skrydal-vm/INFO/locust.runners: All users spawned: {"IngestUser": 100} (100 total users)
[2023-06-26 12:17:16,408] skrydal-vm/INFO/locust.main: --run-time limit reached, stopping test
[2023-06-26 12:17:17,413] skrydal-vm/INFO/locust.main: --autoquit time reached, shutting down
[2023-06-26 12:17:17,413] skrydal-vm/INFO/locust.main: Shutting down (exit code 0)
Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST     /entities?action=ingest                                                        12375     0(0.00%) |     15       3     161     13 |   82.58        0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
         Aggregated                                                                     12375     0(0.00%) |     15       3     161     13 |   82.58        0.00

Response time percentiles (approximated)
Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
POST     /entities?action=ingest                                                                13     15     17     19     25     32     44     56    100    130    160  12375
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
         Aggregated                                                                             13     15     17     19     25     32     44     56    100    130    160  12375

@jjoyce0510 are there still any concerns about the feature?

Change copy in the details modal

jjoyce0510

Thanks for taking the time to address comments! The PR is very nearly ready. There is one thing I'd missed in my previous review, specifically the comment about moving the logic for "resolving" the specific owner types for the resource from the PolicyEngine into a ResolvedResourceSpec object.

This basically allows us to lazily do the lookup in a way that is not visible to the Policy Engine, keeping the responsibility of evaluating policies and fetching the facts required to evaluate a policy separate

Once this change is addressed, we will be ready to merge this PR!

Cheers
John

jjoyce0510 · 2023-06-28T15:53:22Z

datahub-graphql-core/src/main/resources/entity.graphql

@@ -8130,6 +8140,11 @@ input ActorFilterInput {
  """
  resourceOwners: Boolean!

+  """


Thank you for adding these!

jjoyce0510 · 2023-06-28T16:01:27Z

metadata-service/auth-impl/src/main/java/com/datahub/authorization/PolicyEngine.java

+    Urn entityUrn = UrnUtils.getUrn(resourceSpec.getResource());
+    EnvelopedAspect ownershipAspect;
+    try {
+      EntityResponse response = _entityClient.getV2(entityUrn.getEntityType(), entityUrn,


The final comment from me - I'd really prefer to not see this logic in the Policy Engine. This can be abstracted away to the ResolvedResourceSpec and handled internally.

The policy engine should simply ask for the ownership types for the resouce and get it back without knowing how that happens

jjoyce0510 · 2023-06-28T16:06:48Z

Also note that I made a small copy change in one of the files, you may need to pull-then-push your changes!

sgomezvillamor · 2023-06-30T15:37:49Z

Thanks @jjoyce0510 for the review

Correct me if I’m wrong basically, you are proposing to move this logic here

datahub/metadata-service/auth-impl/src/main/java/com/datahub/authorization/PolicyEngine.java

Lines 323 to 332 in 3077809

    
             EntityResponse response = _entityClient.getV2(entityUrn.getEntityType(), entityUrn, 
        
                     Collections.singleton(Constants.OWNERSHIP_ASPECT_NAME), _systemAuthentication); 
        
             if (response == null || !response.getAspects().containsKey(Constants.OWNERSHIP_ASPECT_NAME)) { 
        
               return Collections.emptySet(); 
        
             } 
        
             ownershipAspect = response.getAspects().get(Constants.OWNERSHIP_ASPECT_NAME); 
        
           } catch (Exception e) { 
        
             log.error("Error while retrieving ownership aspect for urn {}", entityUrn, e); 
        
             return Collections.emptySet(); 
        
           }

to the ResolvedResourceSpec class

https://github.com/datahub-project/datahub/blob/3077809e37cd9e55dc57bbb2e30de94c0b7d1550/metadata-auth/auth-api/src/main/java/com/datahub/authorization/ResolvedResourceSpec.java

by adding a sort of method Set<Pair<String,String>> get OwnerTypes

That makes sense and actually this is something we already considered when starting the implementation, however we found a limitation in the current implementation of the ResolverSpecs abstraction: the return type is limited to the FieldResolver.FieldValue, which is Set<String> as you can see here

datahub/metadata-auth/auth-api/src/main/java/com/datahub/authorization/FieldResolver.java

Lines 45 to 53 in 3077809

    
             /** 
        
              * Container for storing the field value, in case we need to extend this to have more types of field values 
        
              */ 
        
             @Value 
        
             @Builder 
        
             public static class FieldValue { 
        
               Set<String> values; 
        
             } 
        
           }

So, we are kindly requesting you to release the feature as it is in this PR and consider the refactor as pending technical debt that goes beyond the scope of this PR. There are several reasons why we think such a refactor should be managed in the future as part of the generalization of the FieldValue:

First of all and more importantly, this is not new tech debt but existing one

There are other cases in policy engine not using the ResolverSpec abstraction such as roles

datahub/metadata-service/auth-impl/src/main/java/com/datahub/authorization/PolicyEngine.java

Lines 382 to 385 in 3077809

    
           try { 
        
             final EntityResponse corpUser = _entityClient.batchGetV2(CORP_USER_ENTITY_NAME, Collections.singleton(actor), 
        
                 Collections.singleton(ROLE_MEMBERSHIP_ASPECT_NAME), _systemAuthentication).get(actor); 
        
             if (corpUser == null || !corpUser.hasAspects()) {

or groups

datahub/metadata-service/auth-impl/src/main/java/com/datahub/authorization/PolicyEngine.java

Lines 415 to 418 in 3077809

    
           final EntityResponse corpUser = _entityClient.batchGetV2(CORP_USER_ENTITY_NAME, Collections.singleton(actor), 
        
                   ImmutableSet.of(GROUP_MEMBERSHIP_ASPECT_NAME, NATIVE_GROUP_MEMBERSHIP_ASPECT_NAME), _systemAuthentication) 
        
               .get(actor); 
        
           if (corpUser == null || !corpUser.hasAspects()) {

FieldValue has been already abused in ResolverSpec

datahub/metadata-auth/auth-api/src/main/java/com/datahub/authorization/ResolvedResourceSpec.java

Lines 38 to 41 in 3077809

    
           Set<String> resourceTypes = 
        
               fieldResolvers.get(ResourceFieldType.RESOURCE_TYPE).getFieldValuesFuture().join().getValues(); 
        
           assert resourceTypes.size() == 1; // There should always be a single resource type. 
        
           return resourceTypes.stream().findFirst().get();

, this clearly shows the missing and pending generalization for the FieldValue

Mixing a heavy refactor and feature development in this PR would be very noisy, hard to review and risky as it would affect many other components.
Overall, our contributions on Authorization resulted on being delayed for many months (custom ownership dependency) and this new refactor is just adding more delay. Also, the review of this PR has been split on different and scattered feedback loops that make the contribution experience quite frustrating. Also and unfortunately, the last review was got yesterday and the author of the PR goes on vacation next week. Which means that either we hand over this to someone else in my team or we postpone again for another week, at least. So more delay again.

Thanks

sgomezvillamor · 2023-07-01T07:28:50Z

Hi @jjoyce0510 , about the failing tests, there are two checks failing:

check datahub client jar fails with:

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.9.2/userguide/command_line_interface.html#sec:command_line_warnings
A problem occurred configuring root project 'datahub'.
> Could not resolve all dependencies for configuration ':classpath'.
   > Could not determine artifacts for org.tomlj:tomlj:1.0.0
      > Could not get resource 'https://plugins.gradle.org/m2/org/tomlj/tomlj/1.0.0/tomlj-1.0.0.jar'.
         > Could not HEAD 'https://jcenter.bintray.com/org/tomlj/tomlj/1.0.0/tomlj-1.0.0.jar'.
            > Connect to jcenter.bintray.com:443 [jcenter.bintray.com/34.95.74.180] failed: connect timed out

The error is not related to our updates. I don't have permissions to re-run this test, but I guess it will eventually succeed after some retries.

Run smoke tests (cypress1). The test failing here is the mutations/managed_ingestion.js... very unlikely to be related to our updates. 🤔 Actually, I just checked master and the same error happens there.

jjoyce0510 · 2023-07-03T16:44:03Z

CI failures are unrelated. Will be ignoring those!

Thanks for the hard work on this PR, and apologies for the delay!

Cheers
John

skrydal added 5 commits March 1, 2023 13:11

Initial approach to fine-grained resource owners handling. Types are

e0ad49f

visible in UI and enforced for users (no groups)

Adjusting logging message

ed670a0

Adjusting existing tests

62dc5c3

Merge branch 'datahub-project:master' into fine_grained_ownership_pol…

65b3a87

…icies

Created tests for new use-cases

9c2657e

github-actions bot added devops PR or Issue related to DataHub backend & deployment product PR or Issue related to the DataHub UI/UX labels Mar 4, 2023

skrydal added 2 commits March 6, 2023 14:05

Merge branch 'master' into fine_grained_ownership_policies

ebce9b6

Merge branch 'master' into fine_grained_ownership_policies

7d02aae

Merge branch 'master' into fine_grained_ownership_policies

0940909

skrydal added 9 commits March 8, 2023 14:07

Merge branch 'master' into fine_grained_ownership_policies

ea10c6e

Merge branch 'master' into fine_grained_ownership_policies

755ac7f

Merge branch 'master' into fine_grained_ownership_policies

feaba4f

Merge branch 'master' into fine_grained_ownership_policies

3918a2a

Merge branch 'master' into fine_grained_ownership_policies

8bcd8e0

Merge branch 'master' into fine_grained_ownership_policies

cc2be41

Merge branch 'master' into fine_grained_ownership_policies

362578b

Merge branch 'master' into fine_grained_ownership_policies

dc5d719

Merge branch 'master' into fine_grained_ownership_policies

f9a4e9c

Merge branch 'master' into fine_grained_ownership_policies

35f7c62

vercel bot deployed to Preview March 23, 2023 14:09 View deployment

Merge branch 'master' into fine_grained_ownership_policies

330d919

vercel bot deployed to Preview March 31, 2023 08:24 View deployment

Merge branch 'master' into fine_grained_ownership_policies

4cb5d84

jjoyce0510 reviewed Jun 20, 2023

View reviewed changes

Merge branch 'master' into fine_grained_ownership_policies

85a97c1

vercel bot deployed to Preview June 21, 2023 08:05 View deployment

skrydal added 2 commits June 21, 2023 13:33

Applied review comments' suggestions

eed5d05

Fixing tests

e221525

skrydal requested a review from jjoyce0510 June 21, 2023 11:40

vercel bot deployed to Preview June 21, 2023 12:05 View deployment

anshbansal added the community-contribution PR or Issue raised by member(s) of DataHub Community label Jun 23, 2023

Merge branch 'master' into fine_grained_ownership_policies

3077809

vercel bot deployed to Preview June 25, 2023 20:30 View deployment

Update PolicyDetailsModal.tsx

d77d110

Change copy in the details modal

jjoyce0510 reviewed Jun 28, 2023

View reviewed changes

vercel bot deployed to Preview June 28, 2023 16:21 View deployment

Merge branch 'master' into fine_grained_ownership_policies

3d8cf6a

vercel bot deployed to Preview June 28, 2023 16:59 View deployment

Merge branch 'master' into fine_grained_ownership_policies

bbe4457

vercel bot deployed to Preview June 29, 2023 08:30 View deployment

Merge branch 'master' into fine_grained_ownership_policies

9b780fa

vercel bot deployed to Preview June 30, 2023 18:18 View deployment

jjoyce0510 approved these changes Jul 3, 2023

View reviewed changes

jjoyce0510 merged commit cfa864e into datahub-project:master Jul 3, 2023
46 of 48 checks passed

sgomezvillamor mentioned this pull request Jul 18, 2023

feat(auth): platform instance ownership in policies #8396

Open

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(auth): Fine grained ownership policies #7499

feat(auth): Fine grained ownership policies #7499

skrydal commented Mar 4, 2023 •

edited

Loading

aditya-radhakrishnan commented Mar 8, 2023

skrydal commented Mar 8, 2023

aditya-radhakrishnan commented Mar 16, 2023

skrydal commented Mar 16, 2023

aditya-radhakrishnan commented Mar 16, 2023

skrydal commented Mar 31, 2023

jjoyce0510 Jun 20, 2023

skrydal Jun 21, 2023

jjoyce0510 Jun 20, 2023

skrydal Jun 21, 2023

jjoyce0510 Jun 20, 2023

skrydal Jun 21, 2023

skrydal commented Jun 21, 2023

skrydal commented Jun 26, 2023

jjoyce0510 left a comment

jjoyce0510 Jun 28, 2023

jjoyce0510 Jun 28, 2023

jjoyce0510 commented Jun 28, 2023

sgomezvillamor commented Jun 30, 2023

sgomezvillamor commented Jul 1, 2023 •

edited

Loading

jjoyce0510 commented Jul 3, 2023

feat(auth): Fine grained ownership policies #7499

feat(auth): Fine grained ownership policies #7499

Conversation

skrydal commented Mar 4, 2023 • edited Loading

aditya-radhakrishnan commented Mar 8, 2023

skrydal commented Mar 8, 2023

aditya-radhakrishnan commented Mar 16, 2023

skrydal commented Mar 16, 2023

aditya-radhakrishnan commented Mar 16, 2023

skrydal commented Mar 31, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

skrydal commented Jun 21, 2023

skrydal commented Jun 26, 2023

jjoyce0510 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jjoyce0510 commented Jun 28, 2023

sgomezvillamor commented Jun 30, 2023

sgomezvillamor commented Jul 1, 2023 • edited Loading

jjoyce0510 commented Jul 3, 2023

skrydal commented Mar 4, 2023 •

edited

Loading

sgomezvillamor commented Jul 1, 2023 •

edited

Loading