S3 Bucket data sharing

[dlpzx]

Is your feature request related to a problem? Please describe.
When I use tools in my ETL or data consumption applications, I often need direct access to the source data S3 Bucket. Lake Formation in some cases does not perform sufficiently fast and S3 access points do not integrate with some of the used tools. At the moment we solve this challenge by manually providing permission in the Bucket policy of the source Bucket.

Describe the solution you'd like
For this particular cases we would like an automated way to manage access to S3 Buckets.

• We want a self-service workflow with sharing request and approval and at the same time
• we want to use S3 Bucket policies under-the-hood.

Describe alternatives you've considered
We can re-use data.all sharing workflow and include a new type of sharing, S3 Bucket. To not over-complicate the resulting bucket policy we would not require a prefix-level grained access, granting access grants access to the whole S3 Bucket.

Changes needed:
(see comments below)

Question

• How should we manage imported Datasets? When we import a Dataset, we are importing an existing S3 Bucket with its corresponding bucket policy which might be managed by other processes outside data.all. We need to define if data.all a) will ignore these buckets, b) can overwrite the bucket policy (I am not sure how) or c) if it will re-create it and manage it.
• How do we define that a dataset is s3-bucket-shareable? We need to clarify what is the user process to categorize a dataset as S3-bucket-shareable.
• Can a dataset be shared with LF-tables, Access points-folders and S3-Bucket mechanisms simultaneously? What are the limitations?
• How do we handle KMS keys for imported buckets? and in sharing how do we specify CMK keys?

Design decisions and assumptions – We should write ADRs in the development fork

• We will first implement the feature on data.all created datasets and then we will deal with imported datasets.
• Users can choose between 2 sharing models, granular-share and full-share. Dataset owners decide which model applies to their Dataset and shares will follow the chosen model. In the granular-share model, we share Tables (Glue tables) using Lake Formation and Folders (S3 prefixes) using S3 Access points, this is what is currently implemented. In the full-share model, the whole S3 Bucket is shared using Bucket policies and the whole Glue database is shared using Lake Formation. We will first implement the full S3 Bucket share and as optional the full database share.

[dlpzx]

Here is a design proposal with the list of tasks that would be needed.

A. Add “dataSharingModel” field to Dataset Creation

As a user I can create a Dataset and assign a different data sharing model depending on the use-case requirements.

Steps and development hints

1. DONE - Frontend - Modify frontend view and add new field called “DataSharingModel”
   frontend/src/views/Datasets/DatasetCreateForm.js

2. DONE - Backend - Modify createDataset api call to include this new field. First go to resolvers and find the db/api operation
   backend/dataall/api/Objects/Dataset/input_types.py
   backend/dataall/db/api/dataset.py

3. DONE - Backend - Modify RDS table “dataset” to include this new field in a new column
   backend/dataall/db/models/Dataset.py

4. Backend [in the future] - Create migration script to modify RDS db and add new column “dataSharingModel” in dataset table AND fill existing ones with default “Granular”
   backend/migrations

B. Add “dataSharingModel” field to Dataset views

Dataset owners should be able to access the information about a Dataset’s data sharing model from the UI. This information should be added to the Overview tab.

Steps and development hints

1. [Added #TODO] Frontend – Modify Dataset overview and add new field called “DataSharingModel”
   frontend/src/views/Datasets/DatasetOverview.js
   frontend/src/views/Datasets/DatasetGovernance.js

2. [Added #TODO] Frontend/Backend - Modify getDataset api call to include the DataSharingModel field. backend/dataall/api/Objects/Dataset/schema.py
   frontend/src/api/Dataset/getDataset.js

C. Show info in Share request creation

We keep the catalog with datasets, tables and folders as is. Whenever there is a new share request, in the share request form we need to add the information about the sharing model used. There are no changes in the createShareRequest API call (maybe we need to include additional metadata later but for the moment let’s keep the share object independent from the dataset info)

Steps and development hints

1. [Added #TODO] Frontend - Show dataset data sharing model in Share request creation. You need to call the “getDataset” API and plot the results in:
   frontend/src/views/Catalog/RequestAccessModal.js

D. Adapt Share request view

When dataset requesters open the share request the view should reflect the data sharing mechanism used. For Granular-shared datasets Tables and Folders can be added as share items to the request. For a Full-share dataset there is only a single share item of the type S3 Bucket which is added automatically when we created the request.
This is how the view for a granular-shared dataset should look like (same as now):

For the full-share Datasets, we should have a single share item, the S3 Bucket and the consumption details should indicate information about the Bucket

Steps and development hints

1. [Added #TODO] Backend – Add a new share item type “S3Bucket” and modify the createShare method to add the S3Bucket share item if the share request is opened on a “Full” Dataset.

2. [Added #TODO] Backend - Modify the list of shareable items to resolve to S3 Bucket. This is to list only the S3 Bucket when we click on “Add item” in the UI. This comes from the getShareObject API output (here: backend/dataall/api/Objects/ShareObject/schema.py:143).
   Modify:
   backend/dataall/api/Objects/ShareObject/resolvers.py
   backend/dataall/db/api/share_object.py

3. [Added #TODO] Backend/Frontend - Conditional rendering based on Dataset sharing Model. We need to have the info on the data sharing model available on the UI. We already get the dataset as part of the ShareObject graphql model ( See backend/dataall/api/Objects/ShareObject/schema.py:118) Add the info in the corresponding resolver:backend.dataall.api.Objects.ShareObject.resolvers.resolve_dataset and modify the view, (ShareView)

E. Modify logic to trigger a different share task for S3 Bucket sharing

When a share request has been approved and the Dataset has “full” as its data sharing model. We need to modify the S3 bucket policy instead of creating access points and LF permissions.

Steps and development hints

1. [Added #TODO] Backend – modify sharing task routing to trigger an S3 bucket sharing task. We do not need to modify any of the following, but just for clarification, in backend.dataall.api.Objects.ShareObject.resolvers.approve_share_object
   we are approving the share, which creates the ‘ecs.share.approve’ task, picked by a decorated handler in backend.dataall.aws.handlers.ecs.Ecs.approve_share which triggers the ECS task defined in backend/dataall/tasks/share_manager.py
2. [Added #TODO] Backend – create an s3_bucket_process_share (processor) with 3 methods: approve, revoke and cleanup (this last one might not be needed)

F. Implement S3 Bucket sharing processor

Now that the sharing task runs a different sharing processor, we need to implement the logic of this processor.

Design decision log (@chamcca to review)

Update July 18th --> to implement data sharing we need to:

• update the requester IAM role policies - grant permission to the S3 bucket and KMS key
• update the S3 Bucket policy and add the requester
• update the KMS key policy and add the requester

For the above we can either:
A) Update the CloudFormation stacks containing the above resources
PROS:

• single view of what is being shared
  CONS:
• conflicts for imported datasets that already have a bucket policy - not possible to attach a policy when a bucket already has one
• data sharing might break dataset infrastructure
• complicated error handling of share failures

B) Perform boto3 calls to grant permissions in IAM, S3 and KMS
PROS:

• allows to work with imported buckets and consumption roles the same way that we work with data.all managed resources
• easy to debug and error handling
• possibility to implement retry strategies
• separated data sharing and dataset infrastructure. Data sharing failures won't corrupt the dataset stack
  CONS:
• external processes that handle bucket policy/kms policy and iam roles can override the granted permissions
• difficult to track the current access to a dataset

Conclusion: after this analysis we still consider that the option B, (B of boto3 calls), advantages overcome the disadvantages of managing CFN stacks. We will go forward with option B, but carefully remediating the CONS that is entails.

Steps and development hints - Update July 18th

1. [Added #TODO] Backend – Implement boto3 calls to give access to requester IAM role (to the S3 Bucket and KMS key) and trigger update of dataset stack in processor and calls to add grants to the dataset bucket policy and KMS key ---> since v1.6 KMS keys can also be imported to data.all

2. [Added #TODO] Backend – list all S3 shares of a dataset in the dataset stack share manager

3. [Added #TODO] Backend – Implement changes in dataset stack share manager to update Bucket policy + KMS key policy depending on the S3_bucket_shares approved or successful for that dataset
   backend/dataall/cdkproxy/stacks/dataset.py

4. (optional in first version) - Implement logging of all the updates on the S3 Bucket policy. We should keep a transactional log of the actions taken on externally managed resources and show an status = "overwritten" when updates outside of data.all happen.

5. (optional in first version) - add visibility on the status of the share for share requesters and dataset owners. If external changes happen, update the share status to failed

6. and allow "Sync-share" functionality for share requesters and/or dataset owners and/or environment owners (we need to agree on this one)

G. Other tasks

1. Create tests for the api calls and for the sharing processor
2. Allow inter-region sharing if we are sharing through bucket sharing

[dlpzx]

I created a new branch from V1.5 so that we start from the latest version and we do not need to rebase. The instructions listed above match the different commits in the branch. In most cases I just added TODOs wherever changes are needed.

[zsaltys]

@dlpzx I think there's an issue with:

C. Show info in Share request creation

The catalog will by default list all S3 folders and tables for any dataset. When using full sharing model we cannot let the catalog show individual tables and folders. Imagine what would happen if a user asks to share 1 table on full share model. He then might think he also needs to request a share on all the other tables so he will make requests for them too. I think we need to make a change that for datasets which are shared with a full model we only index the dataset itself and not individual tables and folders. LMK what you think about this and if you agree please update the issue.

[dlpzx]

Hi @zsaltys I had the same dilemma when designing the feature. We have 2 options, for full-share datasets we either a) index the S3 bucket instead of the tables and folders b) we index tables and folders as we do now and manage the pre-existing shares with messages and UI changes.

I went on with option b) because it adds more visibility on the content of the dataset. Users often search for a table that they know in the catalog in other words I did not want to decrease the discoverability of tables in the Catalog. But this is not set on stone and we are open to changes or new solutions

[zsaltys]

@dlpzx Thanks for your reply. I reviewed again the task but I don't think we currently explain what will happen in this scenario:

1. we create a bucket with s3 bucket policy sharing
2. we create 2 tables X and Y
3. a user from team A will request to share table X and get it approved
4. a different user from team A will see table Y in the catalog. They can request to share this table. However this is meaningless. The table would be already shared with them from step 3. This is confusing to the user.

My proposal would be that if we want to keep showing individual tables then we would need to show an indicator on the UI that the table is already shared with the environment and role that is requesting access.

[dlpzx]

Update on Task G.3 - KMS keys handling and CMK keys - > when implementing imported Dataset

Problem statement

At the moment KMS keys are not imported alongside the S3 Bucket when we import a Dataset. But when we share a dataset through S3 Bucket sharing, we will also need to grant permissions to the requester to the target dataset KMS key.

1. Give permissions (encrypt/decrypt) in the IAM policy of the requester to the KMS key id
2. Give permissions in the KMS policy to the requester IAM role

Solution

Data.all release v1.6.0 is focused on security hardening and one of the task is hardening the IAM policies of all roles in data.all.

• As part of this task it has been necessary to add the KMS key as input when importing a dataset. See feat: limit environment roles permissions + import KMS key in imported datasets #515
• Another task is hardening the KMS key policies of dataset KMS keys, restricting the access to dataset owners only

How does that affect us?

1. For giving permissions (encrypt/decrypt) in the IAM policy of the requester to the KMS key id we can use the same methods that we would use with a created dataset
   2.a) KMS key policies were very open for created datasets, in v1.6 we are restricting the KMS policy which means that it needs to be updated as a Bucket policy
   2.b) Give permissions in the KMS policy of an imported dataset- very similar to giving permissions in the S3 Bucket policy - we need to decide on how to manage pre-existing policies that are created outside of data.all. Should we append new policies? What are the permissions that the pivotRole needs?

[zsaltys]

@dlpzx We've made good progress on bucket sharing support. We have deviated from the initial design and the "Full Share" and "Granular Share" modes will not exist. It will simply be possible to select bucket sharing as part of the share request UI. We also added table prefiltering so that datasets only show tables that belong to the same bucket we think this makes sense to contribute back together.