Enabling S3 bucket share #823

anushka-singh · 2023-10-20T21:39:12Z

Feature or Bugfix

Feature

Detail

Enabling S3 bucket share
Added support for enabling both access point and s3 bucket share together

Relates

S3 Bucket data sharing #284

Security

Please answer the questions below briefly where applicable, or write N/A. Based on
OWASP 10.

Does this PR introduce or modify any input fields or queries - this includes
fetching data from storage outside the application (e.g. a database, an S3 bucket)?
- Is the input sanitized?
- What precautions are you taking before deserializing the data you consume?
- Is injection prevented by parametrizing queries?
- Have you ensured no eval or similar functions are used?
Does this PR introduce any functionality or component that requires authorization?
- How have you ensured it respects the existing AuthN/AuthZ mechanisms?
- Are you logging failed auth attempts?
Are you using or adding any cryptographic features?
- Do you use a standard proven implementations?
- Are the used keys controlled by the customer? Where are they stored?
Are you introducing any new policies/roles/users?
- Have you used the least-privilege principle? How?

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

dbalintx · 2023-10-25T13:19:27Z

backend/dataall/modules/dataset_sharing/api/resolvers.py

@@ -191,6 +191,7 @@ def resolve_consumption_data(context: Context, source: ShareObject, **kwargs):
            return {
                's3AccessPointName': S3AccessPointName,
                'sharedGlueDatabase': (ds.GlueDatabaseName + '_shared_' + source.shareUri)[:254] if ds else 'Not created',
+                's3bucketName': ds.S3BucketName if ds else 'NotFound',


if ds is already checked at line 186, if we want to return some default values it should be in an else branch

I agree. Removed the else portion.

dbalintx · 2023-10-25T14:45:59Z

backend/dataall/modules/dataset_sharing/api/enums.py

@@ -5,6 +5,7 @@ class ShareableType(GraphQLEnumMapper):
    Table = 'DatasetTable'
    StorageLocation = 'DatasetStorageLocation'
    View = 'View'
+    S3Bucket = 'S3Bucket'


minor: add "Dataset" to the name to emphasize the relation?

dbalintx · 2023-10-25T14:46:05Z

backend/dataall/modules/dataset_sharing/db/enums.py

@@ -57,6 +57,7 @@ class ShareableType(Enum):
    Table = 'DatasetTable'
    StorageLocation = 'DatasetStorageLocation'
    View = 'View'
+    S3Bucket = 'S3Bucket'


minor: add "Dataset" to the name to emphasize the relation?

dbalintx · 2023-10-26T07:43:16Z

backend/dataall/modules/dataset_sharing/services/share_managers/lf_share_manager.py

-            role_name=self.share.principalIAMRoleName
-        )
-        principals = [principal_iam_role_arn]
+        principals = [f"arn:aws:iam::{self.target_environment.AwsAccountId}:role/{self.share.principalIAMRoleName}"]


the former function was implicitly checking if the role exists - it would be good to keep this check

This was edited due to merge conflicts. Added it back.

dbalintx · 2023-10-26T08:22:37Z

...d/dataall/modules/dataset_sharing/services/share_processors/s3_access_point_process_share.py

-        target_environment: Environment,
-        source_env_group: EnvironmentGroup,
-        env_group: EnvironmentGroup,
+            self,


identation seems to be wrong, also below

Fixed this in next commit.

dbalintx · 2023-10-26T08:40:39Z

...end/dataall/modules/dataset_sharing/services/share_managers/s3_access_point_share_manager.py

@@ -91,7 +93,7 @@ def manage_bucket_policy(self):
        s3_client = S3Client(self.source_account_id, self.source_environment.region)
        bucket_policy = json.loads(s3_client.get_bucket_policy(self.bucket_name))
        for statement in bucket_policy["Statement"]:
-            if statement.get("Sid") in ["AllowAllToAdmin", "DelegateAccessToAccessPoint"]:


why was this check removed?

We are now adding AllowAllToAdmin from both S3 bucket share and access point share. If bucket share already added AllowAllToAdmin and then access point share is requested, access point share will not happen. Hence, I have removed this check.

dbalintx · 2023-10-26T08:49:07Z

...end/dataall/modules/dataset_sharing/services/share_managers/s3_access_point_share_manager.py

-                    f'but S3 Access point {self.access_point_name} is not included, updating...'
-                )
-                target_resources = [
+            s3_target_resources = [
                    f"arn:aws:s3:::{self.bucket_name}",


identation formatting seems to be wrong

It looks right to me in my editor. Could you please describe exactly what looks wrong?

dbalintx · 2023-10-26T09:11:57Z

backend/dataall/modules/dataset_sharing/services/share_managers/share_manager_utils.py

+            iam_role_policy_name
+    ):
+        """
+        Checks if the bucket and kms resources is in the existing policy. Otherwise, it will add it.


as the function is written in a general way, I'd remove "bucket" and "kms" from the description

Makes sense! Removed it in the next commit.

dbalintx · 2023-10-26T09:17:26Z

backend/dataall/modules/dataset_sharing/services/share_managers/s3_bucket_share_manager.py

+            bucket_policy = S3ControlClient.generate_default_bucket_policy(self.bucket_name, exceptions_roleId)
+        return bucket_policy
+
+    def get_bucket_owner_roleid(self):


maybe move this function to share_manager_utils? could also be re-used in s3_access_point_share_manager in multiple places

dbalintx · 2023-10-26T09:34:42Z

backend/dataall/modules/datasets/cdk/dataset_stack.py

@@ -242,8 +259,7 @@ def __init__(self, scope, id, target_uri: str = None, **kwargs):
                    sid="ListDatasetBucket",
                    actions=[
                        "s3:ListBucket",
-                        "s3:GetBucketLocation",
-                        "s3:GetBucketAcl"


why was this action removed?

Coming from a merge conflict. Added it back.

dbalintx · 2023-10-26T09:36:59Z

backend/dataall/modules/datasets/cdk/dataset_stack.py

@@ -28,6 +29,8 @@

 logger = logging.getLogger(__name__)

+GACCO_ALLOW_CROSS_ACCOUNT_TAG = ("gacco:AllowPublicAccess", "True")


what exactly is this tag used for (I see it is attached to the dataset KMS key but what is the reason) ?

also coming from merge conflict. Removed this as its not needed.

dbalintx · 2023-10-26T09:39:54Z

backend/dataall/modules/datasets/db/dataset_bucket_repositories.py

+        )
+        for bucket in buckets:
+            session.delete(bucket)
+        return True


I believe the return True isn't necessary

Sure! Removed it.

dbalintx · 2023-10-26T09:50:38Z

tests/modules/datasets/tasks/test_s3_access_point_share_manager.py

-    source_environment: Environment,
-    target_environment: Environment,
-    base_bucket_policy,
+        mocker,


identation format seems to be wrong, also in some places below

Fixed this in next commit.

dlpzx · 2023-10-27T10:00:54Z

backend/dataall/core/environment/services/environment_service.py

@@ -81,7 +81,7 @@ def create_environment(session, uri, data=None):
            resource_prefix=env.resourcePrefix,
        ).build_compliant_name()

-        if not data.get('EnvironmentDefaultIAMRoleArn'):
+        if not data.get('EnvironmentDefaultIAMRoleName'):


@anushka-singh I think there is some commits-conflict issues in this PR. From what I am seeing this PR overrides the PR that handles roles by arn instead of by name

I just checked the logs of the PR branch and the last time it got commits from main or v2m1m0 was on Sept 16th. Changes done by @noah-paige in the env-role-arn PR were merged on Sept 26th. That's why they are missing. We might face other conflict issues/overrides besides this one, so the best will be to do a merge of origin/v2m1m0 into your working branch anushka/v2m1m0. You will probably face some conflict issues, if you want @noah-paige or I can do the merge with you. And of course if you have a better way to do it, we can do it in a different way. Git-ing is full possibilities

# Conflicts: # deploy/stacks/backend_stack.py # deploy/stacks/backend_stage.py # deploy/stacks/lambda_api.py # deploy/stacks/pipeline.py # template_cdk.json

### Feature or Bugfix - Feature ### Detail - read KMS keys with an alias prefixed by the environment resource prefix - read KMS keys imported in imported datasets - restrict pivot role policies to the KMS keys created by data.all and those imported in the imported datasets - move kms client from data_sharing to base as it is used in environments and datasets ### Relates - data-dot-all#580 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). This PR restricts the IAM policies of the pivot role, following the least privilege permissions principle - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

### Feature or Bugfix - Bugfix ### Detail - Make `hosted_zone_id` optional, code update ### Relates - data-dot-all#797 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? N/A - Is the input sanitized? N/A - What precautions are you taking before deserializing the data you consume? N/A - Is injection prevented by parametrizing queries? N/A - Have you ensured no `eval` or similar functions are used? N/A - Does this PR introduce any functionality or component that requires authorization? N/A - How have you ensured it respects the existing AuthN/AuthZ mechanisms? N/A - Are you logging failed auth attempts? N/A - Are you using or adding any cryptographic features? N/A - Do you use a standard proven implementations? N/A - Are the used keys controlled by the customer? Where are they stored? N/A - Are you introducing any new policies/roles/users? N/A - Have you used the least-privilege principle? How? N/A By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. YES ### Description Make `hosted_zone_id` optional and provide `HostedZoneId` and `DNSName` in CloudFormation Stack Output, so users can create their own [Route53 AliasTarget](https://docs.aws.amazon.com/Route53/latest/APIReference/API_AliasTarget.html). Following validation checks in `ecs_patterns.ApplicationLoadBalancedFargateService` were considered: * `frontend_alternate_domain` and `userguide_alternate_domain` have to be `None` when the `hosted_zone` is `None`, see checks in [multiple-target-groups-service-base.ts#L463](https://github.com/aws/aws-cdk/blob/c445b8cc6e20d17e4a536f17262646b291a0fe36/packages/aws-cdk-lib/aws-ecs-patterns/lib/base/network-multiple-target-groups-service-base.ts#L463), or else a `A Route53 hosted domain zone name is required to configure the specified domain name` error is raised * for a HTTPS ALB listener, only the `certificate` is ultimately required, and not the `domainName` or `domainZone`, as per evaluation logic in [application-load-balanced-service-base.ts#L509](https://github.com/aws/aws-cdk/blob/c445b8cc6e20d17e4a536f17262646b291a0fe36/packages/aws-cdk-lib/aws-ecs-patterns/lib/base/application-load-balanced-service-base.ts#L509)

### Feature or Bugfix - Bugfix ### Detail - Clean up prints and show better exception message when custom_domain is not provided for SES ### Relates - v2.1.0 ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

…into bucket_share

# Conflicts: # backend/dataall/modules/datasets/services/dataset_service.py

anushka-singh · 2023-11-02T17:43:36Z

Duplicated in #848

@anushka-singh

### Feature or Bugfix - Feature ### Detail - We want to enable bucket sharing along with access point share which already exists in data all right now. - A user will be able to request shares at bucket level and at the folder level with access points. - Please NOTE: There is some common code between Access point share managers and processors and S3 Bucket managers and processors. We will send out a separate PR for that refactoring work at a later time. ### Relates - #284 - #823 - https://github.com/awslabs/aws-dataall/pull/846/files#diff-c1f522a1f50d8bcf7b6e5b2e586e40a8de784caa80345f4e05a6329ae2a372d0 ### Contributors: - Contents of this PR have been contributed by @anushka-singh, @blitzmohit, @rbernotas, @TejasRGitHub ### Security Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Anushka Singh <anushka.singh@yahooinc.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com>

@anushka-singh

* Bigdata867 3 (data-dot-all#24) * Bucket Policy E.1: Modify sharing task routing to trigger a s3 bucket sharing * Bucket Policy E.1: Modify sharing task routing to trigger a s3 bucket sharing * Bucket Policy E.1: Modify sharing task routing to trigger a s3 bucket sharing * Bucket Policy BIGDATA 867: Implement revoke share in data_sharing_service * Bucket Policy BIGDATA 867: Implement revoke share in data_sharing_service * trajopadhye- BIGDATA-756 -> Added Tests for Task D and E * trajopadhye - BIGDATA-756 Corrected file data_sharing_service.py to address revokedStateSM for revoked items * trajopadhye- BIGDATA-756 - Slight correction in comments * trajopadhye- BIGDATA-756 Correction on Share Status for revoke share tests * Addresed changes from the review of PR * [BIGDATA-625] Implement bucket share processor (data-dot-all#21) * Implement bucket share processor * Fix Revoke UI sharetype * BIGDATA-612 - push source from SD container to CodeCommit. Initial Makefile and SD yaml configuration. * Remove synth * Add force push * Add default cdk.context.json * Add param for branchname * Comments. * Fix email address * Add instance specific cdk.context.json * BIGDATA-612 - truncate the cfn encryption policy prefix so that together with branch name, it will fit within 32 char limit. * Update screwdriver.yaml * Change nodejs version in screwdriver Makefile to supported version 16 (data-dot-all#89) (data-dot-all#90) * Change screwdriver node version to 16 * Remove all non-environment setup steps for testing * Skip getting AWS credentials for testing * Fixing npm install version * Remove extra npm install * Restore all prior functions. * Remove AmplifyContext customizations, no longer needed. (data-dot-all#92) * Change nodejs version in screwdriver Makefile to supported version 16 (data-dot-all#89) * Change screwdriver node version to 16 * Remove all non-environment setup steps for testing * Skip getting AWS credentials for testing * Fixing npm install version * Remove extra npm install * Restore all prior functions. * Remove AmplifyContext customizations, no longer needed. (data-dot-all#91) * Fix screwdriver yaml for new EMR template step. (data-dot-all#116) * Bigdata 1397 mvp 3 stagingdeploy 20231129 (data-dot-all#178) * BIGDATA-1211 - Release notes initial commit * Mvp3 deploy 20231129 - S3 Bucket share + KMS explosion fix - MERGE FROM OPENSOURCE (data-dot-all#176) * Enabling S3 bucket share (data-dot-all#848) - Feature - We want to enable bucket sharing along with access point share which already exists in data all right now. - A user will be able to request shares at bucket level and at the folder level with access points. - Please NOTE: There is some common code between Access point share managers and processors and S3 Bucket managers and processors. We will send out a separate PR for that refactoring work at a later time. - data-dot-all#284 - data-dot-all#823 - https://github.com/awslabs/aws-dataall/pull/846/files#diff-c1f522a1f50d8bcf7b6e5b2e586e40a8de784caa80345f4e05a6329ae2a372d0 - Contents of this PR have been contributed by @anushka-singh, @blitzmohit, @rbernotas, @TejasRGitHub Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Anushka Singh <anushka.singh@yahooinc.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * Kms explosion fix (data-dot-all#882) - Bugfix - DataAll currently creates one SID per role in the KMS policy attached to a bucket with RoleID as the SID name. - We want to collapse these SIDs into one SID. - Access point and Bucket share will have different SIDs in KMS policy. - Use role ARN instead of role ID. - NOTE: if KMS policy was previously created, it will remain the same. SID will be the user ID and not the KMS decrypt SID created in this PR. It will not impact any future shares though. - NOTE: This is to be merged after bucket share PR is merged. - Tested this on local dev environment and KMS policy now has 1 statement with kms decrypt and using SID of KMS decrypt. Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Anushka Singh <anushka.singh@yahooinc.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * Updated Release Notes 20231201 * Format changes --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * [BIGDATA-1391] - Fix for cannot see all cognito groups when inviting teams (data-dot-all#177) * trajopadhye | BIGDATA-1391 - Fix for incomplete groups list fetched for invite org and env --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * Bigdata 1397 mvp 3 stagingdeploy 20231129 1 (data-dot-all#180) * BIGDATA-1211 - Release notes initial commit * Mvp3 deploy 20231129 - S3 Bucket share + KMS explosion fix - MERGE FROM OPENSOURCE (data-dot-all#176) * Enabling S3 bucket share (data-dot-all#848) - Feature - We want to enable bucket sharing along with access point share which already exists in data all right now. - A user will be able to request shares at bucket level and at the folder level with access points. - Please NOTE: There is some common code between Access point share managers and processors and S3 Bucket managers and processors. We will send out a separate PR for that refactoring work at a later time. - data-dot-all#284 - data-dot-all#823 - https://github.com/awslabs/aws-dataall/pull/846/files#diff-c1f522a1f50d8bcf7b6e5b2e586e40a8de784caa80345f4e05a6329ae2a372d0 - Contents of this PR have been contributed by @anushka-singh, @blitzmohit, @rbernotas, @TejasRGitHub Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Anushka Singh <anushka.singh@yahooinc.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * Kms explosion fix (data-dot-all#882) - Bugfix - DataAll currently creates one SID per role in the KMS policy attached to a bucket with RoleID as the SID name. - We want to collapse these SIDs into one SID. - Access point and Bucket share will have different SIDs in KMS policy. - Use role ARN instead of role ID. - NOTE: if KMS policy was previously created, it will remain the same. SID will be the user ID and not the KMS decrypt SID created in this PR. It will not impact any future shares though. - NOTE: This is to be merged after bucket share PR is merged. - Tested this on local dev environment and KMS policy now has 1 statement with kms decrypt and using SID of KMS decrypt. Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Anushka Singh <anushka.singh@yahooinc.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * Updated Release Notes 20231201 * Format changes --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * [BIGDATA-1391] - Fix for cannot see all cognito groups when inviting teams (data-dot-all#177) * trajopadhye | BIGDATA-1391 - Fix for incomplete groups list fetched for invite org and env * Bugfix --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * Bugfix (data-dot-all#181) * Bugfix * Bugfix * [Data 409] Athenz Certs Domain and User Pool Domain Changes (data-dot-all#221) (data-dot-all#222) * trajopadhye | DATA-409- Code changes for Athenz certs domain and user pool domain * [Data-413] GA stagingdeploy 20231228 - Fix for email notifications with Athenz. Auto-create Pivot Role (data-dot-all#224) * trajopadhye | DATA-412 - Added Athenz configs and Ports in AWS Worker lambda and enabling Auto Create Pivot Role * DATA-416 - Fix while migrating from manual pivot role to auto created (data-dot-all#230) (data-dot-all#233) * trajopadhye | DATA-416 - Fix for environment updates when using auto pivot role. Changing the way KMS keys are specified in env role * [Data 447] ga stagingdeploy 20240116 (data-dot-all#244) * [Data-446] Fix for consumption role not showing up * [Data 415] Dataset import fix for circular dependency error + local dev setup fixes (data-dot-all#243) * DATA-428 - Local env fixes * Data 448 ga stagingdeploy 20240117 (data-dot-all#246) * trajopadhye | DATA-440 - Adding else if to sync glue tabls in RDS * Data 461 ga deploy 20240125 (data-dot-all#258) * DATA-404 - Add git fetch --all to the CodeCommit repo sync * DATA-420 - Switch from Cognito to Okta on Prod (data-dot-all#254) DATA-420 - Switch from Cognito to Okta on Prod * DATA-455: Shares stuck in progress when AWS does not have root access on KMS key (data-dot-all#256) * Update release notes * Update release notes --------- Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> Co-authored-by: Tejas Rajopadhye <tejas.rajopadhye@yahooinc.com> * Data 466 ga stagingdeploy 20240126 (data-dot-all#263) * trajoadhye | DATA-456 - Removing Lake Formation SLR (data-dot-all#260) * Data-405-Adding max 30 sec delay * Synching Release notes from Staging to y-branch-2-0 (data-dot-all#262) * [Data 484] stagingdeploy 20240206 (data-dot-all#275) * fix: adding cdk synth for checkov scans (data-dot-all#264) * [DATA-452] - Adding Dataset description in shares view (data-dot-all#273) * Added Release note for DATA-481, DATA-452, DATA-480 * Syncing Release notes (data-dot-all#274) --------- Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> Co-authored-by: Anushka Singh <anushka.singh@verizonmedia.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@oath.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> * [Data 607] staging deploy email notification fix (data-dot-all#302) * Data:604: Add local level false positive management for PSECBUG - 73521 (data-dot-all#299) * DATA-600 - Fix for share link not present in email notifications * Merging changes needed for DATA-509 - Updating custom confidentiality values * DATA - 586 - Adding confidentiality values for custom confidentiality * Lower casing as suggested here- DATA-375 --------- Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> * Updating release notes for staging deploy (data-dot-all#301) --------- Co-authored-by: Anushka Singh <anushka.singh@verizonmedia.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> * [Data 611] Disable topics dropdown (data-dot-all#304) * Disabling topics dropdown (data-dot-all#303) * [Data 619] Stagingdeploy env permission fix (data-dot-all#307) * Data:604: Add local level false positive management for PSECBUG - 73521 (data-dot-all#299) * Data:604: Add local level false positive management for PSECBUG - 73521 (data-dot-all#300) * Email notification fix + confidentiality levels config (data-dot-all#298) * DATA-600 - Fix for share link not present in email notifications * Merging changes needed for DATA-509 - Updating custom confidentiality values * Adding confidentiality values for custom confidentiality * Adding confidentiality configs to config.json.PROD * Lower casing as suggested here- DATA-375 --------- Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> * Updating release notes for staging deploy (data-dot-all#301) * Disabling topics dropdown (data-dot-all#303) * DATA-619 - Fix permission for GET_ORGANIZATION when users are in _data teams (data-dot-all#306) * Cherry pick for issue with GET_ORG permission after 2.3 release --------- Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> --------- Co-authored-by: Anushka Singh <anushka.singh@verizonmedia.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> * [Data 631] Staging deploy (data-dot-all#310) * [Data 629] worksheet fix for GET_ENVIRONMENT permission (data-dot-all#309) * Data690 stagingdeploy 20240425 (data-dot-all#319) * DATA-680 - Update node repo to 18.x in Makefile.sd * Data674: Adding auto approval for confidentiality levels (data-dot-all#317) * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Ensuring Secret Confidentiality Type (Yahoo Confidential and Yahoo Highly Confidential) are never auto-approved * Use boolean true instead of string * Update config * Update release notes * Update release notes * Update release notes --------- Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> * Update makefile (data-dot-all#320) * Data690 stagingdeploy 20240425 2 (data-dot-all#321) * Update makefile * Reverting nodejs 16 upgrade * Reverting nodejs 16 upgrade * Data690 stagingdeploy 20240425 3 (data-dot-all#323) * DATA-680 - Update node repo to 18.x in Makefile.sd * Data674: Adding auto approval for confidentiality levels (data-dot-all#317) * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Ensuring Secret Confidentiality Type (Yahoo Confidential and Yahoo Highly Confidential) are never auto-approved * Use boolean true instead of string * Update config * Bugfix (data-dot-all#322) * Reverting nodejs 16 upgrade --------- Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> * Data690 stagingdeploy 20240425 4 (data-dot-all#325) * DATA-680 - Update node repo to 18.x in Makefile.sd * Data674: Adding auto approval for confidentiality levels (data-dot-all#317) * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Ensuring Secret Confidentiality Type (Yahoo Confidential and Yahoo Highly Confidential) are never auto-approved * Use boolean true instead of string * Update config * Bugfix (data-dot-all#322) * Blocking autoApproval edit on backend (data-dot-all#324) * Blocking autoApproval edit on backend * Lint fix * Reverting nodejs 18 upgrade --------- Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> * Data690 stagingdeploy 20240425 5 (data-dot-all#329) * DATA-680 - Update node repo to 18.x in Makefile.sd * Data674: Adding auto approval for confidentiality levels (data-dot-all#317) * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Ensuring Secret Confidentiality Type (Yahoo Confidential and Yahoo Highly Confidential) are never auto-approved * Use boolean true instead of string * Update config * Bugfix (data-dot-all#322) * Blocking autoApproval edit on backend (data-dot-all#324) * Blocking autoApproval edit on backend * Lint fix * DATA-680 - Switch node to version 17 in the Screwdriver makefile (data-dot-all#326) * bugfix (data-dot-all#328) * Remove nodejs upgrade --------- Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> * bugfix (data-dot-all#331) * Data743 stagingdeploy (data-dot-all#351) * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Data743: Update verifier task schedule to run nightly (data-dot-all#350) * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Data743 stagingdeploy (data-dot-all#353) * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * [Data 767] staging deploy (data-dot-all#358) * Bugfix: timeout error when listing Consumption Roles (data-dot-all#1303) - Bugfix - as GraphQL resolvers are 'lazy', for ShareRequest Modal window we simply don't fetch the managedPolicy property -- no timeout - managed policies are fetched, when consumption role is selected from dropdown - data-dot-all#1288 Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: Sofia Sazonova <sazonova@amazon.co.uk> * Updated Release notes --------- Co-authored-by: Sofia Sazonova <sofia-s@304.ru> Co-authored-by: Sofia Sazonova <sazonova@amazon.co.uk> --------- Co-authored-by: Anushka Singh <anushka.singh@yahooinc.com> Co-authored-by: Sofia Sazonova <sofia-s@304.ru> Co-authored-by: Sofia Sazonova <sazonova@amazon.co.uk> * data712 * Data712: Persistent emails * Data712: Persistent emails * Data712: Persistent emails * Data712: Persistent emails * Data712: Persistent emails * Data712: Persistent emails * Data712: Persistent emails * Restore yarn file * Restore yarn file * Update config --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> Co-authored-by: Mohit Arora <marora@yahooinc.com> Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> Co-authored-by: Raj Chopde <rchopde@yahooinc.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@oath.com> Co-authored-by: Sofia Sazonova <sofia-s@304.ru> Co-authored-by: Sofia Sazonova <sazonova@amazon.co.uk>

@anushka-singh

* Bigdata867 3 (data-dot-all#24) * Bucket Policy E.1: Modify sharing task routing to trigger a s3 bucket sharing * Bucket Policy E.1: Modify sharing task routing to trigger a s3 bucket sharing * Bucket Policy E.1: Modify sharing task routing to trigger a s3 bucket sharing * Bucket Policy BIGDATA 867: Implement revoke share in data_sharing_service * Bucket Policy BIGDATA 867: Implement revoke share in data_sharing_service * trajopadhye- BIGDATA-756 -> Added Tests for Task D and E * trajopadhye - BIGDATA-756 Corrected file data_sharing_service.py to address revokedStateSM for revoked items * trajopadhye- BIGDATA-756 - Slight correction in comments * trajopadhye- BIGDATA-756 Correction on Share Status for revoke share tests * Addresed changes from the review of PR * [BIGDATA-625] Implement bucket share processor (data-dot-all#21) * Implement bucket share processor * Fix Revoke UI sharetype * BIGDATA-612 - push source from SD container to CodeCommit. Initial Makefile and SD yaml configuration. * Remove synth * Add force push * Add default cdk.context.json * Add param for branchname * Comments. * Fix email address * Add instance specific cdk.context.json * BIGDATA-612 - truncate the cfn encryption policy prefix so that together with branch name, it will fit within 32 char limit. * Update screwdriver.yaml * Change nodejs version in screwdriver Makefile to supported version 16 (data-dot-all#89) (data-dot-all#90) * Change screwdriver node version to 16 * Remove all non-environment setup steps for testing * Skip getting AWS credentials for testing * Fixing npm install version * Remove extra npm install * Restore all prior functions. * Remove AmplifyContext customizations, no longer needed. (data-dot-all#92) * Change nodejs version in screwdriver Makefile to supported version 16 (data-dot-all#89) * Change screwdriver node version to 16 * Remove all non-environment setup steps for testing * Skip getting AWS credentials for testing * Fixing npm install version * Remove extra npm install * Restore all prior functions. * Remove AmplifyContext customizations, no longer needed. (data-dot-all#91) * Fix screwdriver yaml for new EMR template step. (data-dot-all#116) * Bigdata 1397 mvp 3 stagingdeploy 20231129 (data-dot-all#178) * BIGDATA-1211 - Release notes initial commit * Mvp3 deploy 20231129 - S3 Bucket share + KMS explosion fix - MERGE FROM OPENSOURCE (data-dot-all#176) * Enabling S3 bucket share (data-dot-all#848) - Feature - We want to enable bucket sharing along with access point share which already exists in data all right now. - A user will be able to request shares at bucket level and at the folder level with access points. - Please NOTE: There is some common code between Access point share managers and processors and S3 Bucket managers and processors. We will send out a separate PR for that refactoring work at a later time. - data-dot-all#284 - data-dot-all#823 - https://github.com/awslabs/aws-dataall/pull/846/files#diff-c1f522a1f50d8bcf7b6e5b2e586e40a8de784caa80345f4e05a6329ae2a372d0 - Contents of this PR have been contributed by @anushka-singh, @blitzmohit, @rbernotas, @TejasRGitHub Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Anushka Singh <anushka.singh@yahooinc.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * Kms explosion fix (data-dot-all#882) - Bugfix - DataAll currently creates one SID per role in the KMS policy attached to a bucket with RoleID as the SID name. - We want to collapse these SIDs into one SID. - Access point and Bucket share will have different SIDs in KMS policy. - Use role ARN instead of role ID. - NOTE: if KMS policy was previously created, it will remain the same. SID will be the user ID and not the KMS decrypt SID created in this PR. It will not impact any future shares though. - NOTE: This is to be merged after bucket share PR is merged. - Tested this on local dev environment and KMS policy now has 1 statement with kms decrypt and using SID of KMS decrypt. Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Anushka Singh <anushka.singh@yahooinc.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * Updated Release Notes 20231201 * Format changes --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * [BIGDATA-1391] - Fix for cannot see all cognito groups when inviting teams (data-dot-all#177) * trajopadhye | BIGDATA-1391 - Fix for incomplete groups list fetched for invite org and env --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * Bigdata 1397 mvp 3 stagingdeploy 20231129 1 (data-dot-all#180) * BIGDATA-1211 - Release notes initial commit * Mvp3 deploy 20231129 - S3 Bucket share + KMS explosion fix - MERGE FROM OPENSOURCE (data-dot-all#176) * Enabling S3 bucket share (data-dot-all#848) - Feature - We want to enable bucket sharing along with access point share which already exists in data all right now. - A user will be able to request shares at bucket level and at the folder level with access points. - Please NOTE: There is some common code between Access point share managers and processors and S3 Bucket managers and processors. We will send out a separate PR for that refactoring work at a later time. - data-dot-all#284 - data-dot-all#823 - https://github.com/awslabs/aws-dataall/pull/846/files#diff-c1f522a1f50d8bcf7b6e5b2e586e40a8de784caa80345f4e05a6329ae2a372d0 - Contents of this PR have been contributed by @anushka-singh, @blitzmohit, @rbernotas, @TejasRGitHub Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Anushka Singh <anushka.singh@yahooinc.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * Kms explosion fix (data-dot-all#882) - Bugfix - DataAll currently creates one SID per role in the KMS policy attached to a bucket with RoleID as the SID name. - We want to collapse these SIDs into one SID. - Access point and Bucket share will have different SIDs in KMS policy. - Use role ARN instead of role ID. - NOTE: if KMS policy was previously created, it will remain the same. SID will be the user ID and not the KMS decrypt SID created in this PR. It will not impact any future shares though. - NOTE: This is to be merged after bucket share PR is merged. - Tested this on local dev environment and KMS policy now has 1 statement with kms decrypt and using SID of KMS decrypt. Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Anushka Singh <anushka.singh@yahooinc.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * Updated Release Notes 20231201 * Format changes --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * [BIGDATA-1391] - Fix for cannot see all cognito groups when inviting teams (data-dot-all#177) * trajopadhye | BIGDATA-1391 - Fix for incomplete groups list fetched for invite org and env * Bugfix --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * Bugfix (data-dot-all#181) * Bugfix * Bugfix * [Data 409] Athenz Certs Domain and User Pool Domain Changes (data-dot-all#221) (data-dot-all#222) * trajopadhye | DATA-409- Code changes for Athenz certs domain and user pool domain * [Data-413] GA stagingdeploy 20231228 - Fix for email notifications with Athenz. Auto-create Pivot Role (data-dot-all#224) * trajopadhye | DATA-412 - Added Athenz configs and Ports in AWS Worker lambda and enabling Auto Create Pivot Role * DATA-416 - Fix while migrating from manual pivot role to auto created (data-dot-all#230) (data-dot-all#233) * trajopadhye | DATA-416 - Fix for environment updates when using auto pivot role. Changing the way KMS keys are specified in env role * [Data 447] ga stagingdeploy 20240116 (data-dot-all#244) * [Data-446] Fix for consumption role not showing up * [Data 415] Dataset import fix for circular dependency error + local dev setup fixes (data-dot-all#243) * DATA-428 - Local env fixes * Data 448 ga stagingdeploy 20240117 (data-dot-all#246) * trajopadhye | DATA-440 - Adding else if to sync glue tabls in RDS * Data 461 ga deploy 20240125 (data-dot-all#258) * DATA-404 - Add git fetch --all to the CodeCommit repo sync * DATA-420 - Switch from Cognito to Okta on Prod (data-dot-all#254) DATA-420 - Switch from Cognito to Okta on Prod * DATA-455: Shares stuck in progress when AWS does not have root access on KMS key (data-dot-all#256) * Update release notes * Update release notes --------- Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> Co-authored-by: Tejas Rajopadhye <tejas.rajopadhye@yahooinc.com> * Data 466 ga stagingdeploy 20240126 (data-dot-all#263) * trajoadhye | DATA-456 - Removing Lake Formation SLR (data-dot-all#260) * Data-405-Adding max 30 sec delay * Synching Release notes from Staging to y-branch-2-0 (data-dot-all#262) * [Data 484] stagingdeploy 20240206 (data-dot-all#275) * fix: adding cdk synth for checkov scans (data-dot-all#264) * [DATA-452] - Adding Dataset description in shares view (data-dot-all#273) * Added Release note for DATA-481, DATA-452, DATA-480 * Syncing Release notes (data-dot-all#274) --------- Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> Co-authored-by: Anushka Singh <anushka.singh@verizonmedia.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@oath.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> * [Data 607] staging deploy email notification fix (data-dot-all#302) * Data:604: Add local level false positive management for PSECBUG - 73521 (data-dot-all#299) * DATA-600 - Fix for share link not present in email notifications * Merging changes needed for DATA-509 - Updating custom confidentiality values * DATA - 586 - Adding confidentiality values for custom confidentiality * Lower casing as suggested here- DATA-375 --------- Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> * Updating release notes for staging deploy (data-dot-all#301) --------- Co-authored-by: Anushka Singh <anushka.singh@verizonmedia.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> * [Data 611] Disable topics dropdown (data-dot-all#304) * Disabling topics dropdown (data-dot-all#303) * [Data 619] Stagingdeploy env permission fix (data-dot-all#307) * Data:604: Add local level false positive management for PSECBUG - 73521 (data-dot-all#299) * Data:604: Add local level false positive management for PSECBUG - 73521 (data-dot-all#300) * Email notification fix + confidentiality levels config (data-dot-all#298) * DATA-600 - Fix for share link not present in email notifications * Merging changes needed for DATA-509 - Updating custom confidentiality values * Adding confidentiality values for custom confidentiality * Adding confidentiality configs to config.json.PROD * Lower casing as suggested here- DATA-375 --------- Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> * Updating release notes for staging deploy (data-dot-all#301) * Disabling topics dropdown (data-dot-all#303) * DATA-619 - Fix permission for GET_ORGANIZATION when users are in _data teams (data-dot-all#306) * Cherry pick for issue with GET_ORG permission after 2.3 release --------- Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> --------- Co-authored-by: Anushka Singh <anushka.singh@verizonmedia.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> * [Data 631] Staging deploy (data-dot-all#310) * [Data 629] worksheet fix for GET_ENVIRONMENT permission (data-dot-all#309) * Data690 stagingdeploy 20240425 (data-dot-all#319) * DATA-680 - Update node repo to 18.x in Makefile.sd * Data674: Adding auto approval for confidentiality levels (data-dot-all#317) * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Ensuring Secret Confidentiality Type (Yahoo Confidential and Yahoo Highly Confidential) are never auto-approved * Use boolean true instead of string * Update config * Update release notes * Update release notes * Update release notes --------- Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> * Update makefile (data-dot-all#320) * Data690 stagingdeploy 20240425 2 (data-dot-all#321) * Update makefile * Reverting nodejs 16 upgrade * Reverting nodejs 16 upgrade * Data690 stagingdeploy 20240425 3 (data-dot-all#323) * DATA-680 - Update node repo to 18.x in Makefile.sd * Data674: Adding auto approval for confidentiality levels (data-dot-all#317) * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Ensuring Secret Confidentiality Type (Yahoo Confidential and Yahoo Highly Confidential) are never auto-approved * Use boolean true instead of string * Update config * Bugfix (data-dot-all#322) * Reverting nodejs 16 upgrade --------- Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> * Data690 stagingdeploy 20240425 4 (data-dot-all#325) * DATA-680 - Update node repo to 18.x in Makefile.sd * Data674: Adding auto approval for confidentiality levels (data-dot-all#317) * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Ensuring Secret Confidentiality Type (Yahoo Confidential and Yahoo Highly Confidential) are never auto-approved * Use boolean true instead of string * Update config * Bugfix (data-dot-all#322) * Blocking autoApproval edit on backend (data-dot-all#324) * Blocking autoApproval edit on backend * Lint fix * Reverting nodejs 18 upgrade --------- Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> * Data690 stagingdeploy 20240425 5 (data-dot-all#329) * DATA-680 - Update node repo to 18.x in Makefile.sd * Data674: Adding auto approval for confidentiality levels (data-dot-all#317) * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Ensuring Secret Confidentiality Type (Yahoo Confidential and Yahoo Highly Confidential) are never auto-approved * Use boolean true instead of string * Update config * Bugfix (data-dot-all#322) * Blocking autoApproval edit on backend (data-dot-all#324) * Blocking autoApproval edit on backend * Lint fix * DATA-680 - Switch node to version 17 in the Screwdriver makefile (data-dot-all#326) * bugfix (data-dot-all#328) * Remove nodejs upgrade --------- Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> * bugfix (data-dot-all#331) * Data743 stagingdeploy (data-dot-all#351) * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Data743: Update verifier task schedule to run nightly (data-dot-all#350) * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Data743 stagingdeploy (data-dot-all#353) * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * [Data 767] staging deploy (data-dot-all#358) * Bugfix: timeout error when listing Consumption Roles (data-dot-all#1303) - Bugfix - as GraphQL resolvers are 'lazy', for ShareRequest Modal window we simply don't fetch the managedPolicy property -- no timeout - managed policies are fetched, when consumption role is selected from dropdown - data-dot-all#1288 Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: Sofia Sazonova <sazonova@amazon.co.uk> * Updated Release notes --------- Co-authored-by: Sofia Sazonova <sofia-s@304.ru> Co-authored-by: Sofia Sazonova <sazonova@amazon.co.uk> --------- Co-authored-by: Anushka Singh <anushka.singh@yahooinc.com> Co-authored-by: Sofia Sazonova <sofia-s@304.ru> Co-authored-by: Sofia Sazonova <sazonova@amazon.co.uk> * data712 * Data712: Persistent emails * Data712: Persistent emails * Data712: Persistent emails * Data712: Persistent emails * Data712: Persistent emails * Data712: Persistent emails * Data712: Persistent emails * Restore yarn file * Restore yarn file * Update config * Data712: Persistent emails * Data712: Persistent emails * Data712: Persistent emails --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> Co-authored-by: Mohit Arora <marora@yahooinc.com> Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> Co-authored-by: Raj Chopde <rchopde@yahooinc.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@oath.com> Co-authored-by: Sofia Sazonova <sofia-s@304.ru> Co-authored-by: Sofia Sazonova <sazonova@amazon.co.uk>

@anushka-singh

* Bigdata867 3 (data-dot-all#24) * Bucket Policy E.1: Modify sharing task routing to trigger a s3 bucket sharing * Bucket Policy E.1: Modify sharing task routing to trigger a s3 bucket sharing * Bucket Policy E.1: Modify sharing task routing to trigger a s3 bucket sharing * Bucket Policy BIGDATA 867: Implement revoke share in data_sharing_service * Bucket Policy BIGDATA 867: Implement revoke share in data_sharing_service * trajopadhye- BIGDATA-756 -> Added Tests for Task D and E * trajopadhye - BIGDATA-756 Corrected file data_sharing_service.py to address revokedStateSM for revoked items * trajopadhye- BIGDATA-756 - Slight correction in comments * trajopadhye- BIGDATA-756 Correction on Share Status for revoke share tests * Addresed changes from the review of PR * [BIGDATA-625] Implement bucket share processor (data-dot-all#21) * Implement bucket share processor * Fix Revoke UI sharetype * BIGDATA-612 - push source from SD container to CodeCommit. Initial Makefile and SD yaml configuration. * Remove synth * Add force push * Add default cdk.context.json * Add param for branchname * Comments. * Fix email address * Add instance specific cdk.context.json * BIGDATA-612 - truncate the cfn encryption policy prefix so that together with branch name, it will fit within 32 char limit. * Update screwdriver.yaml * Change nodejs version in screwdriver Makefile to supported version 16 (data-dot-all#89) (data-dot-all#90) * Change screwdriver node version to 16 * Remove all non-environment setup steps for testing * Skip getting AWS credentials for testing * Fixing npm install version * Remove extra npm install * Restore all prior functions. * Remove AmplifyContext customizations, no longer needed. (data-dot-all#92) * Change nodejs version in screwdriver Makefile to supported version 16 (data-dot-all#89) * Change screwdriver node version to 16 * Remove all non-environment setup steps for testing * Skip getting AWS credentials for testing * Fixing npm install version * Remove extra npm install * Restore all prior functions. * Remove AmplifyContext customizations, no longer needed. (data-dot-all#91) * Fix screwdriver yaml for new EMR template step. (data-dot-all#116) * Bigdata 1397 mvp 3 stagingdeploy 20231129 (data-dot-all#178) * BIGDATA-1211 - Release notes initial commit * Mvp3 deploy 20231129 - S3 Bucket share + KMS explosion fix - MERGE FROM OPENSOURCE (data-dot-all#176) * Enabling S3 bucket share (data-dot-all#848) - Feature - We want to enable bucket sharing along with access point share which already exists in data all right now. - A user will be able to request shares at bucket level and at the folder level with access points. - Please NOTE: There is some common code between Access point share managers and processors and S3 Bucket managers and processors. We will send out a separate PR for that refactoring work at a later time. - data-dot-all#284 - data-dot-all#823 - https://github.com/awslabs/aws-dataall/pull/846/files#diff-c1f522a1f50d8bcf7b6e5b2e586e40a8de784caa80345f4e05a6329ae2a372d0 - Contents of this PR have been contributed by @anushka-singh, @blitzmohit, @rbernotas, @TejasRGitHub Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Anushka Singh <anushka.singh@yahooinc.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * Kms explosion fix (data-dot-all#882) - Bugfix - DataAll currently creates one SID per role in the KMS policy attached to a bucket with RoleID as the SID name. - We want to collapse these SIDs into one SID. - Access point and Bucket share will have different SIDs in KMS policy. - Use role ARN instead of role ID. - NOTE: if KMS policy was previously created, it will remain the same. SID will be the user ID and not the KMS decrypt SID created in this PR. It will not impact any future shares though. - NOTE: This is to be merged after bucket share PR is merged. - Tested this on local dev environment and KMS policy now has 1 statement with kms decrypt and using SID of KMS decrypt. Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Anushka Singh <anushka.singh@yahooinc.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * Updated Release Notes 20231201 * Format changes --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * [BIGDATA-1391] - Fix for cannot see all cognito groups when inviting teams (data-dot-all#177) * trajopadhye | BIGDATA-1391 - Fix for incomplete groups list fetched for invite org and env --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * Bigdata 1397 mvp 3 stagingdeploy 20231129 1 (data-dot-all#180) * BIGDATA-1211 - Release notes initial commit * Mvp3 deploy 20231129 - S3 Bucket share + KMS explosion fix - MERGE FROM OPENSOURCE (data-dot-all#176) * Enabling S3 bucket share (data-dot-all#848) - Feature - We want to enable bucket sharing along with access point share which already exists in data all right now. - A user will be able to request shares at bucket level and at the folder level with access points. - Please NOTE: There is some common code between Access point share managers and processors and S3 Bucket managers and processors. We will send out a separate PR for that refactoring work at a later time. - data-dot-all#284 - data-dot-all#823 - https://github.com/awslabs/aws-dataall/pull/846/files#diff-c1f522a1f50d8bcf7b6e5b2e586e40a8de784caa80345f4e05a6329ae2a372d0 - Contents of this PR have been contributed by @anushka-singh, @blitzmohit, @rbernotas, @TejasRGitHub Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Anushka Singh <anushka.singh@yahooinc.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * Kms explosion fix (data-dot-all#882) - Bugfix - DataAll currently creates one SID per role in the KMS policy attached to a bucket with RoleID as the SID name. - We want to collapse these SIDs into one SID. - Access point and Bucket share will have different SIDs in KMS policy. - Use role ARN instead of role ID. - NOTE: if KMS policy was previously created, it will remain the same. SID will be the user ID and not the KMS decrypt SID created in this PR. It will not impact any future shares though. - NOTE: This is to be merged after bucket share PR is merged. - Tested this on local dev environment and KMS policy now has 1 statement with kms decrypt and using SID of KMS decrypt. Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Anushka Singh <anushka.singh@yahooinc.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * Updated Release Notes 20231201 * Format changes --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * [BIGDATA-1391] - Fix for cannot see all cognito groups when inviting teams (data-dot-all#177) * trajopadhye | BIGDATA-1391 - Fix for incomplete groups list fetched for invite org and env * Bugfix --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> * Bugfix (data-dot-all#181) * Bugfix * Bugfix * [Data 409] Athenz Certs Domain and User Pool Domain Changes (data-dot-all#221) (data-dot-all#222) * trajopadhye | DATA-409- Code changes for Athenz certs domain and user pool domain * [Data-413] GA stagingdeploy 20231228 - Fix for email notifications with Athenz. Auto-create Pivot Role (data-dot-all#224) * trajopadhye | DATA-412 - Added Athenz configs and Ports in AWS Worker lambda and enabling Auto Create Pivot Role * DATA-416 - Fix while migrating from manual pivot role to auto created (data-dot-all#230) (data-dot-all#233) * trajopadhye | DATA-416 - Fix for environment updates when using auto pivot role. Changing the way KMS keys are specified in env role * [Data 447] ga stagingdeploy 20240116 (data-dot-all#244) * [Data-446] Fix for consumption role not showing up * [Data 415] Dataset import fix for circular dependency error + local dev setup fixes (data-dot-all#243) * DATA-428 - Local env fixes * Data 448 ga stagingdeploy 20240117 (data-dot-all#246) * trajopadhye | DATA-440 - Adding else if to sync glue tabls in RDS * Data 461 ga deploy 20240125 (data-dot-all#258) * DATA-404 - Add git fetch --all to the CodeCommit repo sync * DATA-420 - Switch from Cognito to Okta on Prod (data-dot-all#254) DATA-420 - Switch from Cognito to Okta on Prod * DATA-455: Shares stuck in progress when AWS does not have root access on KMS key (data-dot-all#256) * Update release notes * Update release notes --------- Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> Co-authored-by: Tejas Rajopadhye <tejas.rajopadhye@yahooinc.com> * Data 466 ga stagingdeploy 20240126 (data-dot-all#263) * trajoadhye | DATA-456 - Removing Lake Formation SLR (data-dot-all#260) * Data-405-Adding max 30 sec delay * Synching Release notes from Staging to y-branch-2-0 (data-dot-all#262) * [Data 484] stagingdeploy 20240206 (data-dot-all#275) * fix: adding cdk synth for checkov scans (data-dot-all#264) * [DATA-452] - Adding Dataset description in shares view (data-dot-all#273) * Added Release note for DATA-481, DATA-452, DATA-480 * Syncing Release notes (data-dot-all#274) --------- Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> Co-authored-by: Anushka Singh <anushka.singh@verizonmedia.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@oath.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> * [Data 607] staging deploy email notification fix (data-dot-all#302) * Data:604: Add local level false positive management for PSECBUG - 73521 (data-dot-all#299) * DATA-600 - Fix for share link not present in email notifications * Merging changes needed for DATA-509 - Updating custom confidentiality values * DATA - 586 - Adding confidentiality values for custom confidentiality * Lower casing as suggested here- DATA-375 --------- Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> * Updating release notes for staging deploy (data-dot-all#301) --------- Co-authored-by: Anushka Singh <anushka.singh@verizonmedia.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> * [Data 611] Disable topics dropdown (data-dot-all#304) * Disabling topics dropdown (data-dot-all#303) * [Data 619] Stagingdeploy env permission fix (data-dot-all#307) * Data:604: Add local level false positive management for PSECBUG - 73521 (data-dot-all#299) * Data:604: Add local level false positive management for PSECBUG - 73521 (data-dot-all#300) * Email notification fix + confidentiality levels config (data-dot-all#298) * DATA-600 - Fix for share link not present in email notifications * Merging changes needed for DATA-509 - Updating custom confidentiality values * Adding confidentiality values for custom confidentiality * Adding confidentiality configs to config.json.PROD * Lower casing as suggested here- DATA-375 --------- Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> * Updating release notes for staging deploy (data-dot-all#301) * Disabling topics dropdown (data-dot-all#303) * DATA-619 - Fix permission for GET_ORGANIZATION when users are in _data teams (data-dot-all#306) * Cherry pick for issue with GET_ORG permission after 2.3 release --------- Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> --------- Co-authored-by: Anushka Singh <anushka.singh@verizonmedia.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> * [Data 631] Staging deploy (data-dot-all#310) * [Data 629] worksheet fix for GET_ENVIRONMENT permission (data-dot-all#309) * Data690 stagingdeploy 20240425 (data-dot-all#319) * DATA-680 - Update node repo to 18.x in Makefile.sd * Data674: Adding auto approval for confidentiality levels (data-dot-all#317) * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Ensuring Secret Confidentiality Type (Yahoo Confidential and Yahoo Highly Confidential) are never auto-approved * Use boolean true instead of string * Update config * Update release notes * Update release notes * Update release notes --------- Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> * Update makefile (data-dot-all#320) * Data690 stagingdeploy 20240425 2 (data-dot-all#321) * Update makefile * Reverting nodejs 16 upgrade * Reverting nodejs 16 upgrade * Data690 stagingdeploy 20240425 3 (data-dot-all#323) * DATA-680 - Update node repo to 18.x in Makefile.sd * Data674: Adding auto approval for confidentiality levels (data-dot-all#317) * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Ensuring Secret Confidentiality Type (Yahoo Confidential and Yahoo Highly Confidential) are never auto-approved * Use boolean true instead of string * Update config * Bugfix (data-dot-all#322) * Reverting nodejs 16 upgrade --------- Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> * Data690 stagingdeploy 20240425 4 (data-dot-all#325) * DATA-680 - Update node repo to 18.x in Makefile.sd * Data674: Adding auto approval for confidentiality levels (data-dot-all#317) * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Ensuring Secret Confidentiality Type (Yahoo Confidential and Yahoo Highly Confidential) are never auto-approved * Use boolean true instead of string * Update config * Bugfix (data-dot-all#322) * Blocking autoApproval edit on backend (data-dot-all#324) * Blocking autoApproval edit on backend * Lint fix * Reverting nodejs 18 upgrade --------- Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> * Data690 stagingdeploy 20240425 5 (data-dot-all#329) * DATA-680 - Update node repo to 18.x in Makefile.sd * Data674: Adding auto approval for confidentiality levels (data-dot-all#317) * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Data674: Adding auto approval for confidentiality levels * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Lint fixes * Ensuring Secret Confidentiality Type (Yahoo Confidential and Yahoo Highly Confidential) are never auto-approved * Use boolean true instead of string * Update config * Bugfix (data-dot-all#322) * Blocking autoApproval edit on backend (data-dot-all#324) * Blocking autoApproval edit on backend * Lint fix * DATA-680 - Switch node to version 17 in the Screwdriver makefile (data-dot-all#326) * bugfix (data-dot-all#328) * Remove nodejs upgrade --------- Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> * bugfix (data-dot-all#331) * Data743 stagingdeploy (data-dot-all#351) * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Data743: Update verifier task schedule to run nightly (data-dot-all#350) * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Data743 stagingdeploy (data-dot-all#353) * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * Update verifier task schedule to run nightly * [Data 767] staging deploy (data-dot-all#358) * Bugfix: timeout error when listing Consumption Roles (data-dot-all#1303) - Bugfix - as GraphQL resolvers are 'lazy', for ShareRequest Modal window we simply don't fetch the managedPolicy property -- no timeout - managed policies are fetched, when consumption role is selected from dropdown - data-dot-all#1288 Please answer the questions below briefly where applicable, or write `N/A`. Based on [OWASP 10](https://owasp.org/Top10/en/). - Does this PR introduce or modify any input fields or queries - this includes fetching data from storage outside the application (e.g. a database, an S3 bucket)? - Is the input sanitized? - What precautions are you taking before deserializing the data you consume? - Is injection prevented by parametrizing queries? - Have you ensured no `eval` or similar functions are used? - Does this PR introduce any functionality or component that requires authorization? - How have you ensured it respects the existing AuthN/AuthZ mechanisms? - Are you logging failed auth attempts? - Are you using or adding any cryptographic features? - Do you use a standard proven implementations? - Are the used keys controlled by the customer? Where are they stored? - Are you introducing any new policies/roles/users? - Have you used the least-privilege principle? How? By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: Sofia Sazonova <sazonova@amazon.co.uk> * Updated Release notes --------- Co-authored-by: Sofia Sazonova <sofia-s@304.ru> Co-authored-by: Sofia Sazonova <sazonova@amazon.co.uk> --------- Co-authored-by: Anushka Singh <anushka.singh@yahooinc.com> Co-authored-by: Sofia Sazonova <sofia-s@304.ru> Co-authored-by: Sofia Sazonova <sazonova@amazon.co.uk> * data712 * Data712: Persistent emails * Data712: Persistent emails * Data712: Persistent emails * Data712: Persistent emails * Data712: Persistent emails * Data712: Persistent emails * Data712: Persistent emails * Restore yarn file * Restore yarn file * Update config * Data712: Persistent emails * Data712: Persistent emails * Data712: Persistent emails * Data712: update import * Data712: update import --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: trajopadhye <tejas.rajopadhye@yahooinc.com> Co-authored-by: Mohit Arora <marora@yahooinc.com> Co-authored-by: rbernota <rbernota@yahooinc.com> Co-authored-by: Rick Bernotas <rbernota@verizonmedia.com> Co-authored-by: Raj Chopde <rchopde@yahooinc.com> Co-authored-by: Noah Paige <69586985+noah-paige@users.noreply.github.com> Co-authored-by: dlpzx <71252798+dlpzx@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jaidisido <jaidisido@gmail.com> Co-authored-by: dlpzx <dlpzx@amazon.com> Co-authored-by: mourya-33 <134511711+mourya-33@users.noreply.github.com> Co-authored-by: nikpodsh <124577300+nikpodsh@users.noreply.github.com> Co-authored-by: MK <manjula_kasturi@hotmail.com> Co-authored-by: Manjula <manjula.kasturi@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@gmail.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@yahooinc.com> Co-authored-by: Daniel Lorch <98748454+lorchda@users.noreply.github.com> Co-authored-by: Tejas Rajopadhye <71188245+TejasRGitHub@users.noreply.github.com> Co-authored-by: Zilvinas Saltys <zilvinas.saltys@oath.com> Co-authored-by: Sofia Sazonova <sofia-s@304.ru> Co-authored-by: Sofia Sazonova <sazonova@amazon.co.uk>

noah-paige requested review from dlpzx and noah-paige October 20, 2023 22:17

dbalintx reviewed Oct 25, 2023

View reviewed changes

dbalintx reviewed Oct 26, 2023

View reviewed changes

dlpzx reviewed Oct 27, 2023

View reviewed changes

dlpzx and others added 4 commits October 30, 2023 18:21

Merge branch 'main' into v2m1m0

6d3c016

# Conflicts: # deploy/stacks/backend_stack.py # deploy/stacks/backend_stage.py # deploy/stacks/lambda_api.py # deploy/stacks/pipeline.py # template_cdk.json

anushka-singh force-pushed the v2m1m0 branch from 3d7752e to 75f4468 Compare October 31, 2023 01:03

feat: Enabling S3 bucket share

5fb7cf8

anushka-singh force-pushed the v2m1m0 branch from 75f4468 to 5fb7cf8 Compare October 31, 2023 01:09

anushka-singh and others added 3 commits October 30, 2023 21:11

feat: Enabling S3 bucket share

cf9afc1

Merge branch 'v2m1m0' of https://github.com/anushka-singh/aws-dataall …

ddf8623

…into bucket_share

Merge branch 'v2m1m0' into anu-s3-copy

a05e548

# Conflicts: # backend/dataall/modules/datasets/services/dataset_service.py

dlpzx mentioned this pull request Oct 31, 2023

Test Enabling S3 bucket share - COPY #846

Closed

dlpzx added 3 commits October 31, 2023 09:19

Revert overwrites 2.

1365e92

Revert overwrites 3.

bbcfbd5

Revert overwrites 4.

9e8cdf1

dlpzx added 12 commits October 31, 2023 09:30

Revert overwrites 4.

5d90797

Revert overwrites 5.

94be491

Revert overwrites 6.

cff577f

Revert overwrites 7.

5ff80fb

Revert overwrites 7.

3383166

Revert overwrites 8.

7ed96af

Revert overwrites 9.

c051896

Revert overwrites 10.

f5d62d7

Revert overwrites 11.

3783a95

Revert overwrites 12.

dacba14

Revert overwrites 13.

3b404cd

Fix down revision for migration script

5d0fe68

anushka-singh mentioned this pull request Oct 31, 2023

Enabling S3 bucket share #848

Merged

feat: Enabling S3 bucket share

158925a

anushka-singh closed this Nov 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enabling S3 bucket share #823

Enabling S3 bucket share #823

anushka-singh commented Oct 20, 2023

dbalintx Oct 25, 2023

anushka-singh Oct 30, 2023

dbalintx Oct 25, 2023

dbalintx Oct 25, 2023

dbalintx Oct 26, 2023

anushka-singh Oct 30, 2023

dbalintx Oct 26, 2023

anushka-singh Oct 30, 2023

dbalintx Oct 26, 2023

anushka-singh Oct 30, 2023

dbalintx Oct 26, 2023

anushka-singh Oct 30, 2023

dbalintx Oct 26, 2023

anushka-singh Oct 30, 2023

dbalintx Oct 26, 2023

dbalintx Oct 26, 2023

anushka-singh Oct 30, 2023

dbalintx Oct 26, 2023

anushka-singh Oct 30, 2023

dbalintx Oct 26, 2023

anushka-singh Oct 30, 2023

dbalintx Oct 26, 2023 •

edited

Loading

anushka-singh Oct 30, 2023

dlpzx Oct 27, 2023

dlpzx Oct 27, 2023

anushka-singh commented Nov 2, 2023

		@@ -28,6 +29,8 @@

		logger = logging.getLogger(__name__)

		GACCO_ALLOW_CROSS_ACCOUNT_TAG = ("gacco:AllowPublicAccess", "True")

Enabling S3 bucket share #823

Enabling S3 bucket share #823

Conversation

anushka-singh commented Oct 20, 2023

Feature or Bugfix

Detail

Relates

Security

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dbalintx Oct 26, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anushka-singh commented Nov 2, 2023

dbalintx Oct 26, 2023 •

edited

Loading